CSFNet: Cosine Similarity Fusion Network

• The paper introduces a two-stage retrieval pipeline that fuses cosine similarity and cosine distance to handle sparse data and enhance information retrieval.
• CSFNet features a trainable encoder-decoder with a Cosine Similarity Attention Fusion Module, efficiently integrating diverse modalities like RGB-X in semantic segmentation.
• The approach achieves notable gains in precision, mIoU, and real-time performance, demonstrating its effectiveness across retrieval and multimodal segmentation tasks.

Cosine Similarity Fusion Network (CSFNet) encompasses both a retrieval-time fusion strategy for information retrieval and a real-time neural encoder–decoder model for multimodal semantic segmentation. The unifying principle is the integration of cosine similarity computations—directly for retrieval and as attention cues for feature fusion in neural architectures. In information retrieval, CSFNet (used here as an Editor's term) refers to a non-neural, two-stage retrieval pipeline fusing cosine similarity and cosine distance to enhance sparse data handling (Juvekar et al., 2024). In multimodal semantic segmentation, CSFNet is a trainable deep network with a dedicated Cosine Similarity Attention Fusion Module, targeting efficient RGB-X fusion for real-time visual tasks (Qashqai et al., 2024).

1. Formal Definitions and Conceptual Foundation

Cosine similarity quantifies the angular closeness between two vectors, typically embeddings:

$$\mathrm{sim}(\mathbf{q},\mathbf{c}) = \cos\angle(\mathbf{q},\mathbf{c}) = \frac{\mathbf{q}^\top \mathbf{c}}{\|\mathbf{q}\|\|\mathbf{c}\|}$$

Values range in $[-1,1]$ for real-valued high-dimensional embeddings. Cosine distance is defined as:

$$\mathrm{dist}(\mathbf{q},\mathbf{c}) = 1 - \mathrm{sim}(\mathbf{q},\mathbf{c})$$

yielding a range $[0,2]$. Cosine similarity is often used in dense retrieval, while its complementary use with cosine distance provides robustness to cases where similarity values are arbitrary, particularly for sparse data with unique information (Juvekar et al., 2024). In neural multimodal fusion, cosine similarity is computed channel-wise to generate adaptive fusion weights reflecting cross-modal agreement (Qashqai et al., 2024).

2. CSFNet in Information Retrieval: Fusion Strategy

The retrieval-oriented instantiation of CSFNet adopts a two-stage pipeline for Retrieval-Augmented Generation (RAG):

1. Preprocessing: Crawl and segment HTML data, embed each chunk by OpenAI’s text-embedding-ada-002.
2. Corpus Partitioning: Partition all chunks $T$ into:
   • $S$: sparse chunks (unique information, not repeated elsewhere)
   • $R = T \setminus S$: non-sparse chunks
3. Hybrid Retriever $H$: Combines BM25 (sparse retriever) and dense vector retrieval for top-$k$ candidate selection.
4. Fusion and Fallback:
   • Retrieve top-$k$ from $[-1,1]$0 via $[-1,1]$1, generate answer $[-1,1]$2.
   • Validate $[-1,1]$3 with an LLM prompt.
   • If validation fails, retrieve from the sparse subset $[-1,1]$4 using $[-1,1]$5, generate $[-1,1]$6, and return $[-1,1]$7.

The fusion is not an explicit weighted sum; instead, similarity-informed retrieval is followed by a fallback on cosine distance within a carefully curated sparse subset if relevance validation fails.

Summary Table: Retrieval CSFNet Steps

Step	Operation	Method/Metric
Chunk embedding	HTML to text → Ada-002	Vector embedding
Corpus partition	Identify $[-1,1]$8 and $[-1,1]$9	Uniqueness criteria
Retrieval (Stage 1)	Hybrid BM25+vector on $$\mathrm{dist}(\mathbf{q},\mathbf{c}) = 1 - \mathrm{sim}(\mathbf{q},\mathbf{c})$$0	Cosine similarity
Validation	LLM-based answer assessment	Prompt evaluation
Fallback (Stage 2)	Hybrid on $$\mathrm{dist}(\mathbf{q},\mathbf{c}) = 1 - \mathrm{sim}(\mathbf{q},\mathbf{c})$$1 (distance focus)	Cosine distance

3. CSFNet for Multimodal Semantic Segmentation

The neural instantiation of CSFNet is an encoder–decoder architecture for real-time RGB-X (e.g., RGB-Depth, RGB-Thermal) semantic segmentation, characterized by early cross-modal fusion and a lightweight inference footprint (Qashqai et al., 2024).

Architectural Components

• Dual-branch then single-branch encoder: Stages 1–3 have two parallel STDC backbones (for RGB and X), fusing at multiple scales via Cosine Similarity Attention Fusion Modules (CS-AFM). Stages 4–5 are single-branch, reducing computational cost without degrading performance.
• CS-AFM module: Computes channel-wise cosine similarity between the two modalities, generating adaptive attention weights through a compact two-layer MLP, followed by fusion and modality rectification operations.
• Efficient context module: Post-encoder, captures long-range context using channel-reducing $$\mathrm{dist}(\mathbf{q},\mathbf{c}) = 1 - \mathrm{sim}(\mathbf{q},\mathbf{c})$$2 convolutions and parallel $$\mathrm{dist}(\mathbf{q},\mathbf{c}) = 1 - \mathrm{sim}(\mathbf{q},\mathbf{c})$$3/$$\mathrm{dist}(\mathbf{q},\mathbf{c}) = 1 - \mathrm{sim}(\mathbf{q},\mathbf{c})$$4 convolutions, achieving high context modeling efficiency.
• Lightweight decoder: Successive upsampling blocks fuse encoder skips back into a full-resolution prediction, utilizing CS-AFM for skip fusion.

Key CS-AFM Operations

Given $$\mathrm{dist}(\mathbf{q},\mathbf{c}) = 1 - \mathrm{sim}(\mathbf{q},\mathbf{c})$$5:

1. Average pool and reshape to $$\mathrm{dist}(\mathbf{q},\mathbf{c}) = 1 - \mathrm{sim}(\mathbf{q},\mathbf{c})$$6.
2. For each channel $$\mathrm{dist}(\mathbf{q},\mathbf{c}) = 1 - \mathrm{sim}(\mathbf{q},\mathbf{c})$$7: $$\mathrm{dist}(\mathbf{q},\mathbf{c}) = 1 - \mathrm{sim}(\mathbf{q},\mathbf{c})$$8
3. Obtain channel-wise scalar vector $$\mathrm{dist}(\mathbf{q},\mathbf{c}) = 1 - \mathrm{sim}(\mathbf{q},\mathbf{c})$$9, process by MLP and sigmoid to get fusion weights $[0,2]$0.
4. Rectify and fuse modalities:
   • $[0,2]$1
   • $[0,2]$2
   • Merge: $[0,2]$3

4. Loss Function, Training, and Hyperparameters

Retrieval Setting

No learnable parameters are introduced for the fusion mechanism. All model components (the embedding model and the LLM for generation/validation) are used out of the box. Hyperparameters tuned via held-out evaluation include chunk size and top-$[0,2]$4.

Segmentation Setting

The segmentation CSFNet uses standard semantic segmentation losses (e.g., cross-entropy, not detailed in the referenced text) and is trained end-to-end. Key hyperparameters:

• CS-AFM MLP reduction ratio: $[0,2]$5 ($[0,2]$6)
• Channel widths: stage-dependent (e.g., $[0,2]$7 at the first stage)
• Decoder upsampling: three $[0,2]$8 and one $[0,2]$9 steps, each with lightweight convolutional blocks.

5. Empirical Results and Comparative Performance

Retrieval CSFNet

Experiments on proprietary web-crawled datasets, with additional metrics on NF-Corpus and SciFact, report:

• Precision: $T$0 vs. $T$1 (hybrid fusion vs. NF-Corpus baseline)
• Recall: $T$2 vs. $T$3
• F-Score: $T$4 vs. $T$5
• Contextual Precision: $T$6 vs. $T$7
• Contextual Recall: $T$8 vs. $T$9
• Contextual Relevancy: $S$0 vs. $S$1
• Answer Relevancy: $S$2 vs. $S$3
• Faithfulness: $S$4 vs. $S$5

On sparse cases, the distance-rerank fallback consistently returns correct answers, and response times are competitive ($S$64--5s vs. 8--9s classical).

Segmentation CSFNet

• Cityscapes: CSFNet-2 achieves 76.36% mIoU at 72.3 FPS (19.37M params, 71.6G FLOPs), outperforming ESANet R34 (75.22% at 43 FPS).
• MFNet: CSFNet-2 yields 59.98% mIoU at 72.7 FPS, leading all real-time methods.
• ZJU: CSFNet-2 reaches 91.40% mIoU at 75.0 FPS.

Maintaining only three dual-branch stages increases speed by 30% with negligible accuracy loss, and use of CS-AFM in the decoder improves mIoU by 0.45 percentage points compared with additive fusion.

6. Context, Applications, and Implications

CSFNet’s retrieval method addresses key limitations of cosine similarity in high-dimensional, sparse contexts by introducing a principled fallback on cosine distance, without incurring additional training or model complexity (Juvekar et al., 2024). Its broad applicability is shown in retrieval-augmented generation and knowledge-intensive information retrieval tasks.

In semantic segmentation, CSFNet demonstrates modality-agnostic design; the same architecture, without modification, performs cross-modal fusion for RGB-Depth, RGB-Thermal, and RGB-Polarization. Early-stage CS-AFM fusion and efficient context modules enable real-time operation for autonomous driving scenarios with a high accuracy–speed tradeoff (Qashqai et al., 2024).

A plausible implication is that channel-wise cosine-based fusion methods, particularly when coupled with early fusion and lightweight architecture design, generalize well across heterogeneous high-dimensional modalities and task domains.

7. Limitations and Notable Considerations

In the retrieval application, there is no end-to-end differentiable path; all "fusion" occurs at the retrieval and context assembly steps, with no parameter learning in the fusion logic itself. Effectiveness is connected to the quality of the initial corpus partitioning and the performance of the off-the-shelf embedding models. In segmentation, parameters such as the stage depth for dual-branch execution and the use of CS-AFM in the decoder present tradeoffs between speed and accuracy, but empirical ablations confirm negligible accuracy degradation for large runtime gains.

In both domains, the fusion via cosine similarity provides a robust, generalizable mechanism and introduces minimal additional computational or representational overhead, positioning CSFNet variants as efficient baselines for future development in retrieval and multimodal fusion tasks.

---

Primary References:

• "COS-Mix: Cosine Similarity and Distance Fusion for Improved Information Retrieval" (Juvekar et al., 2024)
• "CSFNet: A Cosine Similarity Fusion Network for Real-Time RGB-X Semantic Segmentation of Driving Scenes" (Qashqai et al., 2024)