FAR-Net: Multi-Stage Fusion for CIR

• FAR-Net is a multi-stage fusion framework for composed image retrieval that integrates late fusion for global semantic alignment with early fusion for local fine-grained correspondence.
• It employs an Enhanced Semantic Alignment Module (ESAM) and an Adaptive Reconciliation Module (ARM) to preserve modality-specific features and adapt to uncertainty-aware perturbations.
• Experimental results on FashionIQ and CIRR benchmarks show improved Recall@1 and Recall@50 metrics, highlighting the framework’s practical performance gains.

Searching arXiv for the FAR-Net paper and closely related CIR work. Using the arXiv search tool to retrieve paper metadata. FAR-Net is a multi-stage fusion framework for composed image retrieval (CIR), a vision-language retrieval setting in which a target image is retrieved from a reference image and a modification text. The model was introduced in “FAR-Net: Multi-Stage Fusion Network with Enhanced Semantic Alignment and Adaptive Reconciliation for Composed Image Retrieval” and is designed to address a specific limitation in prevailing CIR systems: existing methods typically adopt either early fusion or late fusion, whereas FAR-Net combines both in a Late→Early cascade to preserve modality-specific representations, capture fine-grained token-region alignments, and improve robustness under uncertainty-aware perturbations (Park et al., 17 Jul 2025).

1. Problem setting and design rationale

CIR requires a system to interpret a reference image $R_i$ together with a modification text $M_i$ and retrieve the corresponding target image $T_i$. In the formulation used by FAR-Net, the central methodological issue is cross-modal fusion. Early fusion tends to excessively focus on explicitly mentioned textual details and neglect visual context, whereas late fusion struggles to capture fine-grained semantic alignments between image regions and textual tokens (Park et al., 17 Jul 2025).

FAR-Net addresses this by decomposing fusion into two complementary stages. Stage I emphasizes semantic alignment while preserving modality-specific structure; Stage II reconciles the fused representation with target-image embeddings under both deterministic and perturbed conditions. This design suggests that the model treats global alignment and local cross-modal correspondence as distinct but sequentially coupled optimization problems rather than as a single monolithic fusion step.

A common misconception is to view FAR-Net as a purely early-fusion or purely late-fusion architecture. In the published formulation it is neither: it is explicitly a cascaded framework that first applies late fusion in the Enhanced Semantic Alignment Module (ESAM) and then applies early fusion in the Adaptive Reconciliation Module (ARM) (Park et al., 17 Jul 2025).

2. Late→Early cascade and overall architecture

FAR-Net is organized into two modules.

Stage	Module	Role
I	Enhanced Semantic Alignment Module (ESAM)	Late fusion with cross-attention for fine-grained token-region alignment
II	Adaptive Reconciliation Module (ARM)	Early fusion with uncertainty embeddings for robustness and adaptability

ESAM begins from image features extracted by a frozen vision encoder and text features extracted by a language encoder. The module first forms a global late-fusion representation and then enriches it with a Q-Former-based cross-attention mechanism. ARM subsequently reuses ESAM’s intermediate fused representation and jointly optimizes contrastive alignment and a robustness objective based on uncertainty-aware perturbations of the target embedding (Park et al., 17 Jul 2025).

In the paper’s formulation, ESAM and ARM are trained end-to-end rather than as isolated components. The cascade is therefore not merely architectural sequencing; it is also a coupled loss design in which the representation produced by late fusion conditions the objectives optimized by early fusion. This is the main sense in which FAR-Net “unites the global alignment strengths of late fusion with the cross-modal interaction power of early fusion in a single end-to-end pipeline.”

3. Enhanced Semantic Alignment Module (ESAM)

ESAM extracts reference image features $f_\phi(R_i)\in\mathbb{R}^d$ and modification text features $f_\zeta(M_i)\in\mathbb{R}^d$. Its global alignment component constructs a weighted late-fusion representation

$$F_{u_i}=\lambda_1 f_\phi(R_i)+(1-\lambda_1)f_\zeta(M_i), \qquad \lambda_1\in[0,1],$$

which is projected through an MLP into the shared space to obtain $u_i$. The target image is encoded as $v_i=f_\theta(T_i)$, and a temperature-scaled contrastive loss is applied:

$$L_{\text{Late}}=-\frac{1}{|B|}\sum_{i\in B}\log \frac{\exp\bigl(\cos(\text{MLP}(F_{u_i}),v_i)/\tau\bigr)} {\sum_{j\in B}\exp\bigl(\cos(\text{MLP}(F_{u_i}),v_j)/\tau\bigr)}.$$

To model fine-grained token-region correspondence, ESAM uses a Q-Former block. Let $Q_i\in\mathbb{R}^{N_r\times d}$ be query embeddings derived from $M_i$0 image-region features in $M_i$1, and let $M_i$2 be key/value embeddings from $M_i$3 text tokens of $M_i$4. The attention map is computed as

$M_i$5

For the target-image branch, the model extracts $M_i$6 from $M_i$7 and reuses the same $M_i$8 to compute $M_i$9. It then imposes an attention-alignment contrastive loss,

$T_i$0

The ESAM objective is

$T_i$1

Within the published interpretation, $T_i$2 supplies global alignment, while $T_i$3 supplies token-region matching. The reported ablations support both roles: removing $T_i$4 drops the reported FashionIQ average from 65.24 to 46.28 and the CIRR average from 81.92 to 63.64; removing $T_i$5 drops performance to 57.24 on FashionIQ and 67.11 on CIRR (Park et al., 17 Jul 2025).

4. Adaptive Reconciliation Module (ARM)

ARM operates on ESAM’s intermediate fused representation $T_i$6, which encodes the joint $T_i$7 information, and bridges it to the target image embedding $T_i$8. Its first component is an early-fusion contrastive loss. With $T_i$9 as the projected shared-space representation, the retrieval objective is

$f_\phi(R_i)\in\mathbb{R}^d$0

To improve robustness, ARM perturbs the target embedding according to

$f_\phi(R_i)\in\mathbb{R}^d$1

where

$f_\phi(R_i)\in\mathbb{R}^d$2

with $f_\phi(R_i)\in\mathbb{R}^d$3 the mean and standard deviation of the target-image feature distribution. The same contrastive form is applied to $f_\phi(R_i)\in\mathbb{R}^d$4 to produce $f_\phi(R_i)\in\mathbb{R}^d$5, and the resulting Resilience loss is

$f_\phi(R_i)\in\mathbb{R}^d$6

ARM also includes a Prompt-to-Image loss. The model extracts the ESAM text-prompt embedding $f_\phi(R_i)\in\mathbb{R}^d$7, defined as the output tokens of the Q-Former over $f_\phi(R_i)\in\mathbb{R}^d$8, encodes it as $f_\phi(R_i)\in\mathbb{R}^d$9, and aligns it with the target embedding via

$f_\zeta(M_i)\in\mathbb{R}^d$0

The ARM objective is

$f_\zeta(M_i)\in\mathbb{R}^d$1

and the full training loss is

$f_\zeta(M_i)\in\mathbb{R}^d$2

The ablations attribute distinct functional roles to these terms. Dropping $f_\zeta(M_i)\in\mathbb{R}^d$3 causes decreases of 5.79 points on FashionIQ and 7.79 points on CIRR, while dropping $f_\zeta(M_i)\in\mathbb{R}^d$4 yields decreases of 3.18 and 3.47 points, respectively (Park et al., 17 Jul 2025). This suggests that explicit prompt-to-target alignment and uncertainty-aware robustness are complementary rather than interchangeable.

5. Experimental configuration and reported results

The reported experiments use two standard CIR benchmarks. FashionIQ contains 46 K training images, 18 K train queries, and 12 K each for validation and test across three classes—Dress, Shirt, and Toptee. CIRR contains 36 K open-domain images with human-written modifications and is split 80%/10%/10% into train/validation/test; evaluation also reports results on a visually similar subset. Metrics are Recall@$f_\zeta(M_i)\in\mathbb{R}^d$5 for $f_\zeta(M_i)\in\mathbb{R}^d$6 on both datasets, together with subset@$f_\zeta(M_i)\in\mathbb{R}^d$7 on CIRR (Park et al., 17 Jul 2025).

Implementation choices are specified as follows.

Component	Setting
Image encoder	ViT-L (frozen)
Text encoder	BLIP-2’s BERT
Training	Batch size 32, 50 epochs, AdamW, weight decay 0.05
Input and hyperparameters	Images resized to $f_\zeta(M_i)\in\mathbb{R}^d$8 (pad 1.25), $f_\zeta(M_i)\in\mathbb{R}^d$9, $$F_{u_i}=\lambda_1 f_\phi(R_i)+(1-\lambda_1)f_\zeta(M_i), \qquad \lambda_1\in[0,1],$$0 tuned on validation
Hardware	NVIDIA RTX A40 (48 GB)

On CIRR, FAR-Net reaches $$F_{u_i}=\lambda_1 f_\phi(R_i)+(1-\lambda_1)f_\zeta(M_i), \qquad \lambda_1\in[0,1],$$1 versus the prior best 51.96% (SPRC), corresponding to a gain of 2.43 points; it also reports $$F_{u_i}=\lambda_1 f_\phi(R_i)+(1-\lambda_1)f_\zeta(M_i), \qquad \lambda_1\in[0,1],$$2 with a gain of 0.94 and $$F_{u_i}=\lambda_1 f_\phi(R_i)+(1-\lambda_1)f_\zeta(M_i), \qquad \lambda_1\in[0,1],$$3 with a gain of 0.14. On FashionIQ, FAR-Net obtains average $$F_{u_i}=\lambda_1 f_\phi(R_i)+(1-\lambda_1)f_\zeta(M_i), \qquad \lambda_1\in[0,1],$$4 versus the best state of the art at 54.92%, and average $$F_{u_i}=\lambda_1 f_\phi(R_i)+(1-\lambda_1)f_\zeta(M_i), \qquad \lambda_1\in[0,1],$$5 versus 74.97%. The paper highlights overall Recall@1 improvements “up to 2.4%” and Recall@50 improvements of “1.04%” against competing methods when considering specific classes and runs (Park et al., 17 Jul 2025).

These results place FAR-Net within the line of CIR methods that use contrastive objectives, but the gains reported in the paper are specifically associated with the joint use of late-fusion semantic alignment and early-fusion reconciliation rather than with either fusion paradigm alone.

6. Ablation evidence, limitations, and nomenclature

The paper’s ablation studies report that ESAM only, without ARM, yields 62.44 on FashionIQ and 79.28 on CIRR, whereas ARM only, without ESAM, yields 40.18 and 66.33. The authors interpret this as confirmation that the two modules are complementary: ESAM establishes robust global and fine-grained alignment, and ARM adds adaptability under noise together with explicit text-to-image bridging (Park et al., 17 Jul 2025).

Several limitations and future directions are stated explicitly. FAR-Net uses frozen large pre-trained encoders and a small trainable Q-Former, which makes it relatively efficient to adapt, but it still requires paired CIR triplets for supervised contrastive training. Future directions include zero-shot or few-shot domain transfer, such as direct application to new domains without CIR annotations, and exploration of larger unified vision-language backbones including BLIP-3 and Flamingo within the same two-stage fusion paradigm (Park et al., 17 Jul 2025). A plausible implication is that the architecture is intended as a reusable fusion template rather than as a dataset-specific engineering solution.

The name “FAR-Net” is also potentially ambiguous in the broader arXiv literature. In medical imaging, “Learning image from projection: a full-automatic reconstruction (FAR) net for sparse-views computed tomography” uses “FAR net” to denote an end-to-end CT reconstruction model with a Recon-NN and an AS-NN, addressing sparse-view reconstruction rather than CIR (Ma et al., 2019). In high-dimensional statistics, “Factor Augmented Sparse Throughput Deep ReLU Neural Networks for High Dimensional Regression” introduces FAR-NN as a factor-augmented regression estimator based on diversified projections and deep ReLU networks, again unrelated to composed image retrieval (Fan et al., 2022). In current usage within vision-language retrieval, however, FAR-Net ordinarily refers to the 2025 multi-stage fusion network for CIR (Park et al., 17 Jul 2025).