AdaFusion: Adaptive Multi-modal Fusion

• AdaFusion is an adaptive fusion framework that combines text and diffusion-generated image features in interactive retrieval to mitigate generative noise.
• It employs a dual-branch architecture with a reliability gating module and a semantic-aware mixture-of-experts branch to balance modality contributions.
• Empirical evaluations demonstrate enhanced retrieval accuracy with improved Hits@10 and reduced rank drops compared to static fusion methods.

Searching arXiv for the cited paper and closely related interactive text-to-image retrieval work. AdaFusion denotes the overarching principle of adaptively fusing multi-modal query views in diffusion-augmented interactive text-to-image retrieval (I-TIR). In this setting, a system synthesizes a proxy image from the dialogue context and combines that diffusion-generated visual evidence with the user’s textual feedback to form a retrieval query. ADaFuSE, introduced as “Adaptive Diffusion-Text Fusion with Semantic-aware Experts,” is a concrete instantiation of this principle: a lightweight, plug-in fusion model that aligns and calibrates text and diffusion-generated image features without modifying the backbone encoders, and that is designed to replace static additive fusion in diffusion-augmented I-TIR (Zhang et al., 23 Mar 2026).

1. Problem setting and motivation

Interactive text-to-image retrieval enables iterative image search through multi-round natural-language feedback. At round $n$ for sample $i$, the system has the current textual query $T_{n,i}$, aggregated over rounds, and seeks to retrieve the target image $I_i^*$. In diffusion-augmented I-TIR, instead of a reliable reference image, the system synthesizes a proxy image $I_{n,i}$ from the dialogue context using a diffusion model, then fuses the text and generated image to form a stronger query (Zhang et al., 23 Mar 2026).

This formulation is more challenging than classical composed image retrieval because the synthetic image can be noisy or off-intent. The generated image is embedded by the backbone visual encoder to yield a diffusion-generated feature that complements the text, thereby providing a visual “view” of the user feedback and narrowing the modality gap. However, prior frameworks such as DAR fuse the two modality embeddings by static weighted addition. The central claim behind AdaFusion is that this static and undifferentiated fusion indiscriminately incorporates generative noise produced by the diffusion model, and that the semantic consistency between the generated image and the textual intent varies from instance to instance.

The degradation induced by static fusion is quantified directly in retrieval space. The degradation rate, defined as the proportion of queries where adding diffusion evidence worsens retrieval, reaches up to $55.62\%$ of samples and exceeds $50\%$ from round 2 onward. For degraded queries under static fusion, the average rank drop is approximately $7{,}500$ positions. This establishes the paper’s core motivation: adaptive fusion is required to calibrate the contribution of each modality per instance rather than applying a fixed-weight addition (Zhang et al., 23 Mar 2026).

2. Dual-branch fusion architecture

ADaFuSE instantiates AdaFusion through a dual-branch fusion module positioned between the backbone encoders and the retrieval scorer. The inputs are the text $T_{n,i}$ and the diffusion-generated image $I_{n,i}$. Pretrained text and image encoders, denoted $i$0 and $i$1, produce shared-space embeddings $i$2 and $i$3, while the target image embedding is $i$4 (Zhang et al., 23 Mar 2026):

$i$5

Two modality-specific projection heads then lift these embeddings into a higher-dimensional latent space:

$i$6

where $i$7 is GELU and $i$8. The projected features are concatenated into a joint context vector

$i$9

The first branch is an adaptive gating branch that estimates modality reliability and produces a scalar gate $T_{n,i}$0:

$T_{n,i}$1

with $T_{n,i}$2, $T_{n,i}$3, and biases $T_{n,i}$4. This gate interpolates the original $T_{n,i}$5-dimensional modality embeddings:

$T_{n,i}$6

A notable design choice is that the gate is a single sigmoid scalar rather than a per-modality softmax vector; equivalently, $T_{n,i}$7 and $T_{n,i}$8.

The second branch is a semantic-aware mixture-of-experts branch. It defines $T_{n,i}$9 experts $I_i^*$0 and a routing network $I_i^*$1 that outputs logits $I_i^*$2. The routing distribution and residual representation are

$I_i^*$3

No temperature is used in the softmax. The MoE branch is intended to capture fine-grained cross-modal nuances through specialized experts and instance-conditioned routing.

The final fused query representation is obtained by residual addition of the MoE output to the gated base fusion, followed by $I_i^*$4 normalization:

$I_i^*$5

where $I_i^*$6 projects the MoE output back to the original embedding dimension. In operational terms, the module encodes $I_i^*$7 and $I_i^*$8, forms $I_i^*$9, applies reliability-gated interpolation and MoE residual refinement, normalizes the result, and ranks target images by cosine similarity $I_{n,i}$0 using the backbone’s image embeddings (Zhang et al., 23 Mar 2026).

3. Training objective and plug-in deployment

ADaFuSE is trained end-to-end atop fixed pretrained BLIP encoders using symmetric InfoNCE over the shared embedding space. Let $I_{n,i}$1 be cosine similarity and $I_{n,i}$2 the temperature. The basic contrastive term is

$I_{n,i}$3

The full symmetric objective sums both query-to-image and image-to-query directions over the batch:

$I_{n,i}$4

No additional cross-modal consistency or alignment losses are added beyond InfoNCE. No regularization terms for reliability calibration or expert sparsity are used. This is a deliberate design choice: the module is intended to remain lightweight and plug-in.

Training is performed on DA-VisDial with pretrained BLIP encoders initialization. The optimizer, learning rate, batch size, and number of epochs are not specified in the paper; the released code is identified as the source for exact settings. The hyperparameters $I_{n,i}$5, $I_{n,i}$6, $I_{n,i}$7, $I_{n,i}$8, and $I_{n,i}$9 are tuned empirically (Zhang et al., 23 Mar 2026).

The deployment model is equally constrained. ADaFuSE does not modify $55.62\%$0 or $55.62\%$1, does not require retraining the corpus index, and is compatible with existing diffusion-augmented I-TIR systems in which a generated image is already available. The generated image $55.62\%$2 is embedded by the same backbone visual encoder $55.62\%$3 that encodes corpus images, ensuring modality alignment in a shared space. This plug-in positioning is central to its identity as an adaptive fusion layer rather than an alternative retrieval backbone.

4. Evaluation protocol and empirical performance

The evaluation is conducted on four standard I-TIR benchmarks: VisDial (validation set), ChatGPT_BLIP2, Human_BLIP2, and Flan-Alpaca-XXL_BLIP2. The primary metric is accumulated recall at rank $55.62\%$4, reported mainly as Hits@10. The principal baselines are ChatIR, which uses dialogue text only, and DAR, the diffusion-augmented baseline with static additive fusion of text and generated image features (Zhang et al., 23 Mar 2026).

On VisDial, ADaFuSE consistently outperforms DAR, with Hits@10 gains growing from $55.62\%$5 at round 0 to $55.62\%$6 at round 10. Across ChatGPT_BLIP2, Human_BLIP2, and Flan-Alpaca-XXL_BLIP2, it surpasses both ChatIR and DAR at every round. The latter three benchmarks are described as out-of-distribution, and the reported behavior therefore emphasizes generalization across longer and noisier interactive queries rather than only in-domain improvements.

The model’s efficiency claim is equally specific. ADaFuSE adds only $55.62\%$7 parameters over the backbone while surpassing DAR by up to $55.62\%$8 in Hits@10. FLOPs and runtime or latency are not reported, but the architecture is described as using small MLPs and lightweight experts, making it practical as a plug-in (Zhang et al., 23 Mar 2026).

The paper’s robustness analysis is framed in retrieval-side rather than generation-side terms. Static fusion degrades up to $55.62\%$9 of samples, whereas ADaFuSE maintains lower degradation rates across rounds and reduces the average rank drop on degraded cases from approximately $50\%$0 positions to approximately $50\%$1. This is the strongest empirical evidence that the adaptive fusion mechanism suppresses generative noise rather than simply shifting the fused embedding distribution (Zhang et al., 23 Mar 2026).

5. Mechanistic interpretation and ablation logic

The non-adaptive reference point is static addition, represented by DAR. Its high degradation rates and large negative rank shifts are taken as evidence that fixed fusion is insufficient under generative noise. The gating branch addresses this at a coarse level by performing reliability calibration. The reported analysis shows that the adaptive gate correlates with text-image alignment: as cosine similarity between text and generated image increases, ADaFuSE up-weights the image more. At the same time, the model remains conservative overall, with image weight approximately $50\%$2 versus DAR’s approximately $50\%$3 (Zhang et al., 23 Mar 2026).

This behavior clarifies the intended role of the gate. It is not a generic attention layer over two modalities, but an instance-specific reliability estimator designed to prevent noisy diffusion evidence from dominating the fused query. A common misconception would be to interpret the generated image as a uniformly helpful auxiliary cue. The reported degradation statistics and conservative gating behavior indicate the opposite: the generated image is useful only when its semantics align with the text intent.

The semantic-aware MoE branch operates at a different level. It is described as capturing cross-modal nuances that linear gating cannot, including subtle attribute changes and compositional cues. Routing via softmax enables instance-conditioned specialization among the $50\%$4 experts. The final system therefore separates coarse reliability balancing from fine-grained residual synthesis.

The paper does not report numeric ablations for “gating-only” and “MoE-only.” It instead provides an analytical decomposition of their roles: gating mitigates noise, while the MoE adds fine-grained, context-dependent detail. This suggests that ADaFuSE’s performance improvements depend not only on down-weighting unreliable images, but also on a non-linear residual path that restores cross-modal detail suppressed by a purely convex interpolation (Zhang et al., 23 Mar 2026).

6. Limitations, scope, and broader significance

The paper identifies several failure modes. Poor diffusion generations with severe semantic mismatch can limit utility even with gating. Domain shifts beyond the pretrained encoders’ distribution may reduce alignment quality. Very long or ambiguous dialogues can yield conflicting cues that are hard to calibrate. Potential biases are inherited from the backbone encoders and the diffusion model, and ADaFuSE does not directly debias data (Zhang et al., 23 Mar 2026).

Several future directions are named explicitly. These include enhanced reliability estimation through multi-signal gates, uncertainty modeling, or per-dimension gates; adaptive expert selection with sparsity or top-$50\%$5 routing; and explicit consistency losses or uncertainty-aware regularizers that do not compromise plug-in simplicity. These proposals remain outside the reported system and should therefore be understood as open extensions rather than current capabilities.

Within its stated scope, the broader significance of AdaFusion is methodological. It treats generative augmentation not as a replacement for retrieval encoders and not as an end-to-end fine-tuning objective, but as an auxiliary view that must be calibrated at the fusion stage. In that sense, ADaFuSE advances a particular interpretation of diffusion-augmented I-TIR: generative augmentation coupled with principled fusion provides a simple, generalizable alternative to fine-tuning for interactive retrieval, especially in later rounds where text becomes longer and noisier (Zhang et al., 23 Mar 2026).

In this usage, “AdaFusion” is therefore not merely a model name. It denotes an adaptive fusion principle in which text and diffusion-generated image evidence are balanced per instance through reliability-aware interpolation and semantic-aware residual enrichment. ADaFuSE is the specific realization of that principle introduced for interactive text-to-image retrieval (Zhang et al., 23 Mar 2026).