Multimodal Collaborative Fusion Strategy

• MCF is a collaborative design pattern for multimodal learning that integrates diverse features through shared latent spaces, adaptive weighting, and recursive refinement.
• It dynamically adapts fusion mechanisms based on modality quality and uncertainty, ensuring robust performance even with missing or degraded data.
• MCF finds applications in recommendation, saliency detection, medical retrieval, and depression detection, showcasing its flexible and practical utility.

Multimodal Collaborative Fusion Strategy (MCF) denotes a class of multimodal learning designs in which modalities are not merely concatenated, averaged, or fused at a single fixed stage, but are made to collaborate through shared latent spaces, state-dependent weighting, iterative refinement, uncertainty-aware calibration, or behavior-conditioned interaction. Across recent work, MCF appears in recommendation, saliency detection, medical retrieval, collaborative perception, video fusion, depression detection, and multimodal classification. A clear instance is the recommendation framework “Bridging Collaborative Filtering and LLMs with Dynamic Alignment, Multimodal Fusion and Evidence-grounded Explanations” (Ma et al., 2 Oct 2025), but related formulations also include equilibrium-based recursive fusion (Ni et al., 2023), behavior-level cross fusion for recommendation (Zhang et al., 20 Aug 2025), uncertainty-calibrated decision fusion (Cao et al., 2024), and pre-fusion calibration modules that decide what to suppress, preserve, or amplify before downstream fusion (Liu et al., 1 Jun 2026).

1. Conceptual scope and defining properties

In the literature summarized here, MCF is best understood as a design pattern rather than a single architecture. Its defining property is collaborative dependence among modalities: the representation or weight assigned to one modality is influenced by the state, quality, or summary of the others. This distinguishes MCF from simple early fusion, in which features are merged once and passed downstream, and from pure late fusion, in which modalities are processed independently and only their final predictions are combined.

Recent papers instantiate this idea at different levels. Some methods operate in a shared latent space, as in the recommendation framework where collaborative filtering (CF), text, vision, and audio are projected into a common latent anchored by CF (Ma et al., 2 Oct 2025). Some model collaboration as recursive or equilibrium-seeking interaction between modality-specific states and a fused state, as in Deep Equilibrium Multimodal Fusion (Ni et al., 2023). Others perform collaborative refinement before fusion: VGMR compares summary statistics across modalities, extracts support and discrepancy signals, and converts them into instance-wise and dimension-wise modulation gates (Liu et al., 1 Jun 2026). There are also behavior-level MCF formulations, where text and image signals are fused only after they have been transformed into target-conditioned user-interest sequences (Zhang et al., 20 Aug 2025), and decision-level variants, where per-modality weights are generated from collaborative confidence signals derived from both self-belief and the weakness of other modalities (Cao et al., 2024).

This diversity implies that MCF is not tied to a particular operator such as cross-attention, mixture-of-experts, or tensor fusion. A plausible synthesis is that MCF names any fusion strategy in which multimodal interaction is explicitly organized so that modalities can compensate for one another, preserve complementary information, and remain useful under varying data quality or availability. The concrete mechanism may be a shared latent bottleneck, recursive message exchange, confidence weighting, gating, or distributional transport.

2. Architectural motifs: shared spaces, collaborative states, and adaptive weighting

A recurring motif in MCF is projection into a common representation space followed by collaboration inside that space. In the recommendation setting of DynMM-Explain-LLMRec, each side modality is mapped to the CF-aligned latent dimension by a shared projector $$f_{proj}: \mathbb{R}^{d_m} \rightarrow \mathbb{R}^{d}$$, with $d=256$ in experiments; CLIP ViT-B/32 provides $\mathbf e_i^{vis}\in\mathbb R^{512}$ and Wav2Vec2-Base provides $\mathbf e_i^{aud}\in\mathbb R^{768}$ before projection (Ma et al., 2 Oct 2025). DUP-MCRNet applies a similar common-space principle for RGB, depth, and edge feature maps using $1\times1$ convolutions and per-modality self-attention before weighted fusion (Xiong et al., 28 Aug 2025). DMCL reduces the image branch from $2048$ to $768$ dimensions to match BERT token embeddings and then constructs a joint image-text sequence $K=[I,T_1,T_2,T_3,T_4,T_5]$ for self-attention fusion (Xiang et al., 2024).

A second motif is maintenance of both local modality states and a shared collaborative state. DEQ Fusion explicitly keeps modality-specific equilibrium states $\{\mathbf z_i^*\}$ and a fused equilibrium state $\mathbf z_{\mathrm{fuse}}^*$, and updates them recursively until a root solver reaches a fixed point (Ni et al., 2023). DepMamba uses a related shared/private decomposition in state-space form: audio and visual streams have separate input and output maps, but share the state transition matrix $d=256$0 in the multimodal collaborative SSM, then are concatenated and refined by an enhanced SSM (Ye et al., 2024). Progressive Fusion generalizes the same intuition by computing a fused state $d=256$1, projecting it back into unimodal encoders through $d=256$2, and unrolling this refinement for $d=256$3 steps (Shankar et al., 2022).

A third motif is adaptive weighting. In recommendation, the online update gate is explicit: $d=256$4 so the collaborative latent is corrected only to the degree justified by fresh interaction summaries (Ma et al., 2 Oct 2025). In saliency detection, the written fusion rule is

$d=256$5

which fuses attention-encoded RGB, depth, and edge features by softmax-normalized modality weights (Xiong et al., 28 Aug 2025). In predictive dynamic fusion, the decision-level weight is derived from a calibrated collaborative belief rather than a feature attention map: $d=256$6 where $d=256$7 is based on Mono-Confidence, Holo-Confidence, and relative calibration (Cao et al., 2024).

These motifs show that MCF is less about any single fusion block than about where collaboration is injected: before fusion, during feature refinement, at the latent bottleneck, or at the final decision layer.

3. Behavior-centered and anchor-centered MCF in recommendation

The recommendation literature in the supplied corpus provides two particularly explicit MCF formulations. DynMM-Explain-LLMRec treats CF as the anchor modality and aligns text, vision, and audio around that anchor; DMAE treats text and image as separate behavior-conditioned interest streams and fuses them only after sequence-aware interest modeling.

DynMM-Explain-LLMRec is architecturally explicit. It contains a frozen CF backbone instantiated with SASRec, a frozen base aligner producing $d=256$8 and $d=256$9, a multimodal branch with frozen CLIP ViT-B/32 and Wav2Vec2-Base encoders, a lightweight online adapter $\mathbf e_i^{vis}\in\mathbb R^{512}$0, an LLM projection module that converts joint latents into soft tokens, and an evidence-grounding module that constructs evidence tokens from collaborative neighbors and salient item attributes (Ma et al., 2 Oct 2025). Its MCF character lies in the claim that collaborative signals are the stable preference-bearing representation, while text, vision, and audio are aligned into the same latent space around them. The multimodal alignment term is written as a unified contrastive objective over available modality pairs and explicitly names $\mathbf e_i^{vis}\in\mathbb R^{512}$1, $\mathbf e_i^{vis}\in\mathbb R^{512}$2, and $\mathbf e_i^{vis}\in\mathbb R^{512}$3. The reconstruction term treats the shared latent as a bottleneck from which available modalities in $\mathbf e_i^{vis}\in\mathbb R^{512}$4 are decoded. The resulting fusion is therefore neither pure early concatenation nor late score averaging, but a shared-latent hybrid with contrastive alignment and reconstruction.

That design also connects fusion to explanation. For a user–item pair $\mathbf e_i^{vis}\in\mathbb R^{512}$5, collaborative evidence is defined by nearest neighbors in CF embedding space,

$\mathbf e_i^{vis}\in\mathbb R^{512}$6

and multimodal attribute evidence $\mathbf e_i^{vis}\in\mathbb R^{512}$7 is extracted via attention weights over text and visual features. Evidence tokens $\mathbf e_i^{vis}\in\mathbb R^{512}$8, with $\mathbf e_i^{vis}\in\mathbb R^{512}$9, are inserted into the LLM prompt, making the fused representation serve ranking and rationale generation jointly. This is an important extension of MCF from prediction to evidence-grounded explanation.

The empirical results support the relevance of this formulation. On Movies & TV, Games, Beauty, and Toys, DynMM-Explain-LLMRec reaches Hit@10/NDCG@10 of $\mathbf e_i^{aud}\in\mathbb R^{768}$0, $\mathbf e_i^{aud}\in\mathbb R^{768}$1, $\mathbf e_i^{aud}\in\mathbb R^{768}$2, and $\mathbf e_i^{aud}\in\mathbb R^{768}$3, all above A-LLMRec. The incremental ablation reports A-LLMRec at $\mathbf e_i^{aud}\in\mathbb R^{768}$4, then $\mathbf e_i^{aud}\in\mathbb R^{768}$5Dynamic $\mathbf e_i^{aud}\in\mathbb R^{768}$6, $\mathbf e_i^{aud}\in\mathbb R^{768}$7Multimodal $\mathbf e_i^{aud}\in\mathbb R^{768}$8, and $\mathbf e_i^{aud}\in\mathbb R^{768}$9Evidence $1\times1$0 on Movies & TV. Removing multimodal fusion lowers Hit@10 from $1\times1$1 to $1\times1$2, and removing $1\times1$3 lowers it to $1\times1$4. Under missing modalities on Beauty, Hit@10 falls from $1\times1$5 to $1\times1$6 with vision removed, to $1\times1$7 with text removed, and to $1\times1$8 with both removed. At the same time, the paper leaves two details underdescribed: audio is introduced as a modality but omitted from the written $1\times1$9 equation, and the missing-modality gating mechanism is described qualitatively rather than fully formalized.

DMAE addresses a different recommendation bottleneck. Rather than fusing text and image features at the item level, it computes modality-specific target–history similarity sequences,

$2048$0

encodes them as behavior-level interest tokens with temporal information, applies sliding-window intra-modal attention, and then performs cross fusion by using the pooled context of one modality as a query over the sequence of the other (Zhang et al., 20 Aug 2025). This produces $2048$1 and $2048$2, which are combined with user embedding, target ID embedding, and DIN output for CTR prediction. A KL-based decoder reconstructs the user’s true interest distribution over time segments and similarity bins. In effect, DMAE makes collaboration happen after each modality has already become a behavior-conditioned interest process. That is a different but fully compatible meaning of MCF: behavior-level cross fusion rather than content-level early fusion.

4. Dynamic, recursive, and pre-fusion formulations

Several recent formulations broaden MCF beyond shared-latent recommendation. Deep Equilibrium Multimodal Fusion models multimodal collaboration as a fixed-point problem: $2048$3 with separate modality-specific equilibrium states and a fused equilibrium state. The fused branch computes modality-conditioned soft gates $2048$4, purifies the current fused state with respect to each modality, and updates the shared state recursively until a root solver, implemented with Anderson acceleration, converges (Ni et al., 2023). This formulation is MCF in the strict sense that fusion is the equilibrium of a collaborative dynamical system rather than a one-pass operator.

Progressive Fusion introduces a different recursive principle: compute a fused representation, backproject it into unimodal encoders, and repeat. Formally,

$2048$5

This makes the fused state available to earlier unimodal processing and turns late fusion into iterative co-refinement (Shankar et al., 2022). DepMamba specializes the same progression to bidirectional Mamba/state-space modeling. In its collaborative SSM,

$2048$6

the shared transition matrix $2048$7 captures intermodal dynamics while modality-specific $2048$8 and $2048$9 preserve intramodal structure; concatenation and a second enhanced SSM then strengthen modality cohesion (Ye et al., 2024).

Pre-fusion calibration pushes collaboration even earlier. VGMR first projects each modality into a shared latent space, pools average and max summaries, computes support vectors $768$0 and discrepancy vectors $768$1, and transforms these into an instance-wise scalar $768$2 and a dimension-wise vector $768$3. A final gate

$768$4

modulates the original modality feature before any downstream fusion (Liu et al., 1 Jun 2026). This makes MCF a “what to keep” problem rather than only a “how to combine” problem.

Other papers move collaboration to prediction or branch selection. Predictive Dynamic Fusion defines Mono-Confidence, Holo-Confidence, and Co-Belief to obtain weights whose covariance with own loss is negative and with other modalities’ losses is positive, then applies relative calibration using Distribution Uniformity (Cao et al., 2024). Meta Fusion constructs a cohort of unimodal and multimodal students over many representation combinations, trains them with soft information sharing from top-performing peers, and selects a committee for inference (Liang et al., 27 Jul 2025). FusionFM organizes collaboration simultaneously across source modalities, teacher fusion models, refinement submodules, and tasks over time by learning direct source-to-fusion transport with flow matching, selecting task-aware pseudo-labels from multiple SOTA fusion models, refining those labels, and preserving prior task competence with EWC and replay (Zhu et al., 17 Nov 2025).

Taken together, these formulations show that MCF is not reducible to feature concatenation plus attention. It includes equilibrium solvers, progressive feedback, pre-fusion calibration, confidence-theoretic decision fusion, and model-cohort mutual learning.

5. Robustness, uncertainty, and single-modality operation

A central motivation for MCF is that multimodal systems often degrade under missing, weak, or conflicting modalities. The recent literature treats this not as a peripheral issue but as a structural property of the fusion design.

SiMO provides the clearest diagnosis in collaborative perception. Its claim is that conventional multimodal fusion causes a semantic mismatch between single-modality features and downstream modules, so when one sensor disappears the remaining branch outputs no longer live in the semantic space expected by the task head and the cross-agent collaboration module (Wen et al., 9 Mar 2026). To address this, SiMO pretrains each modality branch independently, aligns them into a common BEV semantic space, and fuses them with Length-Adaptive Multi-Modal Fusion (LAMMA). LAMMA forms queries from the concatenation of available modality tokens, performs per-modality attention, splits the $768$5-length outputs, sums the halves, and then additively combines modality outputs: $768$6 When one modality is missing, the query becomes partially empty and the block degenerates into self-attention rather than collapsing. The design objective is not only accuracy but semantic compatibility under varying modality cardinality.

Uncertainty-aware saliency detection offers a different perspective. DUP-MCRNet refines RGB, depth, and edge features with a Dynamic Uncertainty Graph Convolution stage, then applies Multimodal Collaborative Fusion using per-modality self-attention and softmax-normalized weights (Xiong et al., 28 Aug 2025). Its ablation shows MCF alone improves the base model on ECSSD, HKU-IS, and SOD: for example, on SOD, MAE improves from $768$7 to $768$8, $768$9 from $K=[I,T_1,T_2,T_3,T_4,T_5]$0 to $K=[I,T_1,T_2,T_3,T_4,T_5]$1, $K=[I,T_1,T_2,T_3,T_4,T_5]$2 from $K=[I,T_1,T_2,T_3,T_4,T_5]$3 to $K=[I,T_1,T_2,T_3,T_4,T_5]$4, and weighted-$K=[I,T_1,T_2,T_3,T_4,T_5]$5 from $K=[I,T_1,T_2,T_3,T_4,T_5]$6 to $K=[I,T_1,T_2,T_3,T_4,T_5]$7. At the same time, the paper presents an instructive discrepancy: it describes dynamic scene-dependent modality adjustment, but the written formula uses a global learnable vector $K=[I,T_1,T_2,T_3,T_4,T_5]$8, so the mathematically explicit mechanism is global softmax weighting rather than per-sample gating.

VideoFusion demonstrates that robustness can require temporal as well as cross-modal collaboration. It uses a Cross-modal Differential Reinforcement Module to extract complementary differential information, a Complete Modality-Guided Fusion module that employs a shared query to retrieve infrared and visible information separately, and a Bi-temporal Co-attention Module that fuses previous and next frame contexts through co-attention (Tang et al., 30 Mar 2025). Its variational consistency loss aligns temporal changes in fused and restored outputs with source-video temporal variation, addressing flicker and instability that frame-wise image fusion methods exhibit in sequential settings.

VGMR and PDF address robustness through calibration. VGMR shows that under synthetic corruption, high value does not imply preservation: when $K=[I,T_1,T_2,T_3,T_4,T_5]$9 of text timesteps are replaced with Gaussian noise on MOSI, both global and channel value signals increase, but the average gate response decreases, indicating suppression of corrupted features (Liu et al., 1 Jun 2026). PDF computes Distribution Uniformity from each modality’s softmax output and downweights collaborative belief only when the modality is relatively less certain than its peers (Cao et al., 2024). These two papers converge on the same practical point: collaborative fusion must be able to suppress as well as enhance.

A common misconception is that adding more expressive interaction always improves robustness. The comparative study on concatenation and cross-attention argues otherwise: when visual and textual features are already aligned by a vision-language objective, concatenation outperforms cross-attention by $\{\mathbf z_i^*\}$0–$\{\mathbf z_i^*\}$1 percentage points across all tested scales from $\{\mathbf z_i^*\}$2 to $\{\mathbf z_i^*\}$3 samples, and the theoretical explanation attributes this to $\{\mathbf z_i^*\}$4 rather than $\{\mathbf z_i^*\}$5 sample complexity for learning the fusion projection (Zhou et al., 31 May 2026). This suggests that, for MCF, robustness may depend as much on upstream alignment and semantic compatibility as on the complexity of the fusion operator.

6. Empirical record, trade-offs, and open questions

The empirical record across domains is broadly favorable to MCF-style designs, but it is not uniform and it comes with recurring trade-offs. In biomedical, audiovisual, and RGB-D tasks, DEQ Fusion improves MM-Dynamics on BRCA from $\{\mathbf z_i^*\}$6 to $\{\mathbf z_i^*\}$7 accuracy and raises overall accuracy on VQA-v2 when substituted into Mutan or MCAN (Ni et al., 2023). In colonoscopic polyp re-identification, DMCL improves mAP from $\{\mathbf z_i^*\}$8 to $\{\mathbf z_i^*\}$9 and Rank-1 from $\mathbf z_{\mathrm{fuse}}^*$0 to $\mathbf z_{\mathrm{fuse}}^*$1 relative to the baseline, with image-only and text-only variants both weaker than the joint model (Xiang et al., 2024). In recommendation, DMAE surpasses DIN, SimTier, and other baselines on Amazon-Books, Amazon-Electro, MovieLens, and an industrial dataset, and the online A/B test reports increases of $\mathbf z_{\mathrm{fuse}}^*$2 in $\mathbf z_{\mathrm{fuse}}^*$3, $\mathbf z_{\mathrm{fuse}}^*$4 in $\mathbf z_{\mathrm{fuse}}^*$5, $\mathbf z_{\mathrm{fuse}}^*$6 in overall CTR, and $\mathbf z_{\mathrm{fuse}}^*$7 in revenue (Zhang et al., 20 Aug 2025). Meta Fusion is best in all six simulation settings reported and attains the highest average accuracy on the NACC Alzheimer’s task, with mean accuracy $\mathbf z_{\mathrm{fuse}}^*$8 against $\mathbf z_{\mathrm{fuse}}^*$9 for Best Single and $d=256$00 for simple average (Liang et al., 27 Jul 2025).

These gains coexist with clear costs. DEQ-based fusion introduces root-finding overhead and input-dependent convergence speed (Ni et al., 2023). Progressive Fusion increases training and inference time approximately in proportion to the number of unrolling steps $d=256$01, even though parameter overhead is small (Shankar et al., 2022). VGMR raises parameter count from $d=256$02M to $d=256$03M and inference time per batch from $d=256$04 ms to $d=256$05 ms on MOSEI Transformer (Liu et al., 1 Jun 2026). SiMO’s multi-stage Pretrain-Align-Fuse-RD schedule is slower than naive end-to-end training, and its attention-plus-addition LAMMA is less robust to high Gaussian LiDAR noise than convolutional fusion, even while it is far more robust to complete modality failure (Wen et al., 9 Mar 2026). FusionFM is efficient at inference, with $d=256$06 s at $d=256$07 and $d=256$08 s at $d=256$09, but depends on a pool of strong teacher fusion models and contains several typographically corrupted equations in the refinement and continual-learning sections (Zhu et al., 17 Nov 2025).

Several open questions recur. One is the proper relation between alignment quality and fusion complexity. The alignment study argues that when features are already well aligned, lightweight concatenation may dominate cross-attention, whereas weak alignment may justify heavier interaction (Zhou et al., 31 May 2026). Another is the balance between branch independence and fusion strength. SiMO shows that random modality drop is insufficient unless branches have already been trained to stand on their own (Wen et al., 9 Mar 2026), while PDF shows that collaborative weighting benefits from explicit modeling of own confidence and others’ weakness (Cao et al., 2024). A third is granularity: VGMR operates on summary-level support and discrepancy rather than local token alignment (Liu et al., 1 Jun 2026), whereas DEQ Fusion and VideoFusion model iterative or temporal interactions more explicitly (Ni et al., 2023, Tang et al., 30 Mar 2025). This suggests no single MCF granularity is universally optimal.

A plausible implication is that the field is moving toward alignment-aware, uncertainty-aware, and task-aware collaborative fusion rather than toward a single dominant fusion operator. Shared-latent anchoring, recursive refinement, pre-fusion calibration, cardinality-adaptive fusion, and collaborative decision weighting should therefore be viewed as complementary design axes within MCF, not mutually exclusive alternatives.