Multimodal Fusion Decoder (MFD)

• MFD is a decoder mechanism that fuses latent features from various modalities into a single representation for tasks such as reconstruction, segmentation, and detection.
• It employs strategies like VAE-based reconstruction, co-attention, and modality-specific refinement to align heterogeneous data effectively.
• Empirical studies show that MFDs improve accuracy and efficiency across applications like medical imaging, ASR, and 3D detection by optimizing fusion fidelity.

Searching arXiv for the cited papers and closely related multimodal fusion decoder work. Multimodal Fusion Decoder (MFD) denotes a decoder-side mechanism that integrates information from multiple modalities into a fused latent state, a reconstruction target, or a task-specific prediction. In the surveyed literature, the term is explicit in some systems and only implicit in others. Explicit instances include the multimodal fusion decoder in federated brain-tumor segmentation and the Multimodal Fusion Decoder in multimodal speech recognition, while closely related analogues appear as a VAE reconstruction decoder, a refiner/defuser module, a Transformer reconstruction decoder, a decoder-only co-attentive vision-LLM, and proposal-refinement decoders for 3D detection (Liu et al., 5 Mar 2026, Chen et al., 26 Sep 2025, Majumder et al., 2019, Sankaran et al., 2021, Yuan et al., 2022, Tanaka et al., 2024, Palladin et al., 22 Aug 2025). Across these formulations, the common role of an MFD is to make fused representations useful not only for prediction, but also for preserving modality-specific evidence, aligning heterogeneous feature spaces, or reconstructing outputs under auxiliary constraints.

1. Definition and scope

In its narrowest sense, an MFD is a decoder that receives modality-specific latent features and produces a fused output. This description matches the “multimodal fusion decoder” in FedMEPD, where modality-specific encoders $E_m$ feed a fusion decoder $D_M$ on the server and partially personalized decoders $D_m$ on clients for brain-tumor segmentation (Liu et al., 5 Mar 2026). It also matches Index-MSR, whose Multimodal Fusion Decoder attends separately to speech encoder outputs and OCR-derived visual-text embeddings before combining them for autoregressive ASR decoding (Chen et al., 26 Sep 2025).

A broader and increasingly common usage includes decoder-like modules that refine, reconstruct, or “defuse” a shared multimodal representation rather than directly synthesizing a raw output. In “Variational Fusion for Multimodal Sentiment Analysis,” the decoder reconstructs the concatenated multimodal input $F=f_t\oplus f_a\oplus f_v$ from a shared latent variable $z$, thereby enforcing fidelity between the fused latent and the original modality features (Majumder et al., 2019). In “Multimodal Fusion Refiner Networks,” the refiner decomposes the fused embedding $F_{\text{emb}}$ into modality-specific outputs $R_i=\mathcal{D}_i(F_{\text{emb}})$, optimized against modality-aligned targets with cosine similarity (Sankaran et al., 2021). In “Multimodal Image Fusion based on Hybrid CNN-Transformer and Non-local Cross-modal Attention,” the decoder reconstructs the final fused image from an already fused latent $\Phi^f$, making it a post-fusion refinement decoder rather than the primary site of raw-modality interaction (Yuan et al., 2022).

A persistent source of ambiguity is that not every architecture called “decoder” performs the same function. Some MFDs reconstruct concatenated modality features; some decode modality-specific embeddings; some refine proposal queries in BEV; some output segmentation masks or fused images; and some, such as MUDAIF, make the decoder itself the main locus of multimodal interaction via co-attention inside a decoder-only transformer (Tanaka et al., 2024). This suggests that MFD is best treated as an architectural family defined by decoder-side multimodal integration rather than by a single canonical layer pattern.

2. Canonical architectural pattern

Despite wide variation, most MFD-style systems share a recurring pipeline. First, modality-specific encoders or feature extractors produce latent representations. Second, these representations are either mapped into a shared latent space or aligned into a common geometric or token space. Third, a decoder or decoder-like module integrates those representations into a fused state, a reconstruction, or a downstream prediction. Finally, auxiliary reconstruction, similarity, or task losses shape the decoder’s behavior.

A minimal latent-space version appears in the VAE-based sentiment model. The input concatenation

$F=f_t\oplus f_a\oplus f_v$

is encoded into a Gaussian latent variable $z$, and a fully connected decoder reconstructs

$D_M$0

The latent $D_M$1 is then used for sentiment or emotion classification, while the decoder acts primarily as training-time regularization (Majumder et al., 2019). In this formulation, fusion is latent-variable fusion, but the decoder imposes the condition that the fused latent must remain reconstructive.

A modality-branch version appears in ReFNet. A generic fusion module $D_M$2 produces a fused embedding $D_M$3, and modality-specific decoders $D_M$4 produce outputs $D_M$5 that are matched to modality features $D_M$6. The decoder is therefore not generative in the raw-input sense; it is a self-supervised defusing mechanism that forces the fused space to retain unimodal content (Sankaran et al., 2021).

A dense-prediction version appears in FedMEPD. There, each MRI modality has its own encoder, while the fusion decoder aggregates multiscale features and predicts segmentation masks. The server uses a full-modal decoder $D_M$7, clients use local decoders $D_M$8, and partial personalization is applied at the filter level through

$D_M$9

This places the decoder at the center of multimodal aggregation, personalization, and hetero-modal adaptation (Liu et al., 5 Mar 2026).

A decoder-only token-space version appears in MUDAIF. Images are transformed by the Vision-Token Adapter into pseudo-textual tokens $D_m$0, text is represented as $D_m$1, and multimodal fusion happens inside a shared decoder through bidirectional co-attention: $D_m$2

$D_m$3

Here the MFD is not appended after fusion; it is the multimodal computation itself (Tanaka et al., 2024).

3. Principal design variants

The literature contains several distinct MFD design families.

Variant	Decoder target	Representative papers
Reconstruction-constrained latent decoder	Concatenated multimodal input or fused image	(Majumder et al., 2019, Yuan et al., 2022, He et al., 18 Jun 2025)
Refiner or defuser decoder	Modality-specific feature embeddings	(Sankaran et al., 2021)
Segmentation fusion decoder	Dense volumetric segmentation logits	(Liu et al., 5 Mar 2026, Shen et al., 19 Jan 2025)
Query or proposal refinement decoder	Multimodal proposals or detection queries	(Palladin et al., 22 Aug 2025, Zhang et al., 2024)
Decoder-only multimodal decoder	Shared token-space latent for generation	(Tanaka et al., 2024)
Dual-memory ASR decoder	Token predictions conditioned on speech and OCR text	(Chen et al., 26 Sep 2025)

The reconstruction-constrained family includes VAE fusion, CNN-Transformer image fusion, diffusion-trained encoder-decoder fusion, and flow-matching fusion. The key property is that the decoder is supervised by reconstructive criteria, either on concatenated modality features, on fused images, or on vector-field targets for transport from source modalities to a fused image distribution (Majumder et al., 2019, Yuan et al., 2022, He et al., 18 Jun 2025, Zhu et al., 17 Nov 2025).

The refiner family emphasizes modality-centric responsibility. ReFNet decodes $D_m$4 into modality-specific targets and uses

$D_m$5

to ensure that the fused latent preserves each modality strongly enough to be recoverable (Sankaran et al., 2021).

The segmentation family splits into two subtypes. FedMEPD uses a late-fusion segmentation decoder over modality-specific MRI encoders and extends it with filter-level partial personalization and anchor-guided cross-attention calibration for missing modalities (Liu et al., 5 Mar 2026). EFNet, by contrast, is explicitly not decoder-centric in its multimodal fusion; most RGB-T fusion happens early via MIF, while its MFAD is a lightweight multi-scale decoder using Euclidean distance to semantic class tokens: $D_m$6 This makes EFNet an important counterexample: not every multimodal system with a decoder is an MFD in the strong sense (Shen et al., 19 Jan 2025).

The proposal-refinement family includes SAMFusion and E2E-MFD. SAMFusion first builds multimodal BEV proposals and then refines them with a transformer decoder; the decoder is “sensor-adaptive” because proposal generation already incorporates learned distance-dependent LiDAR/radar blending and gated-camera BEV conditioning (Palladin et al., 22 Aug 2025). E2E-MFD treats the fusion branch as a decoder-like pathway inside an end-to-end fusion-detection system, synchronizing image fusion and diffusion-based object detection in a single training phase (Zhang et al., 2024).

4. Objectives and optimization

MFDs are typically shaped by multi-term objectives that combine task supervision with explicit constraints on multimodal fidelity.

The VAE-based latent decoder uses the ELBO

$D_m$7

together with classification loss. The reconstruction term is the operative fusion-fidelity mechanism, because reconstructing $D_m$8 requires the latent $D_m$9 to preserve recoverable information from all three modalities (Majumder et al., 2019).

ReFNet augments the downstream loss with modality-specific cosine reconstruction and an optional Multi-Similarity contrastive term: $F=f_t\oplus f_a\oplus f_v$0 This makes the decoder a self-supervised regularizer and enables unlabeled pretraining through the refiner loss alone (Sankaran et al., 2021).

FedMEPD uses the commonly used Dice loss plus cross entropy for segmentation, while decoder personalization is driven not by an extra loss but by consistency between server and client filter updates. The personalization mask is updated from cosine similarities between server and client decoder-filter updates over multiple rounds, with a persistence rule once a filter becomes personalized (Liu et al., 5 Mar 2026).

Image-fusion MFDs often use reconstruction-centric objectives. DM-FNet uses a hybrid Stage II loss

$F=f_t\oplus f_a\oplus f_v$1

with $F=f_t\oplus f_a\oplus f_v$2 and $F=f_t\oplus f_a\oplus f_v$3, to balance brightness, structure, and gradient detail in the fused image (He et al., 18 Jun 2025). LKC-FUNet supervises its decoder with $F=f_t\oplus f_a\oplus f_v$4, while its MPAFM recalibrates decoder inputs via channel attention on encoder skips, spatial attention on decoder features, and bidirectional interaction: $F=f_t\oplus f_a\oplus f_v$5 This directly targets decoder-stage feature transmission (He et al., 2024).

Sequence-generation MFDs typically use autoregressive token losses. MUDAIF trains with multimodal next-token prediction conditioned on image-derived pseudo-text tokens and prompts (Tanaka et al., 2024). Index-MSR combines a CTC-regularized speech encoder with a dual cross-attention decoder that fuses speech and OCR features by

$F=f_t\oplus f_a\oplus f_v$6

after which feed-forward and softmax prediction proceed in standard AED style (Chen et al., 26 Sep 2025).

Other works contribute auxiliary optimization principles rather than decoder blocks. Neural dependency coding introduces KL- or MMD-based synergy regularization for fused multimodal representations, which can be attached to a decoder’s hidden multimodal state (Shankar, 2021). MRRF factorizes output-conditioned multimodal tensors with modality-specific Tucker ranks to suppress redundant modality content while preserving high-order interactions (Barezi et al., 2018). Deep Equilibrium Multimodal Fusion replaces an explicit stacked fusion decoder with a fixed-point solve over recursive multimodal interactions, turning fusion into an implicit equilibrium computation (Ni et al., 2023).

5. Application domains and empirical record

The empirical record for MFD-style mechanisms spans sentiment analysis, VQA, multimodal image fusion, segmentation, 3D detection, speech recognition, and vision-language generation.

In multimodal sentiment analysis, reconstruction-constrained latent decoding improved both context-free and context-dependent classifiers. On MOSI, MOSEI, and IEMOCAP, VAE+bc-LSTM improved F1 from $F=f_t\oplus f_a\oplus f_v$7 to $F=f_t\oplus f_a\oplus f_v$8, from $F=f_t\oplus f_a\oplus f_v$9 to $z$0, and from $z$1 to $z$2, and the authors report that VAE+bc-LSTM outperforms state-of-the-art bc-LSTM by $z$3 on average (Majumder et al., 2019). In ReFNet, decoder-based self-supervision improved transformer baselines and was especially effective under label scarcity: on MM-IMDB with $z$4 labels, ReFNet improved micro-F1 by $z$5 and macro-F1 by $z$6, while ReFNet$z$7 raised these to $z$8 and $z$9 (Sankaran et al., 2021).

In federated multimodal segmentation, decoder design is central rather than auxiliary. With modality-specific encoders, a fully personalized decoder achieved average client mDSC $F_{\text{emb}}$0, a fully federated decoder $F_{\text{emb}}$1, and the proposed partially federated / partially personalized decoder $F_{\text{emb}}$2, rising to $F_{\text{emb}}$3 with LACCA and multi-anchor calibration (Liu et al., 5 Mar 2026). In RGB-T segmentation, EFNet’s lightweight MFAD improved over both DeepLab and MLP decoders on MFNet, achieving $F_{\text{emb}}$4 mIoU versus $F_{\text{emb}}$5 and $F_{\text{emb}}$6, while keeping parameters and FLOPs low (Shen et al., 19 Jan 2025).

In multimodal medical image fusion, decoder quality is repeatedly shown to matter. DM-FNet reports that removing AMFF or MSFF degrades performance, and the full model is described as preserving appropriate brightness, comprehensive distribution of radioactive tracers, rich textures, and clear edges across MRI–CT, MRI–PET, and MRI–SPECT fusion (He et al., 18 Jun 2025). LKC-FUNet shows that removing MPAFM reduces SD from $F_{\text{emb}}$7 to $F_{\text{emb}}$8, AG from $F_{\text{emb}}$9 to $R_i=\mathcal{D}_i(F_{\text{emb}})$0, and SF from $R_i=\mathcal{D}_i(F_{\text{emb}})$1 to $R_i=\mathcal{D}_i(F_{\text{emb}})$2, supporting the claim that recalibrating decoder inputs improves transmission of key features (He et al., 2024). FusionFM extends this line by replacing diffusion-style sampling with one-shot source-to-fused transport; its runtime is $R_i=\mathcal{D}_i(F_{\text{emb}})$3 s at $R_i=\mathcal{D}_i(F_{\text{emb}})$4 and $R_i=\mathcal{D}_i(F_{\text{emb}})$5 s at $R_i=\mathcal{D}_i(F_{\text{emb}})$6, with $R_i=\mathcal{D}_i(F_{\text{emb}})$7 M parameters, while remaining competitive across IVF, MIF, MEF, and MFF (Zhu et al., 17 Nov 2025).

In ASR and 3D detection, decoder-side multimodal integration is tied to error correction and robustness. Index-MSR reduces substitution errors by about $R_i=\mathcal{D}_i(F_{\text{emb}})$8 on Chinese-LiPS and nearly $R_i=\mathcal{D}_i(F_{\text{emb}})$9 on its in-house subtitle dataset, with overall WER improving from $\Phi^f$0 to $\Phi^f$1 in the small setting and from $\Phi^f$2 to $\Phi^f$3 in the large setting (Chen et al., 26 Sep 2025). SAMFusion reports that its decoder-side proposal generation and refinement improve average precision by $\Phi^f$4 AP compared to the next best method for vulnerable pedestrians in long distances and challenging foggy scenes, and its multimodal proposals improve night-time pedestrian detection by $\Phi^f$5 over solely point-cloud-based proposals (Palladin et al., 22 Aug 2025). E2E-MFD reports a $\Phi^f$6 mAP50 increase on M3FD and a $\Phi^f$7 mAP50 increase on DroneVehicle, supporting the idea that synchronously optimized fusion-and-detection decoders are beneficial (Zhang et al., 2024).

In decoder-only vision-language modeling, MUDAIF reports higher benchmark scores than InstructBLIP, LLaVA-1.5, and EVE-7B, including VQA-v2 $\Phi^f$8, GQA $\Phi^f$9, VizWiz $F=f_t\oplus f_a\oplus f_v$0, BLEU $F=f_t\oplus f_a\oplus f_v$1, SEED $F=f_t\oplus f_a\oplus f_v$2, and MM-Vet $F=f_t\oplus f_a\oplus f_v$3. Its ablations further show that removing the Vision-Token Adapter or adaptive co-attention degrades VQA-v2 and GQA accuracy substantially (Tanaka et al., 2024).

6. Limitations, ambiguities, and open directions

The first limitation is terminological. MFD is not a standardized label across the literature. Several highly relevant papers do not use the term at all, even when they include a functionally equivalent module. This includes variational reconstruction decoders, refiners, post-fusion Transformer decoders, and adaptive skip-fusion decoders (Majumder et al., 2019, Sankaran et al., 2021, Yuan et al., 2022, He et al., 2024). A second ambiguity concerns the reconstruction target: some decoders reconstruct raw or fused images, some reconstruct concatenated modality features, and some reconstruct modality-specific embeddings rather than inputs. Consequently, “decoder” can denote generation, regularization, or latent defusion, and these should not be conflated.

A second major limitation is under-specification. Several papers state the role of the decoder clearly but omit exact layer inventories, token ordering, positional encoding treatment, masking schemes, or internal attention formulas. This is explicit in MUDAIF, in the hybrid CNN-Transformer image-fusion decoder, and in Index-MSR’s MFD, where the dual cross-attention pattern is described but many engineering details are absent (Tanaka et al., 2024, Yuan et al., 2022, Chen et al., 26 Sep 2025). DM-FNet likewise leaves some AMFF internals under-specified, and FedMEPD inherits RFNet decoder internals rather than restating them fully (He et al., 18 Jun 2025, Liu et al., 5 Mar 2026).

A third limitation is that decoder efficacy is often established only through quasi-ablation rather than direct isolation of a single fusion-decoder term. The variational sentiment model, for example, compares concatenation-based baselines with VAE-fused models but does not provide an explicit decoder-removal ablation (Majumder et al., 2019). More generally, some reported gains may arise from the joint effect of improved encoder representations, auxiliary losses, and decoder design rather than the decoder alone.

Open directions suggested by the surveyed work include more explicit reliability-aware decoding under modality corruption, better specification of decoder-only multimodal transformers, stronger missing-modality handling, and broader use of shared MFDs across heterogeneous tasks. FedMEPD shows one route through partially personalized decoders and anchor-based calibration (Liu et al., 5 Mar 2026); FusionFM shows another through a single decoder regularized by elastic weight consolidation and experience replay (Zhu et al., 17 Nov 2025). DEQ fusion suggests an additional direction in which the decoder need not be finite-depth at all, but may instead be an implicit equilibrium solver over multimodal latent states (Ni et al., 2023).

Taken together, the literature presents MFD not as a single architecture but as a general design principle: decoder-side multimodal integration can be used to preserve fidelity, recover modality-specific evidence, improve dense prediction, refine proposals, regularize latent spaces, or enable unified multimodal generation. The main technical distinction among MFDs is therefore not whether they decode, but what they decode, under which constraints, and for which downstream objective.