MF-Decoder: Adaptive Fusion for Vision-Language

• MF-Decoder is a decoder-only multimodal module that fuses visual and textual data via masked self-attention and adaptive co-attention.
• It employs a Vision-Token Adapter to convert images into pseudo-text tokens, enabling efficient cross-modal integration.
• Empirical results demonstrate improved accuracy, faster inference, and reduced training time compared to conventional encoder-based models.

Searching arXiv for the cited MUDAIF paper and closely related decoder-only vision-language work. Modality-Adaptive Fusion Decoder (MF-Decoder) denotes the multimodal decoder stack at the core of MUDAIF, a decoder-only vision-LLM introduced in “Optimizing Vision-Language Interactions Through Decoder-Only Models” (Tanaka et al., 2024). In that formulation, MF-Decoder operates downstream of a Vision-Token Adapter (VTA) that converts raw pixels into “pseudo-text” tokens, and it fuses visual and textual representations through masked self-attention over a concatenated multimodal sequence together with adaptive co-attention. The architecture is presented as an alternative to conventional systems that rely on separate visual encoders, with the stated goals of enhanced efficiency, flexibility, and cross-modal understanding (Tanaka et al., 2024).

1. Position within MUDAIF

MUDAIF’s decoder-only backbone is organized into three stages: the Vision-Token Adapter (VTA), the Multimodal Decoder Stack identified as the MF-Decoder, and the Output Generation Head (Tanaka et al., 2024). The model accepts an image $I \in \mathbb{R}^{H \times W \times 3}$ and a text token sequence $T = \{t_1, \ldots, t_L\}$ with embedding dimension $d$ (Tanaka et al., 2024).

Within this decomposition, the VTA converts raw pixels into $N$ pseudo-text tokens $V \in \mathbb{R}^{N \times d}$ by means of a small convolutional feature extractor, a linear projection, and an optional intra-visual self-attention refinement (Tanaka et al., 2024). The MF-Decoder then processes both visual and textual streams jointly. After the final decoder layer, only the text-token outputs are fed into a linear plus softmax head for next-token prediction, whereas the visual stream is not directly decoded and instead influences generation through cross-attention throughout the stack (Tanaka et al., 2024).

This organization places MF-Decoder at the center of MUDAIF’s multimodal integration strategy. A plausible implication is that the decoder stack, rather than a separate encoder module, bears primary responsibility for both sequence modeling and cross-modal alignment.

2. Architectural organization of the decoder stack

The MF-Decoder is described as a stack of $L$ identical layers (Tanaka et al., 2024). At layer $\ell$, each token representation, both visual and textual, is updated by three sub-layers: masked self-attention over the concatenated sequence $[T;V]$, adaptive co-attention between vision and text, and a position-wise feed-forward network (Tanaka et al., 2024).

The masked self-attention stage is specified as autoregressive on text and full attend on vision (Tanaka et al., 2024). This design couples language-generation causality with unrestricted access to visual tokens. After self-attention, the visual stream $V_v^{(\ell)}$ and text stream $T_t^{(\ell)}$ cross-interact via bidirectional co-attention (Tanaka et al., 2024). The resulting structure differs from encoder-decoder factorization in that both modalities remain active inside the same decoder stack at every depth.

The output pathway is likewise asymmetric in a precise sense. Text-token states at the final layer are used for token generation, while visual states remain latent conditioning signals rather than direct outputs (Tanaka et al., 2024). This arrangement suggests a generative model in which visual information is progressively injected into an autoregressive textual backbone rather than encoded once and consumed passively.

3. Mathematical formulation

At layer $T = \{t_1, \ldots, t_L\}$0, the VTA projection is written as

$T = \{t_1, \ldots, t_L\}$1

with an optional refinement by one round of intra-visual self-attention (Tanaka et al., 2024). The detailed VTA construction is also given as

$T = \{t_1, \ldots, t_L\}$2

$T = \{t_1, \ldots, t_L\}$3

followed optionally by

$T = \{t_1, \ldots, t_L\}$4

$T = \{t_1, \ldots, t_L\}$5

(Tanaka et al., 2024).

For self-attention in decoder layer $T = \{t_1, \ldots, t_L\}$6, the multimodal input is defined as

$T = \{t_1, \ldots, t_L\}$7

with

$T = \{t_1, \ldots, t_L\}$8

and

$T = \{t_1, \ldots, t_L\}$9

(Tanaka et al., 2024).

The defining multimodal operation is adaptive co-attention. Two cross-modal attention maps are computed at layer $d$0 (Tanaka et al., 2024). For visual-to-text attention,

$d$1

$d$2

For text-to-visual attention,

$d$3

$d$4

The fused updates are then

$d$5

$d$6

followed by residual normalization:

$d$7

$d$8

(Tanaka et al., 2024).

Each stream then passes through the usual two-layer MLP with GeLU and residual,

$d$9

$N$0

(Tanaka et al., 2024). For generation, MUDAIF factorizes

$N$1

with

$N$2

(Tanaka et al., 2024).

4. Training regime and optimization

The reported pretraining dataset consists of 45 M image-text pairs drawn from COCO, LAION, and Visual Genome (Tanaka et al., 2024). Instruction-tuning uses a mixture of multimodal instruction datasets, specifically VQA dialogues, captioning prompts, and reasoning instructions (Tanaka et al., 2024).

The loss functions are given explicitly. Pretraining uses the language-modeling objective

$N$3

while task-specific fine-tuning uses

$N$4

(Tanaka et al., 2024). The combined objective is

$N$5

with $N$6 tuned on hold-out (Tanaka et al., 2024).

The optimization recipe is reported as AdamW with weight decay $N$7, linear learning-rate warmup for the first $N$8 of steps, peak learning rate around $N$9, cosine decay thereafter, and batch size $V \in \mathbb{R}^{N \times d}$0 images on 64 A100 GPUs (Tanaka et al., 2024). No additional modality-specific regularizers beyond dropout $V \in \mathbb{R}^{N \times d}$1 in each sub-layer are used (Tanaka et al., 2024).

These details characterize MF-Decoder not as an isolated module but as part of a large-scale pretrain-and-instruction-tune pipeline. A plausible implication is that the decoder’s fusion behavior is learned primarily through standard autoregressive and instruction-following objectives rather than bespoke alignment losses.

5. Empirical results and ablations

MUDAIF is compared against InstructBLIP, LLaVA-1.5, and EVE-7B on VQA-v2, GQA, VizWiz, captioning BLEU, SEED, and MM-Vet (Tanaka et al., 2024). The reported results are as follows:

Model	VQA-v2 / GQA / VizWiz	BLEU / SEED / MM-Vet
InstructBLIP	78.5 / 62.0 / 50.0	0.72 / 58.6 / 30.5
LLaVA-1.5	78.7 / 63.2 / 51.1	0.74 / 60.4 / 31.0
EVE-7B	76.4 / 60.8 / 41.8	0.69 / 54.3 / 25.6
MUDAIF (Ours)	80.3 / 65.5 / 53.7	0.78 / 62.8 / 33.4

The ablation study on VQA-v2 and GQA isolates several architectural components (Tanaka et al., 2024). Removing the Vision-Token Adapter yields $V \in \mathbb{R}^{N \times d}$2, removing Adaptive Co-Attention yields $V \in \mathbb{R}^{N \times d}$3, and using a simplified decoder with no cross-modality yields $V \in \mathbb{R}^{N \times d}$4 (Tanaka et al., 2024). Within the paper’s experimental framing, these ablations assign a direct empirical role to the adaptive fusion machinery.

Robustness and generalization are reported separately. Under input variations such as low or high resolution and noisy inputs, MUDAIF maintains $V \in \mathbb{R}^{N \times d}$5–$V \in \mathbb{R}^{N \times d}$6 accuracy versus $V \in \mathbb{R}^{N \times d}$7–$V \in \mathbb{R}^{N \times d}$8 for baselines (Tanaka et al., 2024). Zero-shot generalization on unseen tasks is reported as $V \in \mathbb{R}^{N \times d}$9 versus nearest $L$0, and few-shot performance with fewer than 10 examples is reported as $L$1 versus $L$2 (Tanaka et al., 2024).

6. Efficiency, alignment, and limitations

The reported efficiency claim is that eliminating a heavyweight visual encoder yields a $L$3 reduction in training time and $L$4 faster inference on high-resolution images (Tanaka et al., 2024). The paper also states that the decoder-only design supports arbitrary image resolutions and aspect ratios, since the VTA simply slides over raw pixels (Tanaka et al., 2024). In this framing, MF-Decoder participates in an encoder-free architecture whose computational profile differs from conventional vision-language pipelines.

For cross-modal alignment, the paper reports cosine similarity between vision- and text-embeddings during pretraining of $L$5 versus $L$6 for encoder-based approaches (Tanaka et al., 2024). This is presented as evidence of stronger alignment. A plausible implication is that repeated bidirectional co-attention inside the decoder stack promotes tighter representational coupling than one-shot fusion after a frozen or decoupled visual encoder.

The limitations and future directions are also stated directly. The current VTA uses fixed conv+linear layers, and future work could explore dynamic patch sizes or transformer-based vision front-ends (Tanaka et al., 2024). The paper further proposes extending MF-Decoder to video by adding a temporal fusion adapter and suggests that incorporating a small gating network $L$7 inside the co-attention denominator, in the style of learnable fusion biases, could allow finer-grained modality weighting (Tanaka et al., 2024). These proposals indicate that the “modality-adaptive” designation refers to adaptive cross-modal interaction within the decoder, but not yet to an explicitly gated or uncertainty-aware scheduler of the kind explored in separate adaptive-fusion research directions.

7. Relation to adjacent adaptive-fusion work

MF-Decoder should be distinguished from the framework introduced in “Learning to Fuse: Modality-Aware Adaptive Scheduling for Robust Multimodal Foundation Models” (Bennett et al., 15 Jun 2025). That work proposes Modality-Aware Adaptive Fusion Scheduling (MA-AFS), a framework that learns to dynamically modulate the contribution of each modality on a per-instance basis using a lightweight neural scheduler, visual and textual entropy signals, and cross-modal agreement cues (Bennett et al., 15 Jun 2025). The provided record for that paper explicitly notes that there is no discussion of an MF-Decoder in the supplied details, and it does not provide the scheduler architecture, entropy formulas, fusion algorithms, theoretical propositions, or empirical results for direct comparison.

The distinction is conceptually important. MF-Decoder in MUDAIF is defined through a decoder-only architecture with bidirectional adaptive co-attention and pseudo-tokenized visual inputs (Tanaka et al., 2024). MA-AFS, by contrast, is framed around dynamic fusion scheduling under noisy, missing, or misaligned inputs (Bennett et al., 15 Jun 2025). This suggests that “adaptive fusion” can refer either to layerwise co-attention within a unified decoder, as in MUDAIF, or to per-instance modality weighting via an explicit scheduler, as in MA-AFS. Conflating the two would obscure the architectural specificity of MF-Decoder.