--- title: 'MF-Decoder: Adaptive Fusion for Vision-Language' url: https://www.emergentmind.com/topics/modality-adaptive-fusion-decoder-mf-decoder type: topic --- # MF-Decoder: Adaptive Fusion for Vision-Language Searching arXiv for the cited MUDAIF paper and closely related decoder-only vision-language work. Modality-Adaptive Fusion Decoder (MF-Decoder) denotes the multimodal decoder stack at the core of MUDAIF, a decoder-only vision-language model introduced in “Optimizing Vision-Language Interactions Through Decoder-Only Models” [2412.10758]. In that formulation, MF-Decoder operates downstream of a Vision-Token Adapter (VTA) that converts raw pixels into “pseudo-text” tokens, and it fuses visual and textual representations through masked self-attention over a concatenated multimodal sequence together with adaptive co-attention. The architecture is presented as an alternative to conventional systems that rely on separate visual encoders, with the stated goals of enhanced efficiency, flexibility, and cross-modal understanding [2412.10758]. ## 1. Position within MUDAIF MUDAIF’s decoder-only backbone is organized into three stages: the Vision-Token Adapter (VTA), the Multimodal Decoder Stack identified as the MF-Decoder, and the Output Generation Head [2412.10758]. The model accepts an image $I \in \mathbb{R}^{H \times W \times 3}$ and a text token sequence $T = \{t_1, \ldots, t_L\}$ with embedding dimension $d$ [2412.10758]. Within this decomposition, the VTA converts raw pixels into $N$ pseudo-text tokens $V \in \mathbb{R}^{N \times d}$ by means of a small convolutional feature extractor, a linear projection, and an optional intra-visual self-attention refinement [2412.10758]. The MF-Decoder then processes both visual and textual streams jointly. After the final decoder layer, only the text-token outputs are fed into a linear plus softmax head for next-token prediction, whereas the visual stream is not directly decoded and instead influences generation through cross-attention throughout the stack [2412.10758]. This organization places MF-Decoder at the center of MUDAIF’s multimodal integration strategy. A plausible implication is that the decoder stack, rather than a separate encoder module, bears primary responsibility for both sequence modeling and cross-modal alignment. ## 2. Architectural organization of the decoder stack The MF-Decoder is described as a stack of $L$ identical layers [2412.10758]. At layer $\ell$, each token representation, both visual and textual, is updated by three sub-layers: masked self-attention over the concatenated sequence $[T;V]$, adaptive co-attention between vision and text, and a position-wise feed-forward network [2412.10758]. The masked self-attention stage is specified as autoregressive on text and full attend on vision [2412.10758]. This design couples language-generation causality with unrestricted access to visual tokens. After self-attention, the visual stream $V_v^{(\ell)}$ and text stream $T_t^{(\ell)}$ cross-interact via bidirectional co-attention [2412.10758]. The resulting structure differs from encoder-decoder factorization in that both modalities remain active inside the same decoder stack at every depth. The output pathway is likewise asymmetric in a precise sense. Text-token states at the final layer are used for token generation, while visual states remain latent conditioning signals rather than direct outputs [2412.10758]. This arrangement suggests a generative model in which visual information is progressively injected into an autoregressive textual backbone rather than encoded once and consumed passively. ## 3. Mathematical formulation At layer $0$, the VTA projection is written as $$ V^{(0)} = \mathrm{Linear}(\mathrm{Conv}(I)) \in \mathbb{R}^{N \times d}, $$ with an optional refinement by one round of intra-visual self-attention [2412.10758]. The detailed VTA construction is also given as $$ F = \mathrm{Conv}(I), $$ $$ V^{(0)} = W_{\mathrm{proj}} \cdot F + b_{\mathrm{proj}}, $$ followed optionally by $$ Q_v = V^{(0)}W_q^v,\quad K_v = V^{(0)}W_k^v,\quad V' = V^{(0)}W_v^v, $$ $$ V = \mathrm{Softmax}\!\left(\frac{Q_v K_v^T}{\sqrt{d}}\right)V' $$ [2412.10758]. For self-attention in decoder layer $\ell$, the multimodal input is defined as $$ X^{(\ell-1)} = [T^{(\ell-1)};V^{(\ell-1)}] \in \mathbb{R}^{(L+N)\times d}, $$ with $$ Q = X^{(\ell-1)}W_q,\quad K = X^{(\ell-1)}W_k,\quad \hat{V} = X^{(\ell-1)}W_v, $$ and $$ \mathrm{SelfAttn}(X)=\mathrm{Softmax}\!\left(\frac{QK^T}{\sqrt{d}}\right)\hat{V} $$ [2412.10758]. The defining multimodal operation is adaptive co-attention. Two cross-modal attention maps are computed at layer $\ell$ [2412.10758]. For visual-to-text attention, $$ Q_v = V^{(\ell)} W_q^v,\quad K_t = T^{(\ell)} W_k^t,\quad V_t = T^{(\ell)} W_v^t, $$ $$ A_{v\to t} = \mathrm{Softmax}\!\left(\frac{Q_v K_t^T}{\sqrt{d}}\right)\in \mathbb{R}^{N\times L}. $$ For text-to-visual attention, $$ Q_t = T^{(\ell)} W_q^t,\quad K_v = V^{(\ell)} W_k^v,\quad V_v = V^{(\ell)} W_v^v, $$ $$ A_{t\to v} = \mathrm{Softmax}\!\left(\frac{Q_t K_v^T}{\sqrt{d}}\right)\in \mathbb{R}^{L\times N}. $$ The fused updates are then $$ T_{\mathrm{update}} = A_{t\to v}V_v \in \mathbb{R}^{L\times d}, $$ $$ V_{\mathrm{update}} = A_{v\to t}V_t \in \mathbb{R}^{N\times d}, $$ followed by residual normalization: $$ T^{(\ell+1)} = \mathrm{LayerNorm}(T^{(\ell)} + T_{\mathrm{update}}), $$ $$ V^{(\ell+1)} = \mathrm{LayerNorm}(V^{(\ell)} + V_{\mathrm{update}}) $$ [2412.10758]. Each stream then passes through the usual two-layer MLP with GeLU and residual, $$ \mathrm{FFN}(x) = W_2(\mathrm{GeLU}(W_1x+b_1))+b_2, $$ $$ x_{\mathrm{out}} = \mathrm{LayerNorm}(x+\mathrm{FFN}(x)) $$ [2412.10758]. For generation, MUDAIF factorizes $$ p(t_1\ldots t_L \mid I)=\prod_{i=1}^{L} p(t_i \mid t_{