MODA: Modular Duplex Attention for MLLMs

• MODA is a modular duplex attention mechanism designed to tackle the cross-modal attention deficit by mitigating layer-wise decay and modality imbalance in MLLMs.
• It employs a 'correct-after-align' strategy that first aligns tokens into duplex modality spaces and then applies adaptive masked attention for correction.
• Empirical evaluations on 21 benchmarks show that MODA improves performance in OCR, visual-centric tasks, and multimodal reasoning while maintaining low computational overhead.

Searching arXiv for the MODA paper and closely related usages of the acronym. MOdular Duplex Attention (MODA) is an attention mechanism for multimodal LLMs that is designed to address instability in cross-modal attention and the layer-wise decay of attention activation in deep multimodal stacks. Introduced for multimodal perception, cognition, and emotion understanding, MODA decomposes attention into two coordinated processes—inner-modal refinement and inter-modal interaction—and organizes them through a “correct-after-align” strategy: tokens are first aligned into duplex modality spaces and then their attention flows are corrected by adaptive masked attention. In the reported formulation, MODA is inserted into Transformer attention blocks of multimodal models built on CLIP ViT-L/14 with Llama-3-Instruct-8B or Hermes2-Yi-34B, and it is evaluated on 21 benchmarks spanning perception, cognition, and emotion tasks (Zhang et al., 7 Jul 2025).

1. Problem setting and the “attention deficit disorder” diagnosis

The motivating claim behind MODA is that multimodal learning in contemporary MLLMs exhibits an “attention deficit disorder” (DDA) problem. In this diagnosis, cross-modal attention becomes inconsistent across depth, and attention activations decay layer by layer, producing an over-reliance on the dominant modality—typically text—and under-activation of fine-grained visual signals. The reported consequence is degraded performance on tasks that require detailed perception, higher-order multimodal cognition, or emotion understanding (Zhang et al., 7 Jul 2025).

The paper characterizes DDA through two coupled symptoms. The first is inconsistent cross-modal attention: lower layers exhibit a cross-modal focus that is not aligned with the attention distribution in deeper layers, even though deeper layers should possess stronger representational capability. The second is layer-by-layer decayed attention activation: both self-modal and cross-modal link values weaken as depth increases. The formalization given is that “the link value … decays exponentially with depth ($$\alpha_{v,t \rightarrow v}^l \propto \gamma^l,\gamma \neq 1$$) … the cumulative error … grows as”

$\mathbb{E}_{DDA} = \prod_l \gamma^l \epsilon_l.$

The empirical evidence reported for this diagnosis is quantitative as well as visual. In the motivation figure, the work reports a “63% disparity” and states that attention score disparity across modalities can reach “up to 10 times.” In a further attention analysis, baseline MLLMs show average self–cross disparity of 56.97% for text and 62.44% for visual tokens, whereas MODA reduces these values to 50.31% and 41.01%, respectively. This suggests that the mechanism is intended not merely to strengthen cross-modal exchange, but to rebalance attention statistics across modalities and layers.

A plausible implication is that MODA treats multimodal attention failure as a depth-dependent routing problem rather than only a data-alignment problem. On that interpretation, the architecture is not limited to improving cross-modal similarity; it also attempts to stabilize how that similarity is used across the hierarchy of Transformer blocks.

2. Core mechanism: duplex alignment and “correct-after-align”

MODA separates multimodal attention into two explicitly modularized components: inner-modal refinement, corresponding to self-modal attention, and inter-modal interaction, corresponding to cross-modal attention. The central design principle is the “correct-after-align” strategy. In this sequence, tokens are first aligned across modalities into duplex modality spaces, and only then are attention flows corrected with adaptive masks. The stated purpose is to decouple modality alignment from cross-layer token mixing and attention correction (Zhang et al., 7 Jul 2025).

The alignment phase is built from duplex $(V/T)$-Aligners. For a modality $m$ with $N_m$ tokens and key states $\bm{K}^m$, the paper defines a Gram matrix

$\bm{G}^m_{ij} = \sum_{k=1}^{N_m}{\bm{K}^m_{ik}\bm{K}^m_{kj}= {\bm{K}^m}^\top \bm{K}^m,$

and then uses the normed Gram matrix $||\bm{G}^m|| \in \mathbb{R}^{d \times d}$ to define modality bases. These basis vectors are intended to encode intra-modal relations compactly and to support transfer into the other modality’s representational geometry. The cross-modal mapping from the other modality $\bar m$ into modality $m$ is given as

$\mathbb{E}_{DDA} = \prod_l \gamma^l \epsilon_l.$0

As described, the aligned tokens are fused with the original ones through a lightweight fuser implemented with token merging and LoRA-based tuning.

The subsequent interaction phase computes self- and cross-modal attention using aligned keys and values, but it does so in a modularized rather than implicitly mixed manner. This differs from standard multi-head attention by inserting an explicit geometry-aware alignment operator before attention and by preserving a distinction between self-modal and cross-modal pathways. The final correction phase applies modular adaptive masks, which are intended to enforce per-modality priors and to prevent attention collapse.

The paper situates this pipeline as a departure from stock masked attention of the form

$\mathbb{E}_{DDA} = \prod_l \gamma^l \epsilon_l.$1

and notes that, in multimodal settings, attention is split into self- and cross-modal parts. Some printed expressions in the manuscript have missing brackets, but the intended construction is the familiar logits-plus-mask formulation. The important architectural distinction is that MODA first reshapes token geometry through duplex alignment and then corrects routing through modality-specific masking, rather than relying on a single unstructured softmax over mixed multimodal tokens.

3. Adaptive masked attention and modular correction

The correction stage is implemented through adaptive masked attention, described as the mechanism that ensures the correctness of attention scores. For each modality $\mathbb{E}_{DDA} = \prod_l \gamma^l \epsilon_l.$2, the mask is split into a self-modal component $\mathbb{E}_{DDA} = \prod_l \gamma^l \epsilon_l.$3 and a cross-modal component $\mathbb{E}_{DDA} = \prod_l \gamma^l \epsilon_l.$4. The resulting attentions are written as

$\mathbb{E}_{DDA} = \prod_l \gamma^l \epsilon_l.$5

As in the manuscript’s earlier equations, the notation is printed imperfectly, but the operational idea is explicit: self- and cross-modal paths receive separate masks and separate control.

A distinctive element of this masking scheme is pseudo-attention storage and decay. For each query row, only $\mathbb{E}_{DDA} = \prod_l \gamma^l \epsilon_l.$6 positions are allowed; the remainder are assigned pseudo-attention scores that decay with rate $\mathbb{E}_{DDA} = \prod_l \gamma^l \epsilon_l.$7. The matrix form is given as

$\mathbb{E}_{DDA} = \prod_l \gamma^l \epsilon_l.$8

Because $\mathbb{E}_{DDA} = \prod_l \gamma^l \epsilon_l.$9 and $(V/T)$0 are constructed separately, and may optionally be driven by $(V/T)$1 as an indicator, the mask can be customized by modality. The paper explicitly notes that different modalities may use different allowed lengths $(V/T)$2 or decay rates $(V/T)$3. In practical terms, stronger decay can be used when hallucinations dominate, whereas more permissive masking can be retained for tasks requiring broad cross-modal context.

The reported ablations assign a clear role to this masking stage. A mask-only model (“MDM”) improves over the baseline, but the strongest results arise when masking is combined with duplex alignment. In the ablation table, the baseline scores are G/K/O/V = 63.6/44.0/60.8/38.0, mask only yields 69.2/45.4/60.9/42.6, duplex alignment only yields 67.8/47.6/63.3/48.1, and the combined setting (“MDM + DAA”) reaches 69.3/48.3/67.0/54.3. The learnable or attention-conditioned mask (“Attn.”) is reported as the best masking variant, outperforming both “Inf” and “Fix.”

4. Integration into multimodal Transformer blocks and training configuration

MODA is integrated directly into each Transformer attention block of an MLLM. The reported sequence within a block is: compute modality-specific Gram bases $(V/T)$4; align cross-modal keys and values via duplex mapping; fuse aligned and original tokens through token merging and LoRA; compute self- and cross-modal attention with modality-specific masks; concatenate or fuse the outputs; then retain the standard residual, LayerNorm, and FFN design unchanged (Zhang et al., 7 Jul 2025).

The block-level scheduling is therefore not a replacement of the Transformer scaffold but a specialization of the attention sublayer. Both inner-modal refinement and inter-modal interaction occur in every attention block. The repeated use of alignment before token mixing is presented as the mechanism through which MODA mitigates the exponential decay of cross-modal links and reduces the cumulative error $(V/T)$5 across depth.

The training setup follows standard instruction-tuning practice for MLLMs. No additional training losses are introduced specifically for alignment or correction; the objective remains standard next-token prediction with masked attention. The reported visual encoder is CLIP ViT-L/14, and the base LLMs are Llama-3-Instruct-8B and Hermes2-Yi-34B. Optimization uses AdamW with a cosine schedule, for 1 epoch, with batch size 2048, learning rates 2e-5 (LLM) and 2e-6 (vision encoder), and warmup 0.03. For ablations at 8B scale, the models are trained on 700K samples for 1 epoch.

The implementation is described as computationally modest. The alignment stage adds the computation of $(V/T)$6, a single cross-modal mapping $(V/T)$7, and the modular masks, while token merging and LoRA in the fuser are used to keep overhead low. The paper further states that memory footprint remains compatible with common 8B–34B MLLM training settings under the reported training regime.

5. Empirical performance across perception, cognition, and emotion

The empirical evaluation spans 21 datasets covering perception, cognition, and emotion. The perception category includes 16 datasets: MME, MMBench, SEED, GQA, ScienceQA, MMMU, MathVista, AI2D, ChartQA, OCRBench, TextVQA, DocVQA, MMVP, RealworldQA, and CV-Bench 2D/3D. Cognition is evaluated on MMRole across eight dimensions: instruction adherence, fluency, coherency, image-text relevance, response accuracy, personality consistency, knowledge consistency, and tone consistency. Emotion benchmarks include MVSA-S, MVSA-M, TumEmo, and HFM, covering polarity, multi-class basic emotions, and sarcasm (Zhang et al., 7 Jul 2025).

The reported perception results distinguish 8B and 34B settings. At 8B scale with the Llama-3-Ins-8B backbone, MODA-8B achieves 72.1 on General, 61.5 on Knowledge, 72.0 on OCR+Chart, and 66.0 on Vision-centric. These figures are close to or above strong baselines in the specialized categories, with the text noting especially strong gains on fine-grained OCR and visual-centric tasks. At 34B scale with the Hermes2-Yi-34B backbone, MODA-34B obtains 76.7 on General, 69.5 on Knowledge, 74.7 on OCR+Chart, and 69.9 on Vision-centric. The larger model remains approximately tied on General averages while improving on Knowledge, OCR+Chart, and Vision-centric categories relative to the compared baselines.

For cognition, the MMRole results show that MODA-8B reaches an average of 0.972 in the open-ended 8B setting, compared with 0.968 for LLaVA-NeXT-8B. Under cognition-specialized tuning, MODA-8B achieves 0.995–1.000 and remains competitive with MMRole-9B variants, which the paper interprets as evidence of benefits for instruction adherence, fluency, and personality or tone consistency.

For emotion understanding, in the open-ended 8B setting MODA-8B reports an average of 0.588, compared with 0.576 for LLaVA-NeXT-8B and 0.547 for Cambrian-1-8B. Specific reported values include MVSA-S ACC 0.702 and HFM F1 0.563. In an emotion-specialized setting, MODA achieves an average of 0.841, with examples such as HFM ACC 0.885 and F1 0.881, remaining competitive with specialized methods such as SPFVTE and MULSER.

The ablation study supports the interpretation that MODA’s two principal components are complementary rather than redundant. “+CoV” is reported as the best attention alignment variant, concatenation (“Con”) is the best fusion strategy, and learnable or attention-conditioned masking performs best among masking alternatives. A plausible implication is that the architecture depends on the interaction between alignment geometry and routing correction, not only on either ingredient in isolation.

6. Relation to prior attention designs and acronym ambiguity

MODA is positioned against several classes of prior multimodal attention mechanisms. The paper argues that language-centric tuning often leaves visual activation underpowered on fine-grained tasks. It further states that co-attention and Flamingo-style cross-attention do not explicitly decouple alignment from attention correction and do not introduce per-modality adaptive masks. On this account, MODA’s novelty lies in combining duplex alignment with modular masks so as to address the specific DDA failure mode identified in the analysis (Zhang et al., 7 Jul 2025).

Within that framing, MODA is best understood as a multimodal attention-block modification rather than as a general-purpose replacement for all attention architectures. Its intended operating regime is multimodal token processing in MLLMs, especially where stable cross-modal exchange across depth matters for OCR-heavy, vision-centric, cognition-oriented, or emotion-sensitive workloads.

A separate point of terminology is important because the acronym is not unique in the literature. The label MoDA is also used for “Mixture-of-Depths Attention”, a depth-aware mechanism for decoder-only LLMs that augments sequence attention with retrieval over key-value pairs from preceding layers at the same token position (Zhu et al., 16 Mar 2026). Despite the near-identical acronym, that method addresses depth scaling and signal degradation in deep LLMs rather than multimodal alignment. The term MODA has also been used in the BRIMs literature for a duplex attention mechanism that combines bottom-up and top-down signals over recurrent modules, with modular sparsity regulating communication (Mittal et al., 2020). These usages share a concern with structured routing, but they differ in architecture, task domain, and formal mechanism.

This nomenclatural overlap can lead to misidentification in bibliographic searches. In current usage, MOdular Duplex Attention refers specifically to the multimodal method in (Zhang et al., 7 Jul 2025), whereas Mixture-of-Depths Attention and the BRIMs duplex-attention mechanism are separate lines of work.

7. Practical considerations, limitations, and future directions

The paper presents MODA as a practical drop-in modification for standard MLLM backbones. The recommended insertion point is inside each attention block: align with $(V/T)$8, apply modality-specific masked attention, keep residual and FFN structure unchanged, prefer concatenation for fusing self- and cross-modal outputs, and use learnable or attention-conditioned masks. For long sequences or many visual patches, the guidance is to restrict $(V/T)$9 and tune $m$0 so as to prevent attention collapse while preserving important local neighborhoods (Zhang et al., 7 Jul 2025).

The reported complexity characterization emphasizes linear complexity in token number because “the matrix sum among tokens is only conducted in the first round,” with practical costs controlled by token merging and LoRA in the fuser. The design is reported to scale consistently from 8B to 34B, with especially strong improvements in OCR and vision-centric categories at larger scale.

The limitations stated in the paper center on mask sensitivity and basis quality. If $m$1 is too large or $m$2 too small, needed cross-modal context may be suppressed; if masks are too loose, collapse or hallucination may reappear. Likewise, Gram-based alignment assumes that intra-modal geometry is informative; when a modality is noisy or poorly encoded, $m$3 may provide weak guidance for alignment. The paper also notes that gains may be marginal on general, language-heavy benchmarks where vision is less consequential.

The future directions proposed are to explore richer basis learning beyond Gram matrices, including low-rank adaptive subspaces or spectral filtering; to learn mask generators jointly with task prompts or curriculum schedules; and to extend duplex alignment beyond image–text settings to audio and other modalities, including long-context video streams with temporal-aware masks. This suggests that MODA is best viewed as a multimodal routing framework with one concrete instantiation, rather than as a finalized endpoint for multimodal attention design.