Attention Efficiency Index (AEI) in MLLMs

• Attention Efficiency Index (AEI) is a metric that quantifies a modality's effective attention allocation by comparing its attention mass to its token share.
• AEI isolates efficiency from dominance, diagnosing issues like attention dilution in modalities with severe token redundancy during MLLM generation.
• Complemented by the Modality Dominance Index, AEI offers actionable insights into architecture design and token compression for rebalancing attention across modalities.

Attention Efficiency Index (AEI) is an evaluation metric introduced for Multimodal LLMs (MLLMs) to quantify how effectively a modality converts its token budget into attention mass during generation. It was proposed in the context of text dominance, a core problem in which MLLMs depend heavily on text for their inference while underutilizing other modalities, including images, videos, audio, time-series, and graphs. AEI isolates the notion of efficiency from dominance: whereas the Modality Dominance Index (MDI) tells which modality wins on a per-token basis, AEI asks whether a modality is making good use of its token share. Together, the two metrics are intended to reveal when text dominates because it is more efficient versus when dominance arises for other reasons such as fusion architecture design or task formulations that implicitly favor textual inputs (Wu et al., 14 Aug 2025).

1. Text dominance and the rationale for AEI

The motivating problem is attention dilution from severe token redundancy in non-textual modalities. Many non-text encoders produce large sequences, such as hundreds of visual tokens per image or many patches or segments in audio, video, and time-series. When tokens are numerous and semantically redundant, the attention spread over them lowers the average per-token weight, making it hard for these modalities to compete with compact, information-dense text tokens. This creates text dominance even when non-text tokens are the majority of the input (Wu et al., 14 Aug 2025).

AEI was designed to measure that imbalance at the level of efficiency. If a modality gets more attention than its fair share of tokens would suggest, its AEI is greater than 1 and it is attention-efficient. If it gets less, AEI is below 1 and it is attention-inefficient. In this formulation, AEI highlights the role of redundancy: when non-text tokens are replicated or inflated, token share rises but attention share typically does not keep pace, lowering AEI for that modality and increasing AEI for text.

A recurrent misconception is to treat text dominance as a phenomenon confined to vision-language tasks or as a consequence attributable only to data biases or model architectures. The study introducing AEI instead conducts a systematic investigation across images, videos, audio, time-series, and graphs, and identifies three underlying causes: attention dilution from severe token redundancy in non-textual modalities, the influence of fusion architecture design, and task formulations that implicitly favor textual inputs. This suggests that AEI is not merely a score for visual-language benchmarking, but a diagnostic for multimodal token-allocation regimes more broadly.

2. Formal definition and mathematical properties

The analysis relies on attention weights between generated output tokens and input tokens. Let the generated sequence have $N$ output tokens, and let the multimodal input be partitioned into modalities $m \in M$, each with token set $S_m$ and cardinality $|S_m|$. Let $L$ be the set of layers considered and $H$ the set of attention heads. For a chosen layer $l \in L$ and head $h \in H$, the cross-attention weights from output token $j$ to input token $i$ are denoted by

$m \in M$0

The measurements “leverage the cross-attention mechanisms employed by MLLMs during generation,” aggregating attention from output tokens to input tokens across layers and heads (Wu et al., 14 Aug 2025).

For modality $m \in M$1, the modality-level attention mass is

$m \in M$2

The modality attention share and token share are

$m \in M$3

The Attention Efficiency Index is then defined as

$m \in M$4

In the text-vs-others case emphasized in the paper,

$m \in M$5

Several properties are central to interpretation. The range is $m \in M$6. $m \in M$7 means attention share equals token share; $m \in M$8 means the modality garners more attention than its token budget; $m \in M$9 means less. AEI is explicitly sensitive to redundancy: increasing $S_m$0 increases $S_m$1, and unless $S_m$2 grows proportionally, $S_m$3 decreases, directly capturing attention dilution. It also obeys a conservation relation:

$S_m$4

AEI is robust to global scaling of attention because it uses attention share, but it is sensitive to changes in token counts across modalities by design.

3. Measurement protocol in MLLMs

The computation procedure aggregates attention from output tokens to input tokens using cross-attention during autoregressive generation. Early, middle, and late-layer statistics are computed by grouping layers as first two, middle two, and last two. The procedure takes as inputs the attention tensors $S_m$5 for selected layers and heads, modality token sets $S_m$6 with counts $S_m$7, and, optionally, a mask to exclude special tokens such as BOS or SEP if desired; the paper does not mandate masking, but typical practice may remove such tokens from $S_m$8 (Wu et al., 14 Aug 2025).

The metric is then computed in a fixed sequence. First, aggregate attention per modality:

$S_m$9

Second, compute shares:

$|S_m|$0

Third, compute AEI as $|S_m|$1. For text-vs-others analyses, the paper reports $|S_m|$2. Fourth, compute MDI in the binary case. Fifth, repeat the calculation for early, middle, and late layer groups, with average across heads handled implicitly via summation. Sixth, report AEI and MDI per layer group and modality.

Calibration is internal to the metric. AEI is self-normalized by token share, and $|S_m|$3 is the natural baseline corresponding to uniform attention proportional to token count. No external random-attention baseline is required.

The replication protocol is also explicit. The listed datasets are MMMU-Pro for images, MMBench-Video for videos, IEMOCAP for audio, synthetic time-series reasoning, and GraphGPT-eval-instruction for graphs. The listed models are Qwen2.5-VL-7B/32B/72B, LLaVA-1.5-7B, Kimi-VL-A3B-Instruct, VideoLLaMA3-7B, Qwen2-Audio-7B-Instruct, ChatTS-14B, and GraphGPT-7B. Attention extraction is performed by hooking cross-attention modules in the LLM backbone during generation, collecting $|S_m|$4 for chosen layers and heads, mapping input tokens to modality sets, and optionally excluding special tokens. Code availability is not specified; the procedure can be implemented in common deep learning frameworks by registering forward hooks at attention layers.

4. Complementarity with the Modality Dominance Index

The Modality Dominance Index is defined, for text tokens $|S_m|$5 and non-text tokens $|S_m|$6, as

$|S_m|$7

If $|S_m|$8, then

$|S_m|$9

$L$0 indicates text per-token dominance, $L$1 indicates non-text per-token dominance, and $L$2 indicates balance (Wu et al., 14 Aug 2025).

In the binary case, the relation between the two metrics is

$L$3

MDI therefore compares efficiency across modalities. AEI provides per-modality efficiency; MDI contrasts efficiencies to determine dominance. This complementarity is important because the two metrics can diverge. Under token inflation, replicating non-text tokens increases $L$4, typically lowering $L$5 and raising $L$6 even if $L$7 barely changes; MDI increases as $L$8. Under architectural bias, a model may exhibit increasing MDI across layers while AEI declines toward 1 or stays moderate, signaling that dominance is coming from per-token strength rather than disproportionate attention share.

A worked example clarifies the distinction. Suppose $L$9, $H$0, and the aggregated attention mass shares are $H$1 and $H$2. Then

$H$3

The resulting AEI values are

$H$4

The corresponding MDI is

$H$5

The interpretation given in the source is that text is extremely attention-efficient relative to its token budget, non-text is inefficient, and per-token text attention dwarfs per-token image attention. If visual tokens are compressed to 30 while attention shares remain similar, AEI moves toward 1 and MDI moves much closer to balance.

5. Empirical behavior across modalities and layers

Across five modalities and multiple models, the reported results show that text efficiency and text dominance are pervasive, with stronger effects at deeper layers and when non-text tokens are scaled. On images using MMMU-Pro, Qwen2.5-VL-7B shows MDI rising from 2.26 in early layers to 33.10 in late layers, while AEI drops from 14.24 to 1.42. LLaVA-1.5-7B shows MDI increasing from 1.58 to 17.37, while AEI increases from 1.04 to 4.23. The comparison is used to show divergent AEI trajectories: LLaVA’s AEI increases across layers, while Qwen2.5-VL’s AEI decreases, reflecting architecture-dependent efficiency (Wu et al., 14 Aug 2025).

On videos using MMBench-Video, late-layer text dominance is stronger still. Qwen2.5-VL-7B reports late-layer $H$6 and $H$7, while VideoLLaMA3-7B reports late-layer $H$8 and $H$9. The stated interpretation is that text is extraordinarily efficient relative to its small token proportion, consistent with severe attention dilution across many video tokens.

On audio using IEMOCAP with non-text tokens scaled by replication, Qwen2-Audio-7B-Instruct shows late-layer MDI climbing from 1.16 at $l \in L$0 to 6.73 at $l \in L$1 and 8.70 at $l \in L$2, while AEI rises from 1.08 at $l \in L$3 late to 4.31 at $l \in L$4 late and 5.09 at $l \in L$5 late. On synthetic time-series reasoning, ChatTS-14B shows late-layer MDI increasing from 3.52 at $l \in L$6 to 9.28 at $l \in L$7 and 16.25 at $l \in L$8, while AEI increases from 1.37 at $l \in L$9 late to 3.03 at $h \in H$0 late and 5.13 at $h \in H$1 late. In both cases, token replication exacerbates text dominance and raises text attention efficiency.

Graphs constitute the initial exception. On GraphGPT-eval-instruction with GraphGPT-7B, the late-layer values begin at $h \in H$2 and $h \in H$3 for text, meaning graph tokens attract more attention per token and text efficiency is below parity. With $h \in H$4 replication of graph tokens, late-layer MDI rises to 1.35 and text AEI to 1.14. The paper uses this shift to illustrate input-induced dominance: increasing non-text token counts can move a system from non-text per-token advantage to text dominance.

The cross-modality summary is therefore not simply that text usually receives more attention. The more specific finding is that text often has $h \in H$5 and $h \in H$6, especially in deeper layers and under non-text token scaling. A plausible implication is that layerwise AEI trajectories provide information about mechanism that would be obscured by MDI alone.

6. Token compression, practical diagnosis, and limitations

The proposed intervention is a simple token compression method, specifically [CLS]-guided visual token pruning (FasterVLM), applied before fusion in LLaVA-1.5-7B on MMMU-Pro. The stated effect is to reduce redundant visual tokens, rebalance attention by reducing $h \in H$7, and alleviate dilution. At baseline with 0% reduction, the middle-layer values are $h \in H$8 and $h \in H$9, and the late-layer values are $j$0 and $j$1. At 75% reduction, middle MDI becomes 1.81 and AEI 1.33, while late MDI becomes 3.39 and AEI 1.64. At 90% reduction, middle MDI becomes 1.10 and AEI 1.03, while late MDI becomes 1.84 and AEI 1.17. At 95% reduction, middle MDI becomes 0.86 and AEI 0.97, while late MDI becomes 3.39 and AEI 1.64. The highlighted result is that middle-layer MDI drops from 10.23 to 0.86 and AEI correspondingly moves toward approximately 1, indicating more equitable attention allocation (Wu et al., 14 Aug 2025).

The practical guidance is to use AEI alongside MDI. AEI near 1 indicates attention share proportional to token share; AEI greater than 1 flags a modality that is attention-efficient relative to its token budget; AEI less than 1 signals inefficiency, likely dilution. Persistent $j$2 with high MDI suggests text both dominates per token and is disproportionately efficient. If $j$3 but $j$4, dominance stems from per-token strengths rather than disproportionate share. The strongest evidence of text dominance under dilution is the combination $j$5 and $j$6.

The stated limitations and sensitivities are also important. Fusion architecture influences attention pathways; deeper or integrated fusion may amplify text dominance, while simpler bridges can yield different AEI dynamics. Task formulations that embed critical instructions or normalization in text can systematically raise $j$7. Tokenization and encoder design for non-text modalities affect redundancy and thus AEI. AEI focuses on cross-attention from outputs to inputs during generation; models without explicit cross-attention may need adaptations, such as causal self-attention masking to isolate attention from generated tokens to earlier input tokens.

Recommended robustness checks are to evaluate AEI across early, middle, and late layers, vary non-text token counts through replication or compression, and compare across architectures to distinguish intrinsic dominance from input-induced effects. Within this framework, AEI functions as a diagnostic lens for multimodal attention allocation, while token compression functions as an actionable lever for mitigating text dominance.