Layer Image Attention Entropy (LIAE)

• LIAE is a diagnostic metric that quantifies the dispersion of image-token attention using Shannon entropy to identify anomalous layers in multimodal models.
• It applies a layer-wise analysis of attention distributions across heads to inform transformer stability, token pruning, and adaptive inference strategies.
• Empirical results show that higher LIAE correlates with hallucination risks, and adjusting entropy thresholds can improve captioning and VQA performance.

Layer Image Attention Entropy (LIAE) is a layer-wise attention diagnostic that quantifies how attention to image tokens is distributed inside attention-based models. In the formulation introduced for multimodal LLMs, LIAE is used to identify anomalous layers in the attention stack that are prone to causing hallucinations, with high entropy indicating that attention is overly diffuse and thus not well-focused on relevant visual information (Yang et al., 9 Sep 2025). Closely related formulations appear in layer-wise attention analysis for transformer stability, test-time adaptation, active visual exploration, patch pruning, and inference acceleration, where entropy is computed over attention maps, attention heads, or cross-modal token distributions and then used either as a diagnostic or as an optimization criterion (Zhai et al., 2023, Mali, 24 Nov 2025, Pardyl et al., 2023, Aizawa et al., 4 Apr 2026, Liu et al., 7 Apr 2026).

1. Core definitions and mathematical forms

In D-LEAF, LIAE is defined from a layer-wise summary of image-token attention. For an MLLM with $L$ transformer layers and $H$ attention heads, each head produces an attention distribution $A^{(l)}_{h,n}$ over the image tokens. The method first computes the Maximum Attention Matrix (MAM) for each layer,

$$\text{MAM}_{n}^{(l)} = \max_{h=1,\dots,H} A^{(l)}_{h,n},$$

then normalizes it over image tokens,

$$P(\text{MAM}_n^{(l)}) = \frac{\text{MAM}_n^{(l)}}{\sum_{k=1}^{N} \text{MAM}_k^{(l)}},$$

and finally computes

$$\text{LIAE}^{(l)} = - \sum_{n=1}^N P(\text{MAM}_n^{(l)}) \log P(\text{MAM}_n^{(l)}).$$

Low LIAE means that most attention is sharply focused on a few image tokens, while high LIAE means that attention is spread out or diffuse (Yang et al., 9 Sep 2025).

Other papers use closely related layer-wise attention entropy constructions. In the TruthfulQA trace-level study, attention entropy is computed per head as $H = -\sum_i a_i \log a_i$, then averaged across heads to obtain the layer-level attention entropy, also referred to as LIAE in the context of compositional attention “images” (Adeseye et al., 4 Apr 2026). In VLA-InfoEntropy, attention entropy is defined per visual token from cross-attention scores received from text tokens: $$q(w,i) = \frac{ \exp \left(\frac{1}{LH}\sum_{l=1}^L \sum_{h=1}^H A^{(l,h)}_{w\to i} \right) }{ \sum_{w' \in \mathcal{W}} \exp\left(\frac{1}{LH}\sum_{l=1}^L \sum_{h=1}^H A^{(l,h)}_{w'\to i}\right) },$$

$$H_{\text{attn}(i)} = -\sum_{w\in\mathcal{W}} q(w,i) \log_2 q(w,i),$$

with low attention entropy indicating focused, semantically relevant attention (Liu et al., 7 Apr 2026).

Setting	Entropy object	Operational role
D-LEAF	Normalized MAM over image tokens in a layer	Flags anomalous layers
TruthfulQA trace analysis	Mean across heads of attention entropy	Structural analysis across layers
VLA-InfoEntropy	Cross-attention over text tokens for each visual token	Dynamic token selection
AttenDence	CLS-to-patch attention in the final layer	Test-time adaptation objective
AME	Row entropy of attention maps across heads	Next-glimpse selection
Rényi patch pruning	Per-patch Shannon or Rényi attention entropy	Patch pruning

This range of definitions shows that LIAE is not a single fixed formula. What remains constant is the use of Shannon-style entropy, or an extension such as Rényi entropy, to measure whether attention is concentrated or diffuse at a layer, token, or image-conditioned interface.

2. Conceptual lineage in entropy-guided image representation

A direct antecedent of LIAE-like reasoning appears in REMAP, a CNN-based image retrieval system that measures the information gain of each region and layer using KL-divergence rather than softmax-attention entropy. For each spatial region and each layer, REMAP computes the Kullback-Leibler divergence between the pdf of Euclidean distances between descriptors of matching region pairs, $Pr(y|m)$, and the pdf of Euclidean distances between descriptors of non-matching region pairs, $H$0: $H$1 High KL-divergence indicates that a region or layer is highly discriminative; low KL-divergence indicates little discrimination (Husain et al., 2019).

REMAP uses these KL-divergence values to initialize the region weights $H$2 in the aggregation layer, implements KL-divergence Weighting as a 1D convolutional layer, and aggregates pooled regional features as weighted sums,

$H$3

The paper does not use the precise term LIAE, but the principle is stated as equivalent: each layer’s and region’s attention in the aggregation process is proportional to its information entropy—its ability to separate matches from non-matches (Husain et al., 2019).

This establishes an important historical distinction. In REMAP, “entropy-guided” pooling is based on relative entropy between match and non-match distance distributions. In later transformer and multimodal work, LIAE is usually based on the Shannon entropy of an attention distribution itself. The common idea is selective emphasis on semantically useful regions and layers, but the probability spaces and operational objectives differ.

3. Hallucination localization and correction in multimodal models

The most explicit use of LIAE as a named method appears in D-LEAF, where it is proposed as a diagnostic metric to identify anomalous layers in the attention stack of MLLMs that are prone to causing hallucinations. During inference, LIAE is computed for every layer as the model generates output tokens. A dynamic baseline is maintained as the best, that is lowest, LIAE observed up to the current layer,

$H$4

If $H$5 exceeds the running minimum, that layer is marked as potentially anomalous (Yang et al., 9 Sep 2025).

The empirical analysis reported for COCO2014 and POPE shows that LIAE is consistently higher for hallucinated tokens than for real tokens across models such as MiniGPT-4 and Shikra, and that the difference is highly statistically significant with a Wilcoxon signed-rank test, $H$6 (Yang et al., 9 Sep 2025). The same work states that LIAE pinpoints faulty layers and that Image Attention Focus (IAF) reliably ranks heads that warrant correction. Guided by these signals, Dynamic Layer-wise Entropy and Attention Fusion delivers a 53% relative improvement on standard captioning benchmarks, and on VQA both accuracy and F1-score improve by approximately 4% (Yang et al., 9 Sep 2025).

A related but non-entropic diagnosis appears in “Seeing Clearly by Layer Two,” which analyzes the distribution of attention scores for image tokens across each layer and head and links hallucinations to the pattern of attention sinks in the self-attention matrix of image tokens. Shallow layers exhibit dense attention sinks and deeper layers show sparse attention sinks; heads with high-density attention sink in the image part play a positive role in alleviating hallucinations (Zhang et al., 2024). The proposed EAH method is training-free, identifies the attention head that shows the vision sink in a shallow layer, and broadcasts that head’s attention map to other heads in the layer. This is not LIAE, but it is a closely related layer-to-head attention diagnostic aimed at the same failure mode (Zhang et al., 2024).

These results support a layer-centric view of multimodal hallucination. In that view, hallucination is not only an output-level error but also a layer-local disruption in how image tokens are attended to and propagated.

4. Stability, adaptation, and the non-monotonic role of low entropy

Attention entropy is also used to study transformer training dynamics. “Stabilizing Transformer Training by Preventing Attention Entropy Collapse” tracks the attention entropy for each attention head during the course of training and identifies a common pattern across different architectures and tasks: low attention entropy is accompanied by high training instability, which can take the form of oscillating loss or divergence (Zhai et al., 2023). The paper denotes pathologically low attention entropy as entropy collapse and proposes $H$7Reparam, which reparametrizes all linear layers as

$H$8

with spectral normalization and an additional learned scalar, to prevent entropy collapse in the attention layers (Zhai et al., 2023).

The same paper proves a tight lower bound of the attention entropy that decreases exponentially fast with the spectral norm of the attention logits, and uses this to motivate spectral control of query and key projections (Zhai et al., 2023). In this setting, attention entropy is a proxy for model sharpness, and extremely low entropy is undesirable because it signals instability rather than reliable focus.

By contrast, AttenDence proposes minimizing the entropy of attention distributions from the CLS token to image patches as a test-time adaptation objective. Let $H$9 denote the attention tensor from the final transformer layer. After extracting CLS-to-patch attention, averaging over heads, and normalizing over patch tokens,

$A^{(l)}_{h,n}$0

is minimized episodically using a single gradient step for each test image, after which model weights are reset before the next image (Mali, 24 Nov 2025). The reported result is that attention entropy minimization improves robustness across diverse corruption types while not hurting performance on clean data on a single sample stream of images at test time (Mali, 24 Nov 2025).

Taken together, these papers rule out a simple interpretation in which “lower LIAE is always better.” Low entropy may be beneficial when the goal is to focus attention during test-time adaptation, but pathologically low entropy can also indicate entropy collapse and unstable optimization. The meaning of a given entropy regime depends on architecture, training stage, and task objective.

5. Token selection, pruning, and compute allocation

Several later systems operationalize attention entropy as a selection policy over patches or tokens. In Active Visual Exploration Based on Attention-Map Entropy, a transformer-based masked autoencoder computes attention maps in the decoder and assigns each patch an entropy score,

$A^{(l)}_{h,n}$1

then selects the next glimpse as

$A^{(l)}_{h,n}$2

Already observed patches are set to zero in the entropy map, and the patch with the highest entropy is chosen as the next observation (Pardyl et al., 2023). Here, high entropy is treated as internal uncertainty and therefore as an indicator of the most informative next glimpse.

In patch pruning for vision transformers, the criterion is reversed: low-entropy patches, which receive selective and concentrated attention, are kept as important, while high-entropy patches with attention spread across many locations are treated as redundant (Aizawa et al., 4 Apr 2026). The Shannon version is

$A^{(l)}_{h,n}$3

and the Rényi extension is

$A^{(l)}_{h,n}$4

The method sorts non-class patch tokens by entropy, keeps the top $A^{(l)}_{h,n}$5 fraction with lowest entropy, and reports that at keep rate $A^{(l)}_{h,n}$6, accuracy drop is nearly zero with over 34% FLOPs reduction and about 55% throughput gain on DeiT-S (Aizawa et al., 4 Apr 2026).

VLA-InfoEntropy extends the entropy-based selection idea to Vision-Language-Action inference. It combines visual entropy, attention entropy, and timestep information so that early steps use more visual tokens and later steps use more attention-guided tokens: $A^{(l)}_{h,n}$7 The important token set is selected from the top visual-entropy tokens and the top attention-information tokens,

$A^{(l)}_{h,n}$8

where $A^{(l)}_{h,n}$9 (Liu et al., 7 Apr 2026). The paper reports a 1.53× speedup in FLOPs, a 34.9% reduction in floating point operations, and 39.8% lower CUDA latency, while average success rate remains as high or higher than the baseline (Liu et al., 7 Apr 2026).

These uses show that LIAE-like quantities can support opposite decision rules depending on the problem. High entropy can mean “query this region next,” as in active exploration; low entropy can mean “keep this patch,” as in pruning; and low normalized entropy can mean “this token is semantically important,” as in VLA inference acceleration.

6. Related entropy-attention mechanisms, limitations, and interpretive issues

Entropy-guided attention is broader than the named LIAE formulations. In efficient single image super-resolution, Entropy Attention computes channel-wise differential entropy under a Gaussian approximation,

$$\text{MAM}_{n}^{(l)} = \max_{h=1,\dots,H} A^{(l)}_{h,n},$$0

and uses it in a channel recalibration block

$$\text{MAM}_{n}^{(l)} = \max_{h=1,\dots,H} A^{(l)}_{h,n},$$1

This mechanism is not LIAE, but it is an entropy-based attention module that weights channels according to information content and is reported to increase PSNR by $$\text{MAM}_{n}^{(l)} = \max_{h=1,\dots,H} A^{(l)}_{h,n},$$2–$$\text{MAM}_{n}^{(l)} = \max_{h=1,\dots,H} A^{(l)}_{h,n},$$3 dB over the base model on standard SR benchmarks, with entropy computation under the Gaussian assumption about $$\text{MAM}_{n}^{(l)} = \max_{h=1,\dots,H} A^{(l)}_{h,n},$$4–$$\text{MAM}_{n}^{(l)} = \max_{h=1,\dots,H} A^{(l)}_{h,n},$$5 faster than traditional entropy computation (Zhao et al., 2024).

A more general entropy-guidance program for dense and convolutional networks measures entropy change through linear and convolutional layers and uses loss terms such as

$$\text{MAM}_{n}^{(l)} = \max_{h=1,\dots,H} A^{(l)}_{h,n},$$6

to encourage architecture-specific entropy patterns (Meni et al., 2023). The paper states that similar analytical derivations could extend to attention layers, which suggests a possible bridge between determinant-based entropy propagation and layer-wise attention entropy (Meni et al., 2023).

Several limitations recur across the LIAE literature. D-LEAF notes that excessive correction can lead to overly terse or incomplete outputs, lower F1 scores, and sensitivity to how many heads or layers are corrected; it also states that architectures with radically different attention mechanisms or tokenization schemes might require metric recalibration (Yang et al., 9 Sep 2025). AttenDence notes that attention entropy minimization relies on the quality and interpretability of transformer attention maps and that overconfident but wrong attention can be an issue in high-uncertainty or very severe corruptions (Mali, 24 Nov 2025). The TruthfulQA study reports that balanced regimes with moderate and stable entropy differ from both deterministic low-entropy regimes and highly diffuse high-entropy regimes, and concludes that truthfulness emerges from structured entropy and attention dynamics rather than from a single universal entropy target (Adeseye et al., 4 Apr 2026).

A common misconception is therefore that LIAE is a scalar confidence score with a fixed semantic interpretation. The literature does not support that reading. Depending on the application, high LIAE may indicate hallucination risk, useful uncertainty, or redundant context; low LIAE may indicate semantically grounded focus, aggressive compression, or entropy collapse. The consistent technical role of LIAE is not a universal optimum, but a quantitative handle on how image-conditioned attention is distributed across layers and how that distribution can be monitored, regularized, or acted upon.