Attention-Map Entropy (AME)

• Attention-Map Entropy (AME) is a metric that quantifies the uncertainty or concentration of attention distributions using Shannon entropy in neural architectures.
• AME is applied to guide model compression through head/token pruning, extreme quantization, and entropy-matched linear attention, resulting in efficiency gains and controlled accuracy trade-offs.
• AME serves as an uncertainty measure for active exploration and test-time adaptation, enhancing robustness and interpretability in vision and sequence models.

Attention-Map Entropy (AME) quantifies the uncertainty or concentration of attention distributions in neural architectures, particularly self-attention models and attention-based mechanisms in convolutional and recurrent networks. AME is defined as the Shannon entropy of attention weights or attention-generated spatial maps, serving as a principled metric to measure, regularize, or exploit the internal behavior of attention modules for objectives ranging from model compression to uncertainty-driven decision making and interpretability.

1. Mathematical Definitions and Frameworks

AME fundamentally measures the randomness of an attention distribution $p = (p_1, ..., p_N)$ via the Shannon entropy:

$H(p) = -\sum_{j=1}^N p_j \log p_j$

In self-attention, for a row (corresponding to a query position) of the attention matrix $A_{i,:}$, AME is computed as $H(A_{i,:})$. For head-level or map-level entropy, values are averaged or summed across rows and, if relevant, over multiple heads and layers. Extensions occur in different domains:

• Image Regions: The entropy can be computed over spatial maps derived from histograms of pixel intensities or from joint center-neighbor value pairs, as in spatial entropy maps (Su et al., 2021).
• Transformer Attention: For each head, layer, or even individual element, AME can be measured on softmax-normalized attention weights (Maisonnave et al., 22 Aug 2025, Mao et al., 2023, Mali, 24 Nov 2025).
• Active Exploration: AME is computed at each timestep over the attention matrix spanning observed and yet-unseen patches, driving the next observation decision (Pardyl et al., 2023).
• Visual Attention Heatmaps: For gaze data, the heatmap probability distribution is subjected to the same entropy measure to yield a scalar capturing fixation spread (Gu et al., 2018).

This formalization ensures compatibility across architectures and application domains.

2. AME in Model Compression and Quantization

AME provides a theoretically grounded measure to guide pruning and quantization in transformer models:

• Head and Token Pruning: High-entropy heads (those where attention is diffused and unspecialized) are pruned, as their uniformity suggests redundancy, with empirical evidence showing 13–44% FLOPs reduction and, in some cases, modest accuracy improvements (Mao et al., 2023). A per-head AME is computed:

$$E^{h,l} = -\sum_{i=1}^N \sum_{j=1}^N A^{h,l}_{i,j} \log A^{h,l}_{i,j}$$

• Extreme Quantization: Entropy Attention Maps (EAM) (Maisonnave et al., 22 Aug 2025) use AME to identify low-entropy, "deterministic" attention weights that can be fixed and quantized. Dataset-wide AME is computed efficiently via histogram binning over a calibration set. Freezing low-entropy entries and quantizing all entries to 4 bits yields accuracy parity or improvement up to 30% sparsity on ImageNet.
• Entropy-Matched Linear Attention: By matching the entropy of a linear attention surrogate to that of the computationally expensive softmax distribution, one can approximate softmax attention in $O(N)$ time without sacrificing weight "sharpness" or distributional structure (Zhang et al., 5 Nov 2025). AME is central in determining the appropriate scaling parameter for the linear surrogate, ensuring that the resulting distributions remain close in KL divergence according to their strict concavity properties.

3. AME as an Uncertainty and Exploration Signal

AME directly quantifies internal model uncertainty in both vision and sequence models:

• Active Visual Exploration: At each decision step, patches or observations with maximum AME are selected, leveraging the model’s own internal uncertainty for glimpse selection, eliminating the need for separate selection models, loss terms, or hyperparameters (Pardyl et al., 2023). This approach enhances performance in reconstruction, segmentation, and classification tasks under partial observation.
• Test-Time Adaptation: AME is employed as a test-time loss to encourage deterministic (low-entropy) attention over image regions, which leads to improved robustness under distribution shift—yielding +5–7% mCA on corrupt CIFAR-10 without harming clean accuracy (Mali, 24 Nov 2025). The AME loss is

$$L_{\text{attn}} = -\sum_{i=1}^m \hat{a}_i \log \hat{a}_i$$

where $\hat{a}$ represents normalized CLS-to-patch attention.

• Visual Attention Heatmaps and Behavioral Studies: AME (as Visual Attention Entropy) computed over gaze probability maps tightly correlates ($r=-0.65$ for normalized VAE) with subjective quality ratings, enabling rapid assessment of first-impression aesthetics and perceptual fluency (Gu et al., 2018).

4. Applications Across Modalities and Architectures

The utility of AME spans a diverse range of model classes and data modalities:

Application Domain	AME Role	Key Result/Metric
Video object detection (SGE-Net)	Zero-parameter spatial attention	+1.1% mAP, negligible parameter cost
Transformer pruning (AMG)	Head/token redundancy quantification	13–44% FLOPs ↓, ≤0.4% top-1 drop
Energy efficient ViTs (EAM)	Entr. for freezing & quantization	Same/higher accuracy at 10–30% sparsity
Time series forecasting (EALA)	Entropy-matched linear attention	Equal accuracy with 10–40% less memory
Test-time adaptation (AttenDence)	Robust attention sharpening	+5–7% mCA (CIFAR-10-C), same clean acc
Human eye-tracking / UI aesthetics	Gaze entropy analysis	rVAE correl. r=–0.65, 85% page accuracy
Active visual exploration (MAE-guided)	Patch acquisition via uncertainty	3–7% accuracy ↑, 5–10% RMSE ↓

AME is agnostic to input modality. It applies to pixel regions, image patches, sequence positions, or even behavioral heatmaps.

5. Computation, Implementation, and Training Strategies

AME is straightforward to compute, as it leverages existing post-softmax attention weights. For typical transformer-based settings, the following procedure is canonical:

1. Obtain attention matrix $A^{h,l} = \mathrm{softmax}(QK^T/\sqrt{d})$.
2. For each row (query position), compute Shannon entropy.
3. Aggregate (average or sum) over positions and/or heads for a scalar AME.
4. Use as a loss regularizer, pruning criterion, freezing mask, exploration driver, or visualization input.

In settings where spatial attention is needed (e.g., video detection), AME is computed directly over image histograms or local patch windows (Su et al., 2021). For robust experimental design, AME can be accumulated over mini-batches or calibration sets and quantized via histogram binning for large-scale models (Maisonnave et al., 22 Aug 2025). No additional loss terms are needed unless used as an explicit regularizer; AME is often exploited via inference-time or architecture-level operations.

6. Experimental Outcomes and Limitations

AME-driven schemes demonstrate:

• Efficiency: Substantial reductions in model parameters and computation at minimal or positive accuracy impact (Mao et al., 2023, Maisonnave et al., 22 Aug 2025, Zhang et al., 5 Nov 2025).
• Robustness: Superior or at least maintained performance under input corruptions, distribution shifts, or sparse observations (Mali, 24 Nov 2025, Pardyl et al., 2023).
• Interpretability and Behavioral Correlation: Strong correspondence between low-entropy attention and relevant/semantically meaningful regions, as well as tight linkages to human perceptual responses (Gu et al., 2018).

Limitations include sensitivity to the calibration set (for quantization/pruning), dependence on attention calibration (poorly structured attention impairs uncertainty-based methods), absence of coverage guarantees in exploration strategies, and potential overconfidence on true out-of-distribution samples when entropy is forcibly minimized.

7. Theoretical Properties and Generalization

AME’s power rests on the strict concavity of the Shannon entropy over the probability simplex, which ensures that for attention distributions preserving both sort order and entropy, proximity in KL divergence is guaranteed (Zhang et al., 5 Nov 2025). This property underpins efficiency gains in linear attention surrogates and validates the rationale for entropy-guided model pruning and fusion. More generally, AME serves as an information-theoretic bridge between model internals and global performance, enabling principled interventions without the need for ad hoc regularization or externally imposed objectives. Its broad applicability, strong empirical performance, and theoretical grounding position AME as a central tool in both analysis and efficient engineering of attention-based models.