Attention Layer Redundancy

• Attention layer redundancy is the phenomenon where transformer layers perform superfluous computations, indicated by minimal performance drop when perturbed.
• Experimental methods such as hidden state randomization, layer pruning, and similarity scoring use metrics like cosine similarity and entropy to quantify redundancy.
• Exploiting redundancy enables dynamic layer pruning and adaptive attention reuse, leading to significant efficiency gains in language, vision, and multimodal models.

Attention layer redundancy refers to the phenomenon in deep neural architectures—especially transformer-based models—where one or more attention layers or heads perform functionally superfluous computations, extracting information that is already provided or mirrored by preceding layers. This redundancy manifests as high similarity in the transformations or attention maps produced by successive layers, or as a low contribution to the model’s predictive or generative accuracy when those layers are perturbed, pruned, or replaced. Identifying and quantifying attention layer redundancy is foundational for advancing both interpretability and efficiency in large language, vision, and multimodal models.

1. Formal Definitions and Quantitative Criteria

The canonical formalization of attention layer redundancy is as follows. For a transformer model with $L$ layers, let $h_\ell \in \mathbb{R}^d$ denote the matrix of hidden states at layer $\ell$. A layer $k$ is called “attention-redundant” for a given task if replacing its past-token representations $h_k$ with random vectors $r_k$—drawn i.i.d. from $\mathcal{N}(0, I)$ and rescaled to match $\|r_k\| = \|h_k\|$—results in only a negligible drop in performance. This is quantified by the accuracy drop: $\Delta \text{Acc}(k) = \text{Acc}_0 - \text{Acc}_k$ where $\text{Acc}_0$ is the model’s original performance and $\text{Acc}_k$ is the performance after hidden state manipulation at layer $k$.

A layer is considered redundant if $\Delta \text{Acc}(k) \leq \tau$, for some small threshold $\tau$ (typically 1–2% absolute). These criteria extend naturally to other metrics (e.g., $\Delta \text{ROUGE}_1$ in summarization tasks), and have direct analogues for image and multimodal models using similarity-based scores or entropy-based measures for attention map information content (Ben-Artzy et al., 2024, Maisonnave et al., 22 Aug 2025).

Alternative metrics include:

• Cosine similarity between input and output hidden activations of a layer: High similarity indicates redundancy (He et al., 2024, Men et al., 2024).
• Jensen–Shannon divergence or KL divergence between attention maps or head-averaged distributions of adjacent layers (Mu et al., 2024, Li et al., 9 Mar 2025).
• Shannon entropy of attention map entries: Low entropy implies deterministic, low-information computations and thus redundancy (Maisonnave et al., 22 Aug 2025).

2. Empirical Methodologies for Redundancy Identification

A range of experimental manipulations and measurement protocols have been proposed to expose attention layer redundancy:

• Hidden State Randomization: Replacing the history representations at a given layer $k$ with noise and measuring downstream accuracy. If late layers are randomized, minimal degradation is typically observed; early layer randomization is catastrophic (Ben-Artzy et al., 2024).
• Layer Pruning (“ShortGPT”): Measuring the Block Influence (BI) as one minus the normalized cosine similarity between the input and output of each block; removing layers with the lowest BI causes minimal accuracy loss up to moderate pruning rates (Men et al., 2024).
• Similarity-Based Scoring: Computing $$1 - \text{cosine similarity}(X_\text{in}, X_\text{out})$$ for each attention or MLP sublayer and ranking for joint pruning (He et al., 2024).
• KL-Divergence Across Layer Attention Distributions: Quantifying how much the attention distribution changes between adjacent layers; low divergence flags a redundant layer (Li et al., 9 Mar 2025).
• Attention-Head Reuse and Sharing: Empirical comparison of attention maps across layers and heads using total variation or cosine similarity. High similarity (e.g., 0.8–0.91 between adjacent layers) motivates direct reuse and cross-layer sharing strategies (Bhojanapalli et al., 2021, Mu et al., 2024).
• Token- or Entry-Level Redundancy: In vision transformers, fine-grained redundancy is measured by entropy of individual attention map entries or by cumulative information flow via token-to-token tracking (Maisonnave et al., 22 Aug 2025, Tong et al., 26 May 2025, Zhang et al., 2024).

Pseudocode routines for these measurements typically involve a calibration set and a forward pass to collect required statistics (cosine similarities, divergence measures, entropy) per layer and, where applicable, per attention head or token.

3. Manifestations in Language, Vision, and Multimodal Models

Redundancy in attention layers is now well-documented across several domains:

Model Family	Empirical Manifestation of Redundancy	Quantitative Effects
Decoder-only LLMs	Late layers robust to randomization, skipping, or patching	Top 30–50% of layers removable with ≤2% loss
Vision Transformers (ViT)	Many heads/entries have near-constant/frozen outputs across images	Up to 60% of MHSA sparsified with <2% loss
LVLMs and Multimodal	Vision->Vision self-attention redundant past middle layers	40–50% compute/attention removable, 0–2% loss

In large decoder LLMs (e.g., Llama2-7B, Mistral-7B), a sharp phase transition is observed: manipulating the bottom half of layers destroys performance, but manipulating the top 30–50% leaves accuracy unchanged (Ben-Artzy et al., 2024). Similar trends appear in ViTs, where masking, freezing, or quantizing low-entropy entries enables heavy pruning without performance degradation (Maisonnave et al., 22 Aug 2025). In LVLMs, both computation-level (per-token) and entry-level redundancy can be squeezed out via strategies such as ProxyV or information-flow-based token pruning (Wu et al., 21 May 2025, Tong et al., 26 May 2025, Zhang et al., 2024).

4. Algorithmic and Architectural Exploitation

Redundancy motivates several practical strategies:

• Dynamic Layer Pruning/Slicing: Methods such as dynamic slicing or ShortGPT use per-layer redundancy scores to set pruning budgets or remove entire blocks; dynamic slicing achieves up to 7% lower perplexity vs. static slicing (Dumitru et al., 2024, Men et al., 2024).
• Adaptive Attention Skipping and Reuse: Cross-layer attention sharing (LiSA) and reuse transformers buffer or synthesize attention maps for use in multiple layers, yielding 6–24× Q/K compression and up to 32% speedup with minimal loss (Mu et al., 2024, Bhojanapalli et al., 2021).
• Redundant Head or Token Elimination: In vision models, freezing or sparsifying low-entropy heads and tokens (EAM) reduces computation and memory, with DeiT/Swin models achieving up to 40% sparsity at no cost (Maisonnave et al., 22 Aug 2025); ProxyV and FlowCut prune vision tokens/layers via information flow metrics (Wu et al., 21 May 2025, Tong et al., 26 May 2025).
• Consolidation of Attention with FFN: Recognizing the two-phase “attend first, consolidate later” behavior, upper attention layers can be replaced or augmented with specialized FFN-only "consolidation" modules, or skipped entirely in decoding (Ben-Artzy et al., 2024).
• Channel-level Redundancy Suppression: Redundancy Reduction Attention (RRA) applies channel gating and iterative suppression to ensure multi-glimpse representations attend to distinct, non-redundant features (Zhu et al., 2018).

5. Theoretical and Empirical Insights

Layer redundancy is not architectural noise but emerges from the iterative refinement and stabilization properties of deep self-attention. Early layers integrate information and structure input, while deeper layers increasingly consolidate and refine, often without introducing new, task-relevant interactions (Ben-Artzy et al., 2024, Bhojanapalli et al., 2021). High cross-layer similarity in attention weights is a learned, dataset-sensitive phenomenon; on random data, redundancy vanishes (Bhojanapalli et al., 2021).

Redundancy is not uniform across tasks, domains, or sequence positions. Shallow layers remain sensitive and cannot be blindly pruned. Methodologies that align or compensate for head permutations and correct shallow-layer mismatches (e.g., via low-rank deltas in LiSA) are critical for robust performance (Mu et al., 2024). Additional implications include the design of models with explicit bimodal (retrieval + consolidation) partitions, and more aggressive hybrid layer-drop strategies combining attention and MLP pruning for further efficiency (He et al., 2024).

6. Efficiency, Robustness, and Model Design Implications

Exploiting attention layer redundancy brings substantial benefits:

Approach	Metric	Result/Impact
Top-layer attention skip	Inference speedup	Up to 48% (LLaMA-2-70B @ 50% attent. prune)
Layer-sharing (LiSA)	Compression & speed	6x–24x Q/K compression; 19–32% higher throughput
Entry-wise entropy prune	MHSA FLOPs	40–60% pruned at <2% accuracy loss (ViT/Swin)
Dynamic slicing	Perplexity	3–7% lower vs. constant slicing at same prune rate
Head/token removal (ViT)	Memory/FLOPs	50% of tokens removable, 3.2x faster, accuracy↑

Redundancy-based pruning and compression are orthogonal to quantization and other efficiency techniques. Combined methods yield multiplicative savings in resource-constrained deployment (Men et al., 2024, Maisonnave et al., 22 Aug 2025). Future architectures may explicitly minimize depth in the attention path beyond the critical consolidation transition, allocate redundancy adaptively based on context or domain, and integrate redundancy metrics into neural architecture search or dynamic runtime slicing.

7. Open Challenges and Future Research Directions

• Dynamic/Instance-wise Adaptation: Further refining redundancy exploitation at inference by learning to skip or prune based on per-instance signals, not just global averages (Ben-Artzy et al., 2024, Bhojanapalli et al., 2021).
• Generalization Across Modality and Scale: Extending these analyses and techniques to encoder-only, encoder-decoder, and very large (>70B) scale LLMs, or to complex multimodal and structured prediction settings (Ben-Artzy et al., 2024, Wu et al., 21 May 2025).
• Alignment and Safety: Investigating how pruning upper attention layers affects alignment, factuality, and model editing or safety interventions.
• Redundancy Evolution: Tracing redundancy emergence during pretraining and fine-tuning, and its modulation by domain-specific or complex-reasoning data (He et al., 2024).
• Robustness to Distribution Shifts: Testing whether redundancy-based optimization preserves robustness and generalizability to out-of-domain tasks.

A plausible implication is that, given the highly structured redundancy profile of modern deep attention networks, significant advances in training and inference efficiency remain possible—without commensurate sacrifices in accuracy or generative fidelity—if redundancy is measured and exploited with context- and architecture-specific precision.

---

Key References:

• "Attend First, Consolidate Later: On the Importance of Attention in Different LLM Layers" (Ben-Artzy et al., 2024)
• "Cross-layer Attention Sharing for LLMs" (Mu et al., 2024)
• "What Matters in Transformers? Not All Attention is Needed" (He et al., 2024)
• "Enhancing Layer Attention Efficiency through Pruning Redundant Retrievals" (Li et al., 9 Mar 2025)
• "Streamline Without Sacrifice -- Squeeze out Computation Redundancy in LMM" (Wu et al., 21 May 2025)
• "FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-LLMs" (Tong et al., 26 May 2025)
• "Exploiting Information Redundancy in Attention Maps for Extreme Quantization of Vision Transformers" (Maisonnave et al., 22 Aug 2025)
• "ShortGPT: Layers in LLMs are More Redundant Than You Expect" (Men et al., 2024)
• "Change Is the Only Constant: Dynamic LLM Slicing based on Layer Redundancy" (Dumitru et al., 2024)
• "Leveraging redundancy in attention with Reuse Transformers" (Bhojanapalli et al., 2021)
• "Fine-grained Video Categorization with Redundancy Reduction Attention" (Zhu et al., 2018)
• "Continual Transformers: Redundancy-Free Attention for Online Inference" (Hedegaard et al., 2022)