Nonuniform Per-Head Compaction

• Nonuniform per-head compaction is a strategy that allocates resources based on the unique importance and utilization of each head in transformer and LSM-tree systems.
• It employs adaptive techniques such as dynamic retention rate allocation and head-level pruning to optimize performance while reducing operational costs.
• Empirical results demonstrate significant gains, including an 18% speedup in LVLMs and up to a 10x reduction in write amplification in LSM-trees compared to uniform methods.

Nonuniform per-head compaction refers to a class of algorithms and policies that allocate compaction resources or pruning quotas independently to each component ("head") in a multi-headed architecture or data system, instead of applying a uniform compaction or pruning rate across all heads. This strategy has achieved state-of-the-art efficiency in transformer-based models, large vision-LLMs (LVLMs), and large-scale data structures such as LSM-trees, by acknowledging heterogeneity in head specialization, attentional behavior, and sensitivity to compaction. Empirical and theoretical results demonstrate that nonuniform per-head compaction can yield substantial cost reductions, latency improvements, and negligible or sometimes even improved downstream accuracy when compared to uniform schemes (Meng et al., 20 Feb 2025, Zweiger et al., 18 Feb 2026, Mathieu et al., 2020).

1. Principles of Nonuniform Per-Head Compaction

Nonuniform per-head compaction operates on the observation that different heads—whether they are transformer attention heads, storage components, or data structure partitions—have highly variable utility and responsivity to compaction. In transformer and LVLM architectures, attention heads focus on different modalities and spatial or contextual regions, and their importance varies non-monotonely across layers and heads. Consequently, pruning (in neural nets) or compacting (in storage systems) should be targeted according to per-head, per-layer contextual importance and not distributed equally.

A canonical example is the “Vision Token Re-attention” phenomenon in LVLM decoders: lower or intermediate layers may attend more strongly to visual tokens, while upper layers’ attention shifts to text or abstract features, and attention heads specialize to distinct segments of input (Meng et al., 20 Feb 2025). In data systems such as the LSM-tree model in databases, compaction decisions are best made based on nonuniform batch input and query statistics across “heads” (run levels), rather than a fixed, equal policy (Mathieu et al., 2020).

2. Methodologies in Transformer and Vision-LLMs

In large vision-LLMs, nonuniform per-head compaction typically involves two structurally coupled stages:

1. Layer-Level Retention Rate Allocation: For each decoder layer $l$, the average attention to vision tokens, denoted $\gamma^l$, is computed as

$$\gamma^l = \frac{1}{H} \sum_{h=1}^H \sum_{k \in \text{vision token indices}} A^{l,h}_{S,k}$$

with $A^{l,h} \in \mathbb{R}^{S \times S}$ the attention weights for head $h$ in layer $l$. The retention fraction $r^l$ for each layer is dynamically set based on $\gamma^l$ using thresholds $(\alpha, \beta, r, \Delta r)$:

$$r^l = \begin{cases} r + \Delta r & \text{if } \gamma^l \geq \alpha \ r - \Delta r & \text{if } \gamma^l < \beta \ r & \text{otherwise} \end{cases}$$

For example, with $\gamma^l$0, up to 70% of vision tokens can be retained in highly attentive layers, but only 10% in nearly vision-indifferent layers.

2. Head-Level Pruning: Each head $\gamma^l$1 within a layer $\gamma^l$2 selects exactly $\gamma^l$3 tokens for each image $\gamma^l$4, prioritizing tokens with the highest attention from the “generation token.” A binary mask $\gamma^l$5 is constructed so only those vision tokens and all text tokens remain in the key-value cache of each head. Pruning is then applied per head via:

$\gamma^l$6

This ensures fine-grained, nonuniform compaction, with each head retaining a minimal but sufficient subset of tokens tailored to its actual usage (Meng et al., 20 Feb 2025).

3. Attention Matching and Per-Head KV Compaction

Attention Matching formulates nonuniform per-head compaction as an optimization problem that directly matches the outputs and mass of the attention mechanism for each head:

• Each head $\gamma^l$7 has its original key-value cache $\gamma^l$8 of length $\gamma^l$9, and compacts this to $$\gamma^l = \frac{1}{H} \sum_{h=1}^H \sum_{k \in \text{vision token indices}} A^{l,h}_{S,k}$$0 of length $$\gamma^l = \frac{1}{H} \sum_{h=1}^H \sum_{k \in \text{vision token indices}} A^{l,h}_{S,k}$$1.
• The loss function sums two terms: the squared difference in attention “mass” and in attention outputs over a reference query set:

$$\gamma^l = \frac{1}{H} \sum_{h=1}^H \sum_{k \in \text{vision token indices}} A^{l,h}_{S,k}$$2

where $$\gamma^l = \frac{1}{H} \sum_{h=1}^H \sum_{k \in \text{vision token indices}} A^{l,h}_{S,k}$$3 and $$\gamma^l = \frac{1}{H} \sum_{h=1}^H \sum_{k \in \text{vision token indices}} A^{l,h}_{S,k}$$4 are original and compacted masses; $$\gamma^l = \frac{1}{H} \sum_{h=1}^H \sum_{k \in \text{vision token indices}} A^{l,h}_{S,k}$$5, $$\gamma^l = \frac{1}{H} \sum_{h=1}^H \sum_{k \in \text{vision token indices}} A^{l,h}_{S,k}$$6 are original and compacted attention outputs.

• Nonuniformity enters as the per-head compaction budget $$\gamma^l = \frac{1}{H} \sum_{h=1}^H \sum_{k \in \text{vision token indices}} A^{l,h}_{S,k}$$7, allocated by optimizing per-head loss curves (sensitivity) under a global quota.

Efficient closed-form solutions are available for fitting the value and bias parameters per head (using nonnegative least squares for biases and ordinary least squares for values), and practical heuristics (e.g., Orthogonal Matching Pursuit and highest-attention selection) let heads independently choose their compacted key sets. The global compaction ratio is met by greedily allocating units of quota to heads with the highest marginal benefit, yielding a nonuniform head distribution (Zweiger et al., 18 Feb 2026).

4. Nonuniform Compaction in Data Structures and LSM-Trees

In log-structured merge (LSM) trees and related databases, nonuniform per-head compaction is modeled as an online set-cover problem:

• Each batch flush $$\gamma^l = \frac{1}{H} \sum_{h=1}^H \sum_{k \in \text{vision token indices}} A^{l,h}_{S,k}$$8 is a new “head” or run, and the compaction policy must minimize either total build+query cost (“Min-Sum Dynamization”) or build cost under a query cap $$\gamma^l = \frac{1}{H} \sum_{h=1}^H \sum_{k \in \text{vision token indices}} A^{l,h}_{S,k}$$9 (“$A^{l,h} \in \mathbb{R}^{S \times S}$0-Component Dynamization”).
• Algorithms such as Adaptive-Binary and Greedy-Dual are proven $A^{l,h} \in \mathbb{R}^{S \times S}$1- and $A^{l,h} \in \mathbb{R}^{S \times S}$2-competitive, outperforming uniform-threshold compaction. Adaptive-Binary merges are triggered based on dynamically adapting to batch sizes and run weights, while Greedy-Dual ensures the maximum number of runs never exceeds $A^{l,h} \in \mathbb{R}^{S \times S}$3 by merging those with the highest accumulated credits (Mathieu et al., 2020).

Empirical studies show that these nonuniform policies exploit large or heterogeneous batch arrivals by deferring or accelerating compaction adaptively, reducing write amplification by up to an order of magnitude compared to uniform compaction, especially when data skew and arrival patterns are highly nonuniform.

5. Empirical Results and Impact

Multiple studies confirm that nonuniform per-head compaction yields superior efficiency and minimal accuracy degradation:

Domain/System	Speedup / Savings	Quality Tradeoff / Notes	Reference
LVLMs (PLPHP)	18% decoding speedup	0.46% avg. drop in ROUGE-L/CIDEr; gains in multi-image tasks	(Meng et al., 20 Feb 2025)
Transformer KV Compaction	Up to 50x cache reduction	1–2 pt QA accuracy gain vs. uniform; closes gap to full context	(Zweiger et al., 18 Feb 2026)
LSM-trees	Up to 10x write reduction	Optimal $A^{l,h} \in \mathbb{R}^{S \times S}$4-competitive bound; avg read/write costs lower than binomial/binary	(Mathieu et al., 2020)

By allocating compaction effort nonuniformly, systems concentrate resources where marginal utility is greatest—either by focusing on heads most vulnerable to accuracy loss, layers with high vision attentiveness, or runs (in storage systems) with disproportionate query or write impact.

6. Comparison to Uniform Compaction Strategies

Uniform compaction methods allocate an equal budget or quota to each head, layer, or run, regardless of actual usage, importance, or access statistics. Empirical and theoretical analyses indicate that this approach misallocates resources: “easy” heads (diminishing returns past a minimal quota) receive more resource than justified, while “critical” heads starved of quota suffer disproportionate accuracy or performance loss. Uniformly dropping tokens in all attention heads, or forcing all runs in LSM-trees to the same merge schedule, can induce significant degradation in target metrics, especially under extreme compaction (e.g., 50x and up), while nonuniform policies adapt and close a substantial portion of this quality gap (Zweiger et al., 18 Feb 2026, Mathieu et al., 2020).

7. Broader Context and Future Implications

The adoption of nonuniform per-head compaction is a direct response to the intrinsic diversity and specialization encountered in modern attention architectures and large-scale storage systems. Fine-grained, context- and head-adaptive compaction is likely to become foundational in large model deployment, multimodal reasoning, and real-time data management workloads. A plausible implication is that further integration of attention statistics, query/load analytics, and even reinforcement-learned compaction schedules will enhance system flexibility and cost-effectiveness across both AI and database domains.

---

References:

• [PLPHP: Per-Layer Per-Head Vision Token Pruning for Efficient Large Vision-LLMs, (Meng et al., 20 Feb 2025)]
• [Fast KV Compaction via Attention Matching, (Zweiger et al., 18 Feb 2026)]
• [Competitive Data-Structure Dynamization, (Mathieu et al., 2020)]