---
title: Nonuniform Per-Head Compaction
url: https://www.emergentmind.com/topics/nonuniform-per-head-compaction
type: topic
---

# Nonuniform Per-Head Compaction

Nonuniform per-head compaction refers to a class of algorithms and policies that allocate compaction resources or pruning quotas independently to each component ("head") in a multi-headed architecture or data system, instead of applying a uniform compaction or pruning rate across all heads. This strategy has achieved state-of-the-art efficiency in transformer-based models, large vision-language models (LVLMs), and large-scale data structures such as LSM-trees, by acknowledging heterogeneity in head specialization, attentional behavior, and sensitivity to compaction. Empirical and theoretical results demonstrate that nonuniform per-head compaction can yield substantial cost reductions, latency improvements, and negligible or sometimes even improved downstream accuracy when compared to uniform schemes [2502.14504][2602.16284][2011.02615].

## 1. Principles of Nonuniform Per-Head Compaction

Nonuniform per-head compaction operates on the observation that different heads—whether they are transformer attention heads, storage components, or data structure partitions—have highly variable utility and responsivity to compaction. In transformer and LVLM architectures, attention heads focus on different modalities and spatial or contextual regions, and their importance varies non-monotonely across layers and heads. Consequently, pruning (in neural nets) or compacting (in storage systems) should be targeted according to per-head, per-layer contextual importance and not distributed equally.

A canonical example is the “Vision Token Re-attention” phenomenon in LVLM decoders: lower or intermediate layers may attend more strongly to visual tokens, while upper layers’ attention shifts to text or abstract features, and attention heads specialize to distinct segments of input [2502.14504]. In data systems such as the LSM-tree model in databases, compaction decisions are best made based on nonuniform batch input and query statistics across “heads” (run levels), rather than a fixed, equal policy [2011.02615].

## 2. Methodologies in Transformer and Vision-Language Models

In large vision-language models, nonuniform per-head compaction typically involves two structurally coupled stages:

1. **Layer-Level Retention Rate Allocation:** For each decoder layer $l$, the average attention to vision tokens, denoted $\gamma^l$, is computed as
   $$
   \gamma^l = \frac{1}{H} \sum_{h=1}^H \sum_{k \in \text{vision token indices}} A^{l,h}_{S,k}
   $$
   with $A^{l,h} \in \mathbb{R}^{S \times S}$ the attention weights for head $h$ in layer $l$. The retention fraction $r^l$ for each layer is dynamically set based on $\gamma^l$ using thresholds $(\alpha, \beta, r, \Delta r)$:
   $$
   r^l =
   \begin{cases}
     r + \Delta r & \text{if } \gamma^l \geq \alpha \\
     r - \Delta r & \text{if } \gamma^l < \beta  \\
     r & \text{otherwise}
   \end{cases}
   $$
   For example, with $(\alpha=0.25, \beta=0.1, r=0.4, \Delta r=0.3)$, up to 70% of vision tokens can be retained in highly attentive layers, but only 10% in nearly vision-indifferent layers.

2. **Head-Level Pruning:** Each head $h$ within a layer $l$ selects exactly $K_j^{l,h} = \lceil r^l S_j^{(I)} \rceil$ tokens for each image $j$, prioritizing tokens with the highest attention from the “generation token.” A binary mask $M^{l,h}$ is constructed so only those vision tokens and all text tokens remain in the key-value cache of each head. Pruning is then applied per head via:
   $$
   K^{l,h} \leftarrow K^{l,h}[M^{l,h}=1], \quad V^{l,h} \leftarrow V^{l,h}[M^{l,h}=1]
   $$
   This ensures fine-grained, nonuniform compaction, with each head retaining a minimal but sufficient subset of tokens tailored to its actual usage [2502.14504].

## 3. Attention Matching and Per-Head KV Compaction

Attention Matching formulates nonuniform per-head compaction as an optimization problem that directly matches the outputs and mass of the attention mechanism for each head:
- Each head $(\ell,h)$ has its original key-value cache $(K_{\ell,h}, V_{\ell,h})$ of length $T$, and compacts this to $(C_{k,\ell,h}, \beta_{\ell,h}, C_{v,\ell,h})$ of length $t_{\ell,h} \ll T$.
- The loss function sums two terms: the squared difference in attention “mass” and in attention outputs over a reference query set:
  $$
  L_\text{total} = \sum_{i=1}^n (m_i - m'_i)^2 + \sum_{i=1}^n \| y_i - x_i \|_2^2
  $$
  where $m_i$ and $m'_i$ are original and compacted masses; $y_i$, $x_i$ are original and compacted attention outputs.
- Nonuniformity enters as the per-head compaction budget $t_{\ell,h}$, allocated by optimizing per-head loss curves (sensitivity) under a global quota.

Efficient closed-form solutions are available for fitting the value and bias parameters per head (using nonnegative least squares for biases and ordinary least squares for values), and practical heuristics (e.g., Orthogonal Matching Pursuit and highest-attention selection) let heads independently choose their compacted key sets. The global compaction ratio is met by greedily allocating units of quota to heads with the highest marginal benefit, yielding a nonuniform head distribution [2602.16284].

## 4. Nonuniform Compaction in Data Structures and LSM-Trees

In log-structured merge (LSM) trees and related databases, nonuniform per-head compaction is modeled as an online set-cover problem:
- Each batch flush $I_t$ is a new “head” or run, and the compaction policy must minimize either total build+query cost (“Min-Sum Dynamization”) or build cost under a query cap $k$ (“$k$-Component Dynamization”).
- Algorithms such as Adaptive-Binary and Greedy-Dual are proven $\Theta(\log^* m)$- and $k$-competitive, outperforming uniform-threshold compaction. Adaptive-Binary merges are triggered based on dynamically adapting to batch sizes and run weights, while Greedy-Dual ensures the maximum number of runs never exceeds $k$ by merging those with the highest accumulated credits [2011.02615].

Empirical studies show that these nonuniform policies exploit large or heterogeneous batch arrivals by deferring or accelerating compaction adaptively, reducing write amplification by up to an order of magnitude compared to uniform compaction, especially when data skew and arrival patterns are highly nonuniform.

## 5. Empirical Results and Impact

Multiple studies confirm that nonuniform per-head compaction yields superior efficiency and minimal accuracy degradation:

| Domain/System                  | Speedup / Savings         | Quality Tradeoff / Notes                   | Reference      |
|------------------------------- |--------------------------|--------------------------------------------|---------------|
| LVLMs (PLPHP)                  | 18% decoding speedup      | 0.46% avg. drop in ROUGE-L/CIDEr; gains in multi-image tasks | [2502.14504]  |
| Transformer KV Compaction      | Up to 50x cache reduction | 1–2 pt QA accuracy gain vs. uniform; closes gap to full context | [2602.16284]  |
| LSM-trees                      | Up to 10x write reduction | Optimal $k$-competitive bound; avg read/write costs lower than binomial/binary | [2011.02615]  |

By allocating compaction effort nonuniformly, systems concentrate resources where marginal utility is greatest—either by focusing on heads most vulnerable to accuracy loss, layers with high vision attentiveness, or runs (in storage systems) with disproportionate query or write impact.

## 6. Comparison to Uniform Compaction Strategies

Uniform compaction methods allocate an equal budget or quota to each head, layer, or run, regardless of actual usage, importance, or access statistics. Empirical and theoretical analyses indicate that this approach misallocates resources: “easy” heads (diminishing returns past a minimal quota) receive more resource than justified, while “critical” heads starved of quota suffer disproportionate accuracy or performance loss. Uniformly dropping tokens in all attention heads, or forcing all runs in LSM-trees to the same merge schedule, can induce significant degradation in target metrics, especially under extreme compaction (e.g., 50x and up), while nonuniform policies adapt and close a substantial portion of this quality gap [2602.16284][2011.02615].

## 7. Broader Context and Future Implications

The adoption of nonuniform per-head compaction is a direct response to the intrinsic diversity and specialization encountered in modern attention architectures and large-scale storage systems. Fine-grained, context- and head-adaptive compaction is likely to become foundational in large model deployment, multimodal reasoning, and real-time data management workloads. *A plausible implication is* that further integration of attention statistics, query/load analytics, and even reinforcement-learned compaction schedules will enhance system flexibility and cost-effectiveness across both AI and database domains.

---
**References**:  
- [PLPHP: Per-Layer Per-Head Vision Token Pruning for Efficient Large Vision-Language Models, 2502.14504]  
- [Fast KV Compaction via Attention Matching, 2602.16284]  
- [Competitive Data-Structure Dynamization, 2011.02615]

Source: https://www.emergentmind.com/topics/nonuniform-per-head-compaction