Token-to-Head Contribution Score

• Token-to-head contribution score is a quantitative metric that defines the impact of each input token and attention head in Transformer models.
• It enables dynamic pruning and adaptive routing by selectively retaining only the most critical tokens and heads to optimize computational efficiency.
• Variants like SpAtten and MoH demonstrate how this score drives advances in model compression, interpretability, and hardware-aware acceleration.

A token-to-head contribution score quantifies the degree to which each input token, and each attention head, affects the output of a Transformer-based neural network. This concept has become central in contemporary research on efficiency, interpretability, and model compression in attention-based architectures. The score captures either the cumulative effect of tokens across attention heads, the routing of tokens to specific heads, or the selective preservation or pruning of tokens and heads based on their empirical contributions during inference. Multiple variants, including algorithmic, architectural, and hardware-level formulations, have been proposed to optimize large-scale models for efficiency while maintaining accuracy.

1. Definition and Architectural Motivation

Transformer architectures feature multi-headed self-attention modules, where each head computes attention outputs over the input tokens. However, not all tokens and not all heads contribute equally to the network's final output or to its performance on a downstream task. The token-to-head contribution score is a quantitative measure—usually computed per token and per head in each layer—that reflects the importance of these units for subsequent computations.

In SpAtten (Wang et al., 2020), importance is explicitly accumulated:

• For tokens, the score is the sum of attention probabilities over all query positions and heads.
• For heads, the score is the L1 norm of the absolute attention outputs after value aggregation and linear projection.

This principle underpins pruning, sparsification, and acceleration schemes, ensuring redundant computation is minimized without degrading model capacity.

2. Calculation of Contribution Scores

The calculation of token and head contribution scores can take several forms, which are frequently context- and architecture-specific:

Variant	Token Contribution Computation	Head Contribution Computation
SpAtten	$$\displaystyle s_t[\mathrm{token\_id}] += a[\mathrm{head}, q, \mathrm{token\_id}]$$	$$\displaystyle s_h[\mathrm{head\_id}] += |E[\mathrm{head\_id}, q, d]|$$
MoH (Jin et al., 15 Oct 2024)	Per-token, softmax-routed head-weighting ($g_i$ per token, per head)	Router computes soft activation $g_i$
PLPHP (Meng et al., 20 Feb 2025)	Head-wise top-$K$ selection over softmax attention weights	Not directly scored; pruning is token-wise per head
HAVE (Tong et al., 8 Sep 2025)	Aggregate attention × value norm, renormalized per-head and then across heads	Head-adaptive gating (instance-adaptive softmax)
CAOTE (Goel et al., 18 Apr 2025)	L2-normed output error on token removal, scaled by retention prob.	Not applicable (token-focused)

Mathematically, tokens are scored according to their cumulative influence measured by attention probabilities, modified in some schemes by multiplying with the norm of the value vector or by explicit routing weights.

3. Pruning, Routing, and Dynamic Allocation

Practical usage of the token-to-head contribution score centers on model compression (pruning) and efficient routing:

• Cascade token and head pruning (SpAtten): After each attention layer, a top-$k$ selection (via a hardware-accelerated engine) discards a specified fraction of least important tokens/heads. These removals persist across layers, reducing overall memory and computation costs.
• Adaptive routing (MoH): Each token dynamically selects the most relevant heads for itself through a gating mechanism, which assigns a softmax-normalized contribution score $g_i$ to each head. A fraction of heads are always active (shared/expert), while others are toggled per-token via a top-K mechanism.
• Per-head token pruning (PLPHP): Each attention head keeps only the tokens with top attention scores per head per layer, constructing a fine-grained, layer- and head-specific token retention map.
• Contextual head weighting (HAVE): Instance- and input-specific weighting of heads amplifies heads that deliver context-relevant evidence for each decoding step, relying on both attention and value vector statistics.

Such dynamic pruning and routing inherently require reliable per-token and per-head metrics, for which the token-to-head contribution score is foundational.

4. Hardware and Algorithmic Implementation

To operationalize dynamic pruning and routing in real-world systems, hardware-aware acceleration is often necessary:

• SpAtten’s top-$k$ engine uses a quick-select partition and FIFO buffers for efficient selection among tokens and heads, supporting online execution with $O(n)$ expected time.
• Progressive quantization: For memory-bound inference, only most significant bits (MSBs) are fetched for QKV vectors; additional bits fetched only when output confidence is low (softmax is flat). This further saves DRAM access, as quantization error is dampened for peaked distributions, minimizing bitwidth without accuracy loss.

Architectural modifications in accelerators, including crossbars to route memory and FIFO buffers for token/head queues, are incorporated to support these adaptive operations in real-time.

5. Empirical Impact and Efficiency Gains

The use of token-to-head contribution scores has yielded substantial empirical benefits:

• SpAtten demonstrates a 10x average reduction in DRAM access, speedups up to 162x over GPUs, and energy savings up to 4059x over Xeon CPUs, across 30 NLP benchmarks, without measurable accuracy loss.
• Cascade head/token pruning removes up to 80–90% of computation for uninformative tokens/heads.
• PLPHP realizes an 18% decoding speedup and over 50% lower KV-cache size for large vision-LLMs, with only a 0.46% performance drop, demonstrating robustness on both single- and multi-image LVLM tasks.
• MoH outperforms classical multi-head attention despite using only 50–90% of the heads, achieving increased efficiency with accuracy gains in vision and LLMs.

These improvements are made possible because the contribution score-based mechanisms effectively retain only the units that meaningfully impact model output.

6. Theoretical Considerations and Design Trade-offs

The choice of scoring scheme and pruning threshold introduces trade-offs among computational savings, memory efficiency, and accuracy preservation:

• Cumulative attention-based scores (SpAtten) are simple to compute and robust but may not capture token redundancy when attention distributions are diffuse.
• Weighting by attention×value (CAOTE, HAVE) considers information content of retained tokens, preventing the eviction of low-attention but semantically crucial tokens.
• Softmax-normalized router scores (MoH) provide token-specific, context-aware head allocation, but require specialized routing networks and introduce additional parameters or control logic.
• Hardware-coupled dynamic pruning (SpAtten) leverages online importance ranking for flexible, input-adaptive pruning, which is particularly advantageous for real-time inference on resource-constrained devices.

The optimal strategy generally balances cumulative importance, computational simplicity, and hardware efficiency, with dynamic, instance-adaptive approaches being preferable for tasks exhibiting strong input variance in token/head utility.

7. Broader Implications and Future Directions

Token-to-head contribution scoring informs ongoing research in model interpretability, adaptive computation, and neural network hardware co-design. It provides a quantifiable basis for:

• Interpretability analyses, by surfacing which input tokens or heads contribute most to opaque model decisions;
• Training efficiency improvements, by focusing learning or fine-tuning on consistently high-contribution tokens/heads;
• The design of sparsity-aware hardware accelerators and compression schemes, which co-evolve with algorithmic advances in dynamic routing and pruning.

Future research may focus on refining these scores for finer granularity, learning the pruning thresholds adaptively, and fusing contribution scoring with uncertainty quantification or causal attribution techniques in models with more complex computation graphs.

---

In summary, the token-to-head contribution score is a pivotal construct driving recent advances in efficient, interpretable, and scalable attention-based models, enabling real-time token and head selection at both algorithmic and hardware levels, while maintaining accuracy and adaptability across a wide range of tasks and model families (Wang et al., 2020).