Bank of Values: Transformer Innovation

• Bank of Values (BoV) is an architectural modification for transformer attention layers that uses static, token-specific lookups to compute value vectors in deep layers.
• It replaces the conventional, dense context-dependent value projections with a per-layer learnable lookup table, preserving token identity and reducing computational overhead.
• Empirical results demonstrate that BoV enhances model accuracy and efficiency in large language models, yielding improvements in tasks such as SQuAD and CoQA.

The Bank of Values (BoV) is an architectural innovation for transformer attention layers that replaces the conventional, context-dependent computation of value vectors in the deepest model layers with static, token-specific, context-free lookups. This mechanism aims to preserve token identity information while simultaneously delivering efficiency improvements in compute and memory. BoV is introduced and systematically evaluated in “Do Value Vectors in Deep Layers Need Context from the Residual Stream?” (He et al., 1 Jun 2026), where it demonstrates both improved model quality and significant efficiency gains in LLMs.

1. Motivation and Theoretical Rationale

In standard transformer attention, the value vectors $V$ are computed as dense projections of the residual stream:

$V = x_i W_V$

where $x_i \in \mathbb{R}^d$ is the residual input at position $i$ and $W_V$ is a learned projection. While this enables rich, context-dependent representations across layers, it introduces two key shortcomings in deeper layers:

• Dilution of token identity: Repeated cross-token mixing through many stacked attention layers causes $V$ to lose the context-free, token-specific information present at the model’s input.
• Inference and cache overhead: At inference time, $V$ must be calculated and cached at every layer and position, resulting in $O(T \cdot d^2)$ computation and memory consumption per layer, where $T$ is context length and $d$ is hidden dimensionality.

Empirical studies revealed that introducing a context-free value component in deep layers produces nearly all measurable performance gains, rendering the additional context-dependent part nearly redundant for aggregate performance. The core intuition is that since value vectors are passively weighted by attention (not actively mixed across tokens), it is more beneficial in deep layers to supply a clean, token-specific signal rather than a highly entangled contextual one (He et al., 1 Jun 2026).

2. BoV Architectural Modifications

BoV revises only the last $V = x_i W_V$0 layers of a transformer with $V = x_i W_V$1 total layers. In these layers:

• Query and key computation: Unchanged, as projections of the residual stream.

$V = x_i W_V$2

• Value computation via lookup: Instead of $V = x_i W_V$3, a per-layer, learnable lookup table $V = x_i W_V$4 is introduced, where $V = x_i W_V$5 is vocabulary size. For each token position $V = x_i W_V$6 with token index $V = x_i W_V$7:

$V = x_i W_V$8

with $V = x_i W_V$9 a learned, per-layer scalar.

• Attention mechanism: Standard attention is performed, but with $x_i \in \mathbb{R}^d$0 replacing context-dependent $x_i \in \mathbb{R}^d$1.

$x_i \in \mathbb{R}^d$2

where $x_i \in \mathbb{R}^d$3, and $x_i \in \mathbb{R}^d$4 is the number of heads.

• Output projection: Concatenation of attended head outputs, output projection, and residual addition remain as usual.

This removes the need for $x_i \in \mathbb{R}^d$5 in these layers and replaces token-dependent value computation with a static lookup followed by scalar scaling (He et al., 1 Jun 2026).

3. Formal Description and Memory Analysis

The BoV mechanism for layer $x_i \in \mathbb{R}^d$6 ($x_i \in \mathbb{R}^d$7) is defined by:

• $x_i \in \mathbb{R}^d$8: input residuals
• $x_i \in \mathbb{R}^d$9: projections
• $i$0: per-layer lookup table
• $i$1: learnable scalar

Value vectors at each position:

$i$2

Memory savings for the value cache per layer:

$i$3

For $i$4, BoV provides net memory reduction, as the dense cache is replaced with a (potentially sparse) static table; only accessed rows need active memory. As the context length $i$5 grows, benefits increase (He et al., 1 Jun 2026).

4. Implementation Aspects

A BoV attention layer operates as follows (notation as above):

1. Compute $i$6, $i$7 via projections from $i$8.
2. Lookup $i$9 for each position from $W_V$0, apply $W_V$1 scaling.
3. Reshape $W_V$2, $W_V$3, $W_V$4 for multi-head computation.
4. For autoregressive inference, append $W_V$5, $W_V$6 to caches.
5. Compute attention weights and apply them.
6. Concatenate and linearly project the outputs, then add to the residual.

The PyTorch-style pseudocode below captures the essential modifications: $T$4 Key points:

• $W_V$7 is implemented as an nn.Embedding, enabling on-demand and memory-sparse lookups.
• $W_V$8 is entirely absent from BoV layers.
• There is no need to cache value projections for past tokens beyond the current batch lookup (He et al., 1 Jun 2026).

5. Empirical Evaluation and Ablation Findings

Ablation studies conducted at $W_V$9M and $V$0M scale under fixed compute budgets validate the BoV approach:

• Substitutive vs. Additive: Entirely replacing $V$1 with a context-free variant outperforms additive combinations of context-free and context-dependent $V$2.
• Learnable scaling: Per-layer, learnable scalar coefficients $V$3 lower loss compared to fixed scalings (e.g., $V$4).
• Lookup source: Layer-specific context-free values ($V$5 or dedicated $V$6) surpass shared, early-layer sources.
• Layer targeting: Restricting BoV substitution to only the final $V$7 layers yields gains; replacing value computation in shallow layers degrades performance.

Summary of validation results (bits per byte, 135M model):

Configuration	Validation Loss (BPB)
Baseline: $V$8	0.854
$V$9, substitute last 4 layers, learnable $V$0	0.845
BoV, lookup $V$1 in last 4 layers, learnable $V$2	0.845

At 780M scale:

• Baseline: 0.722 BPB, CORE score 0.260
• BoV: 0.714 BPB, CORE score 0.272
• Largest gains observed on SQuAD (0.40 → 0.42), CoQA (0.29 → 0.32), and symbolic tasks; BoV matches or exceeds competitive token-lookup approaches while consuming less compute (He et al., 1 Jun 2026).

6. Efficiency and Resource Utilization

Compute savings: Standard $V$3 computation is $V$4 per token per layer; BoV lookups require only $V$5 gather, netting a $V$6 FLOP reduction in the last $V$7 layers. Using $V$8, $V$9, BoV yields approximately $O(T \cdot d^2)$0 fewer multiplies per token.

Memory tradeoff: Standard attention maintains a $O(T \cdot d^2)$1 value cache per layer (total $O(T \cdot d^2)$2), whereas BoV substitutes this with a fixed $O(T \cdot d^2)$3 table per layer, eliminating per-token cache requirements for those layers. For $O(T \cdot d^2)$4, the reduction in GPU memory usage can be substantial; only lookup rows are brought to device memory on demand. This can also benefit distributed and memory-constrained inference workflows (He et al., 1 Jun 2026).

7. Hyperparameter Selection and Deployment Considerations

Experimental and scaling evidence support the following recommendations for BoV:

• Restrict BoV to the last $O(T \cdot d^2)$5 transformer layers ($O(T \cdot d^2)$6).
• Use a learnable, unbounded scalar $O(T \cdot d^2)$7 per affected layer, rather than fixing it.
• Maintain separate $O(T \cdot d^2)$8 tables for each BoV layer; do not share across depths.
• Initialize each $O(T \cdot d^2)$9 with $T$0, optionally RMS-normalized, to match baseline context-free $T$1 at start.
• Set the embedding dimensionality $T$2 of $T$3 to match the transformer’s hidden size.

Under these settings, BoV achieves lower validation loss, better accuracy on reading-comprehension and symbolic tasks, and reduced resource consumption compared to standard and previously proposed value-lookup methods (He et al., 1 Jun 2026).