---
title: 'Bank of Values: Transformer Innovation'
url: https://www.emergentmind.com/topics/bank-of-values-bov
type: topic
---

# Bank of Values: Transformer Innovation

The Bank of Values (BoV) is an architectural innovation for transformer attention layers that replaces the conventional, context-dependent computation of value vectors in the deepest model layers with static, token-specific, context-free lookups. This mechanism aims to preserve token identity information while simultaneously delivering efficiency improvements in compute and memory. BoV is introduced and systematically evaluated in “Do Value Vectors in Deep Layers Need Context from the Residual Stream?” [2606.02780], where it demonstrates both improved model quality and significant efficiency gains in large language models (LLMs).

## 1. Motivation and Theoretical Rationale

In standard transformer attention, the value vectors $V$ are computed as dense projections of the residual stream:
$$
V = x_i W_V
$$
where $x_i \in \mathbb{R}^d$ is the residual input at position $i$ and $W_V$ is a learned projection. While this enables rich, context-dependent representations across layers, it introduces two key shortcomings in deeper layers:

- **Dilution of token identity**: Repeated cross-token mixing through many stacked attention layers causes $V$ to lose the context-free, token-specific information present at the model’s input.
- **Inference and cache overhead**: At inference time, $V$ must be calculated and cached at every layer and position, resulting in $O(T \cdot d^2)$ computation and memory consumption per layer, where $T$ is context length and $d$ is hidden dimensionality.

Empirical studies revealed that introducing a context-free value component in deep layers produces nearly all measurable performance gains, rendering the additional context-dependent part nearly redundant for aggregate performance. The core intuition is that since value vectors are passively weighted by attention (not actively mixed across tokens), it is more beneficial in deep layers to supply a clean, token-specific signal rather than a highly entangled contextual one [2606.02780].

## 2. BoV Architectural Modifications

BoV revises only the last $L/3$ layers of a transformer with $L$ total layers. In these layers:

- **Query and key computation**: Unchanged, as projections of the residual stream.
$$
Q = x_i W_Q,\quad K = x_i W_K
$$
- **Value computation via lookup**: Instead of $V_{\text{ctx}} = x_i W_V$, a per-layer, learnable lookup table $E_v \in \mathbb{R}^{|\mathcal{V}| \times d}$ is introduced, where $|\mathcal{V}|$ is vocabulary size. For each token position $p$ with token index $i_p$:
$$
V_b(p) = \gamma_v \cdot E_v[i_p]
$$
with $\gamma_v \in \mathbb{R}$ a learned, per-layer scalar.

- **Attention mechanism**: Standard attention is performed, but with $V_b$ replacing context-dependent $V$.
$$
\alpha = \operatorname{softmax}\left(\frac{Q K^\top}{\sqrt{d_h}}\right)
$$
where $d_h = d/H$, and $H$ is the number of heads.

- **Output projection**: Concatenation of attended head outputs, output projection, and residual addition remain as usual.

This removes the need for $W_V$ in these layers and replaces token-dependent value computation with a static lookup followed by scalar scaling [2606.02780].

## 3. Formal Description and Memory Analysis

The BoV mechanism for layer $i$ ($i > 2L/3$) is defined by:
- $x_i \in \mathbb{R}^{T \times d}$: input residuals
- $W_Q, W_K, W_O \in \mathbb{R}^{d \times d}$: projections
- $E_v^{(i)} \in \mathbb{R}^{|\mathcal{V}| \times d}$: per-layer lookup table
- $\gamma_v^{(i)}$: learnable scalar

Value vectors at each position:
$$
V_b(p) = \gamma_v^{(i)} \cdot E_v^{(i)}[i_p]
$$

Memory savings for the value cache per layer:
$$
\Delta_{\text{mem}} = \frac{L}{3} \cdot d \cdot (T - |\mathcal{V}| )
$$
For $T > |\mathcal{V}|$, BoV provides net memory reduction, as the dense cache is replaced with a (potentially sparse) static table; only accessed rows need active memory. As the context length $T$ grows, benefits increase [2606.02780].

## 4. Implementation Aspects

A BoV attention layer operates as follows (notation as above):

1. Compute $Q$, $K$ via projections from $x$.
2. Lookup $V_b$ for each position from $E_v$, apply $\gamma_v$ scaling.
3. Reshape $Q$, $K$, $V_b$ for multi-head computation.
4. For autoregressive inference, append $K$, $V_b$ to caches.
5. Compute attention weights and apply them.
6. Concatenate and linearly project the outputs, then add to the residual.

The PyTorch-style pseudocode below captures the essential modifications:
```python
class BoVAttention(nn.Module):
    def __init__(self, d, H, vocab_size):
        super().__init__()
        self.W_Q = nn.Linear(d, d, bias=False)
        self.W_K = nn.Linear(d, d, bias=False)
        self.W_O = nn.Linear(d, d, bias=False)
        self.gamma_v = nn.Parameter(torch.ones(1))        # per-layer scalar
        self.E_v = nn.Embedding(vocab_size, d)            # context-free lookup

    def forward(self, x, token_ids, kv_cache=None):
        Q = self.W_Q(x)
        K = self.W_K(x)
        V_b = self.gamma_v * self.E_v(token_ids)          # context-free
        # multi-head split and caching omitted for brevity
        scores = torch.matmul(Q, K.transpose(-2,-1)) / math.sqrt(d_h)
        alpha = torch.softmax(scores, dim=-1)
        O = torch.matmul(alpha, V_b)
        return x + self.W_O(O)
```
Key points:
- $E_v$ is implemented as an `nn.Embedding`, enabling on-demand and memory-sparse lookups.
- $W_V$ is entirely absent from BoV layers.
- There is no need to cache value projections for past tokens beyond the current batch lookup [2606.02780].

## 5. Empirical Evaluation and Ablation Findings

Ablation studies conducted at $135$M and $780$M scale under fixed compute budgets validate the BoV approach:

- **Substitutive vs. Additive**: Entirely replacing $V$ with a context-free variant outperforms additive combinations of context-free and context-dependent $V$.
- **Learnable scaling**: Per-layer, learnable scalar coefficients $\gamma_v$ lower loss compared to fixed scalings (e.g., $0.5$).
- **Lookup source**: Layer-specific context-free values ($x_0 W_V$ or dedicated $E_v$) surpass shared, early-layer sources.
- **Layer targeting**: Restricting BoV substitution to only the final $L/3$ layers yields gains; replacing value computation in shallow layers degrades performance.

Summary of validation results (bits per byte, 135M model):

| Configuration                 | Validation Loss (BPB) |
|-------------------------------|----------------------|
| Baseline: $V_{\text{ctx}}$    | 0.854                |
| $x_0W_V$, substitute last 4 layers, learnable $\gamma_v$ | 0.845        |
| BoV, lookup $E_v$ in last 4 layers, learnable $\gamma_v$ | 0.845        |

At 780M scale:

- Baseline: 0.722 BPB, CORE score 0.260
- BoV: 0.714 BPB, CORE score 0.272
- Largest gains observed on SQuAD (0.40 → 0.42), CoQA (0.29 → 0.32), and symbolic tasks; BoV matches or exceeds competitive token-lookup approaches while consuming less compute [2606.02780].

## 6. Efficiency and Resource Utilization

**Compute savings**: Standard $V_{\text{ctx}}$ computation is $O(d^2)$ per token per layer; BoV lookups require only $O(d)$ gather, netting a $(L/3)\cdot(d^2-d)$ FLOP reduction in the last $L/3$ layers. Using $L=24$, $d=1536$, BoV yields approximately $1.8 \times 10^7$ fewer multiplies per token.

**Memory tradeoff**: Standard attention maintains a $T \cdot d$ value cache per layer (total $(L/3)\cdot T\cdot d$), whereas BoV substitutes this with a fixed $|\mathcal{V}| \cdot d$ table per layer, eliminating per-token cache requirements for those layers. For $T \gg |\mathcal{V}|$, the reduction in GPU memory usage can be substantial; only lookup rows are brought to device memory on demand. This can also benefit distributed and memory-constrained inference workflows [2606.02780].

## 7. Hyperparameter Selection and Deployment Considerations

Experimental and scaling evidence support the following recommendations for BoV:

- Restrict BoV to the last $L/3$ transformer layers ($i > 2L/3$).
- Use a learnable, unbounded scalar $\gamma_v$ per affected layer, rather than fixing it.
- Maintain separate $E_v$ tables for each BoV layer; do not share across depths.
- Initialize each $E_v[i]$ with $x_0[i] W_V$, optionally RMS-normalized, to match baseline context-free $V$ at start.
- Set the embedding dimensionality $d$ of $E_v$ to match the transformer’s hidden size.

Under these settings, BoV achieves lower validation loss, better accuracy on reading-comprehension and symbolic tasks, and reduced resource consumption compared to standard and previously proposed value-lookup methods [2606.02780].

Source: https://www.emergentmind.com/topics/bank-of-values-bov