MoVE: Scalable Memory for Transformers

• Mixture-of-Value-Embeddings (MoVE) is a mechanism that decouples memory capacity from network compute by using a global bank of value embeddings.
• The approach integrates a differentiable per-token routing mechanism, allowing all attention layers to access increased memory with negligible computational cost.
• Experimental results in text and image generation demonstrate that MoVE outperforms standard and layer-wise memory baselines while remaining highly efficient.

Mixture-of-Value-Embeddings (MoVE) is a mechanism for scaling parametric memory in autoregressive models that decouples memory capacity from network compute, enabling substantially increased factual or pattern memory at negligible computational overhead. Through the use of a global, shared, trainable bank of value embeddings accessed by all attention layers—and a differentiable, per-token-routing mechanism—MoVE enables the construction of "memory-dense" Transformer models. This architecture achieves consistent improvements over both standard and local layer-wise memory baselines in text and image generation domains, establishing a fundamentally new axis for Transformer scaling distinct from changes in network depth or width (Li, 30 Jan 2026).

1. Motivation and Conceptual Foundation

Conventional autoregressive Transformers tightly couple their capacity for factual or pattern memory to the product of network depth and width: increasing parameter count for memory inevitably increases computation, due to the structure of dense feedforward and attention layers. Traditional approaches to expanding model knowledge, such as simply enlarging the model, thus suffer from a linear scaling of computational overhead (FLOPs) with model capacity.

MoVE addresses this bottleneck by introducing a single, shared, global bank of value vectors. This bank is accessible to all attention layers, and its capacity—the number of value slots—is not tied to network depth or size. The decoupling allows memory expansion by increasing only the number of learnable value slots, without impacting the number of forward-pass operations committed by the network backbone (Li, 30 Jan 2026).

Conceptually, three architectures can be contrasted:

• Standard value projection: Each attention layer computes $V^\ell = X W_V^\ell$, encoding all memory in layer weights.
• Layer-wise memory (LaVE): Each layer maintains a small, independent memory and corresponding gate, scaling memory with the number of layers.
• MoVE: Employs a single, global, per-token bank with routing, allowing true independence between network depth and memory capacity.

2. Architectural Details

In MoVE, the standard value computation in multi-head attention is replaced by a mixture of a global embedding bank and the standard per-layer path. For attention head $h$ at token $t$, the output is:

$$V_{\mathcal{S}^{(h)}} = g_{t,0}^{(h)} V^{(h)} + \sum_{i=1}^M g_{t,i}^{(h)} \mathcal{M}_{t,i}^{(h)}$$

where $V^{(h)}$ is the conventional value projection, $\mathcal{M}_{t,i}^{(h)}$ are concept vectors drawn from the global memory tensor $$\mathcal{E}\in\mathbb{R}^{|\mathcal V| \times M \times d}$$ (for vocabulary or codebook size $|\mathcal V|$, slot count $M$, and dimension $d$), and $h$0 are soft gate values.

For each token $h$1, its slice of the bank $h$2 is retrieved, then routed into the attention computation.

The gating mechanism employs a scaled-sigmoid router, producing gate values $h$3 for $h$4, with input $h$5 projected by $h$6. The $h$7 gate corresponds to the standard value path, and $h$8 to slots in the global bank.

3. Capacity Scaling and Computational Overhead

MoVE parametrically scales memory size as $h$9, with $t$0 chosen independently of model depth $t$1 or width $t$2. Compute cost per token for MoVE is:

$t$3

compared to the standard self-attention compute

$t$4

where $t$5 is hidden size, $t$6 is number of heads, $t$7 value slots, and $t$8 is sequence length. The additional computational overhead is

$t$9

which remains below 2% for typical Transformer parameters ($$V_{\mathcal{S}^{(h)}} = g_{t,0}^{(h)} V^{(h)} + \sum_{i=1}^M g_{t,i}^{(h)} \mathcal{M}_{t,i}^{(h)}$$0, $$V_{\mathcal{S}^{(h)}} = g_{t,0}^{(h)} V^{(h)} + \sum_{i=1}^M g_{t,i}^{(h)} \mathcal{M}_{t,i}^{(h)}$$1, $$V_{\mathcal{S}^{(h)}} = g_{t,0}^{(h)} V^{(h)} + \sum_{i=1}^M g_{t,i}^{(h)} \mathcal{M}_{t,i}^{(h)}$$2, $$V_{\mathcal{S}^{(h)}} = g_{t,0}^{(h)} V^{(h)} + \sum_{i=1}^M g_{t,i}^{(h)} \mathcal{M}_{t,i}^{(h)}$$3), illustrating that memory can be made orders of magnitude larger without significant increases in compute (Li, 30 Jan 2026).

4. Experimental Validation and Results

MoVE was evaluated in text generation (nanochat framework, FineWeb-Edu dataset) and image generation (LlamaGen, ImageNet-1K), as well as with Multi-Head Latent Attention (MLA).

Experimental Model and Baseline Overview

Domain	Models	Baselines	Evaluation Metric
Text	D12, D20, D32	Standard, LaVE×1, LaVE×2	Bits-Per-Byte (BPB)
Images	GPT-B, GPT-L	Standard, LaVE×1	FID, IS, Precision, Recall
MLA	D12, D20	MLA, MLA+LaVE	BPB

Key Results

• Text generation: Consistent BPB reductions at all scales. Example: D12 standard 0.838 → MoVE×8 0.797 (Δ = –0.041).
• Image generation: MoVE achieves lower FID scores. GPT-B: Standard 6.53 → MoVE×1 5.62 (Δ = –0.91); GPT-L: 3.47 → 3.10.
• MLA: D12-MLA: base 0.8826 → MoVE×32 0.8690.
• Ablation studies: Use of global memory (MoVE) outperforms local (LaVE), and gating the standard value path ($$V_{\mathcal{S}^{(h)}} = g_{t,0}^{(h)} V^{(h)} + \sum_{i=1}^M g_{t,i}^{(h)} \mathcal{M}_{t,i}^{(h)}$$4) provides an additional ~0.005 BPB gain at higher memory densities.

5. Comparative Analysis with Related Architectures

MoVE contrasts with standard dense scaling and local memory approaches:

• Standard scaling: Expanding dense vectors increases both parametric memory and compute proportionally; all memory is embedded in layer weights.
• Layer-wise value embeddings (LaVE): Distributes small, layer-local memories with per-layer gating; total memory still scales with layer count and offers saturated gains at high densities.
• MoVE: Allows for "super-dense" memory regimes ($$V_{\mathcal{S}^{(h)}} = g_{t,0}^{(h)} V^{(h)} + \sum_{i=1}^M g_{t,i}^{(h)} \mathcal{M}_{t,i}^{(h)}$$5), exceeding the saturation point of LaVE, with further performance gains and qualitative evidence of slot specialization, especially for tokens with high polysemy or complex syntax (Li, 30 Jan 2026).

6. Limitations and Future Directions

Notable limitations include the stress imposed on memory bandwidth and hardware (e.g., HBM) due to large bank retrievals, despite the minimal arithmetic overhead. The parameter-efficiency per added slot is also lower than for equivalent increases in dense FFN capacity.

Future work directions proposed include hybridization with Mixture-of-Experts (MoE)—combining block-level sparse expert routing for complex reasoning with MoVE's slot-level memory for factual storage. Other avenues include more compact or adaptive bank designs, sharing slots across semantically related tokens, head-wise gating projections, or slot pruning to optimize bandwidth and parameter utilization (Li, 30 Jan 2026).

7. Applications and Use Cases

MoVE has been validated in both text and image generation, facilitating "memory-dense" models for generative applications without incurring significant compute cost increases. The approach is applicable to both standard Transformer and MLA architectures. Cases where MoVE provides the greatest benefit include settings requiring extremely high memory capacity—for instance, tasks with high token polysemy or sensitivity to rare patterns—where previous memory scaling approaches saturate (Li, 30 Jan 2026).

A plausible implication is that MoVE, by decoupling memory from computation, enables new network scaling regimes addressing domains or tasks characterized by vast ground-truth complexity while remaining within practical runtime budgets.