Register Attention in Vision Transformers

• Register attention is a mechanism that adds learnable register tokens to transformer input sequences to act as global scratchpads.
• It decouples local patch processing from global context encoding, mitigating high-norm artifacts and enhancing performance in dense visual tasks.
• Empirical studies show that while in-distribution token replacements maintain model accuracy, zero-ablation leads to significant performance drops in both self-supervised and generative settings.

Register attention refers to the interaction pattern induced by learnable “register” tokens within the self-attention mechanism of vision transformers (ViTs) and diffusion transformers (DiTs). These tokens are appended to the input token sequence, enabling decoupling of global and local feature processing, structural buffering of dense features, and improved quality of both learned representations and downstream model outputs. Register attention has become critical in self-supervised visual pretraining (e.g., DINOv2+R, DINOv3) and pixel-space diffusion generation, fundamentally altering the interpretability, robustness, and performance of modern large-scale models.

1. Definition and Structural Role of Register Tokens

In standard ViT architectures, an image is divided into $P=H \times W$ patch tokens, and a global “[CLS]” token is prepended to encode whole-image information. DINOv2+registers and DINOv3 augment this scheme by injecting $R$ additional learnable register tokens per layer—typically, $R=4$ for ViTs and up to $M=32$ for pixel-space DiTs (Parodi et al., 15 Apr 2026, Starodubcev et al., 15 May 2026). Register tokens lack fixed spatial positions and do not correspond to any input region; instead, they serve as global scratchpads, absorbing computational burdens and preventing global information from leaking into local patch representations.

The principal motivations are:

• Mitigation of high-norm patch artifacts: Large ViTs without registers repurpose patch token slots to store global features, generating high-norm outliers and corrupted attention maps (Lappe et al., 9 May 2025, Parodi et al., 15 Apr 2026).
• Structural decoupling: Register tokens enable a clean separation between the channels processing local features (patches) and those aggregating global context, thereby improving dense (patch-level) task performance and model interpretability.

2. Mathematical Formulation of Register Attention

With $X \in \mathbb{R}^{N \times d}$ denoting patch token embeddings and $R \in \mathbb{R}^{M \times d}$ the register tokens, the augmented input is $\bar{X} = [X;R] \in \mathbb{R}^{(N+M) \times d}$. Each transformer block operates on $\bar{X}$, computing multi-head self-attention in the standard fashion:

$$Q = \bar{X}W^Q, \quad K = \bar{X}W^K, \quad V = \bar{X}W^V,$$

$$A = \mathrm{Softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right),$$

$R$0

After the attention operation, output embeddings are split back into patch and register streams. Only patch outputs are passed to the next block for standard visual tasks, while registers persist as auxiliary feature channels.

In DiTs, registers participate in attention but are excluded from the pixel-space diffusion loss. In most designs, registers are appended only in deeper layers ($R$1 to $R$2), specializing in outlier absorption and global context encoding (Starodubcev et al., 15 May 2026).

3. Register Ablation, Replacement, and Content Dependence

Ablation techniques:

• Zero-ablation: Sets register activations to $R$3 at each block, simulating their removal.
• In-distribution replacements:
  • Mean-substitution: Replace each register with the empirical mean over a calibration set.
  • Noise-substitution: Replace with Gaussian samples matching mean and variance of true register activations.
  • Register-shuffling: Randomly permute registers across images in a batch, breaking image-specificity but retaining plausible feature distributions (Parodi et al., 15 Apr 2026).

Empirical findings:

• Zero-ablation of registers in DINOv3 (ViT-S) causes steep performance drops (CLS $R$4 pp, segmentation $R$5 pp).
• All plausible in-distribution replacements incur negligible drops ($R$6 pp), despite genuinely perturbing internal representations (per-patch cosine $R$70.95–0.999 for mean, noise, shuffle vs. $R$80.58–0.61 for zeroing).
• The Jensen–Shannon divergence of attention patterns is $R$9 for mean-substitution but $R=4$0 for zero-ablation, indicating the latter is a strongly out-of-distribution intervention (Parodi et al., 15 Apr 2026).

This demonstrates that plausible register-like activations suffice for downstream performance; dependence on exact per-image register content is minimal in frozen-feature evaluations.

4. Decoupling of Local and Global Features

Register tokens induce a structural decoupling: global (whole-image) information is funneled through register channels, while patch tokens preserve detailed local features. Attentional mass in the final layer’s [CLS] attention is increasingly borne by registers as model size grows:

Model size	Patch mass	Register mass
Small	0.92 ± 0.03	0.08 ± 0.03
Base	0.85 ± 0.05	0.15 ± 0.05
Large	0.47 ± 0.12	0.53 ± 0.12
Giant	0.18 ± 0.20	0.82 ± 0.20

CKA similarity between global features and patch/reigster contributions further supports this: in small models, global features align with patch-only outputs (CKA$R=4$11.0), but in large models with registers, CKA$R=4$2 dominates (Lappe et al., 9 May 2025). One-shot ImageNet testing shows that global accuracy collapses if restricted to patches in large models, but is largely preserved via registers.

A similar phenomenon occurs in models lacking explicit registers; the [CLS] token with residual skip (the "skip path") inherits a register-like function, becoming the dominant route for global context (Lappe et al., 9 May 2025).

5. Impact on Diffusion Transformers and Feature Map Quality

Pixel-space DiTs benefit substantially from register attention:

• Registers absorb high-magnitude activations, regularizing patch channels.
• Empirically, appending $R=4$3 register tokens to deeper layers of pDiTs reduces FID by 20–30% (e.g., pDiT-B/16: 7.39$R=4$45.30, pDiT-L/16: 4.13$R=4$53.17) (Starodubcev et al., 15 May 2026).
• Registers induce smoother (lower total variation) and tighter (lower $R=4$6-norm) feature maps at high-noise diffusion steps, facilitating improved sample quality and convergence.
• Linear probing of intermediate register features reveals a spectrum: high-norm registers act as "sinks" with near-zero class accuracy, while mid-norm registers encode semantic information (object class, scene layout).

Recent architectural innovations introduce dual-stream blocks, splitting normalization and MLP parameters between patch and register streams, further reducing FID with minimal parameter or compute overhead.

A pivotal observation is that any auxiliary tokens appended in deeper layers—such as text or class embeddings in SD3.5 and JiT—function as implicit registers, underpinning the broad empirical advantages of register-like mechanisms in generative transformers.

6. Influence on Patch Geometry and Interpretability

The addition of register tokens leads to compression of patch geometry, as measured by the effective rank of the patch Gram matrix. For instance, at layer 11 (ViT-S):

• DINOv2: erank $R=4$7 13.5,
• DINOv2+R: $R=4$8 8.7 (–36% vs. DINOv2),
• DINOv3: $R=4$9 4.0 (–54% vs. DINOv2+R) (Parodi et al., 15 Apr 2026).

This compression reflects the channeling of global variation into registers, resulting in more homogeneous patch features and improved robustness for dense visual tasks.

However, a consequence is that attention maps in large models with register tokens (or with strong [CLS] residual pathways) become unreliable indicators of which image regions contribute to global features. Genuine patch integration is lost—global outputs become convex combinations dominantly of register entries. For models where interpretability and patch-global coupling are paramount, recommendations include elimination of registers, weakening of residual [CLS] skips, and regularization towards patch homogeneity (Lappe et al., 9 May 2025).

7. Methodological Implications and Guidelines

Ablation studies relying solely on zeroing token activations can substantially overstate dependence on register content, due to distributional shifts propagated by the zero vector (Parodi et al., 15 Apr 2026). Complementary in-distribution controls (mean-, noise-substitution, shuffling) are essential for accurate assessment of token significance. Register slots are often structurally indispensable as context channels or feature buffers, even if their exact runtime content is replaceable.

Interpretation of attention maps in register-augmented transformers must account for the global information pathway shift. When tight patch-global coupling or attention-based interpretability is required, model design should restrict or regularize register and [CLS] skip pathways, and diagnostic validation (e.g., CKA alignment) should be conducted to ensure meaningful integration of local features (Lappe et al., 9 May 2025).

A plausible implication is that as model scale and complexity increase, structural token roles and global-local decoupling are likely to grow in significance for both discriminative and generative vision transformer architectures.