Revisiting [CLS] and Patch Token Interaction in Vision Transformers

Abstract: Vision Transformers have emerged as powerful, scalable and versatile representation learners. To capture both global and local features, a learnable [CLS] class token is typically prepended to the input sequence of patch tokens. Despite their distinct nature, both token types are processed identically throughout the model. In this work, we investigate the friction between global and local feature learning under different pre-training strategies by analyzing the interactions between class and patch tokens. Our analysis reveals that standard normalization layers introduce an implicit differentiation between these token types. Building on this insight, we propose specialized processing paths that selectively disentangle the computational flow of class and patch tokens, particularly within normalization layers and early query-key-value projections. This targeted specialization leads to significantly improved patch representation quality for dense prediction tasks. Our experiments demonstrate segmentation performance gains of over 2 mIoU points on standard benchmarks, while maintaining strong classification accuracy. The proposed modifications introduce only an 8% increase in parameters, with no additional computational overhead. Through comprehensive ablations, we provide insights into which architectural components benefit most from specialization and how our approach generalizes across model scales and learning frameworks.

• The paper demonstrates that implicit token conflation in ViTs undermines dense prediction, motivating explicit architectural specialization.
• It introduces splitting normalization and QKV layers into separate [CLS] and patch-specific paths, leading to +2 mIoU improvements in segmentation benchmarks.
• Extensive experiments across tasks and pre-training regimes validate the approach, offering a scalable, parameter-efficient enhancement for vision models.

Revisiting [CLS] and Patch Token Interaction in Vision Transformers

Introduction

The paper "Revisiting [CLS] and Patch Token Interaction in Vision Transformers" (2602.08626) interrogates a fundamental architectural decision in Vision Transformers (ViTs): the indistinguishable treatment of global ([CLS]) and local (patch) tokens across all model layers despite their divergent functional roles. By conducting a comprehensive analysis of token dynamics—especially their interactions within normalization layers—the authors uncover implicit bias and propose explicit architectural specializations that disentangle the computational flows of [CLS] and patch tokens. The resulting modifications yield substantial gains in dense prediction tasks, with over +2 mIoU improvements in segmentation benchmarks, minimal parameter increase (~8%), and no inference computational penalty.

Token Interaction Dynamics and Friction

ViTs split an image into patch tokens and typically insert a trainable [CLS] token that aggregates information for global tasks. Despite the distinct semantic function of these tokens, classical architectures process them identically. The paper systematically analyzes cosine similarities between [CLS] and patch tokens across principal layers.

It is demonstrated that LayerNorms preceding attention operations induce sharp token disentanglement (cosine similarity near zero), while self-attention subsequently increases their similarity. This effect is consistent across SOTA ViT backbones (DINOv2, variants with registers [darcet2023vision], attention bias [an2025systematic], and supervised DEIT-III). Notably, patch tokens maintain positive intra-similarity, indicating regularization but not collapse.

Figure 1: LayerNorm pre-attn. Similarity statistics show strong [CLS]-patch differentiation before attention.

At deeper blocks, dimension-level specialization emerges: some dimensions are exclusively leveraged by either patches or [CLS], facilitating normalization layers' implicit separation. This architectural friction suggests that the model allocates capacity to implicit token differentiation instead of direct feature learning.

Figure 2: Block 7. High-magnitude feature dimensions become token-specialized in mid-to-deep layers, driving separation.

Architectural Specialization: Explicit Disentanglement

Building on the observed phenomenon, the authors propose explicit architectural specialization: normalization layers and early QKV projections are split into [CLS] and patch-specific weight sets. The tokens continue to share the attention mechanism to preserve mutual information flow, but their internal transformations are disentangled.

Figure 3: Architecture specialization. [CLS] and patch tokens traverse specialized layers but interact via shared attention.

By selectively applying specialization to normalization layers and QKV projections, especially in the first third of model depth, the approach achieves optimal trade-offs between performance and parameter efficiency. Specializing only normalization layers increases parameters by 0.05%, QKV by 8%, and both yields best dense performance. Notably, specialization does not increase inference FLOPs.

Empirical Evaluation and Benchmarking

The evaluation spans dense semantic segmentation (ADE20k, Cityscapes, VOC), depth estimation (KITTI, NYU, SUN RGB-D), and detection (COCO), coupled with ImageNet classification accuracy.

Specializing normalization layers increases segmentation mIoU by +1.1 points and depth RMSE by -0.054, while the combined normalization+QKV approach improves segmentation by +2.2 mIoU. For DINOv2 ViT-L with attention bias, the best configuration maintains ImageNet accuracy at 85.2%, offers segmentation mIoUs in the 67.4 range, and depth RMSE as low as 1.165, highly competitive with state-of-the-art.

The effectiveness is consistent across:

• Pre-training frameworks: self-supervised DINOv2, supervised DEIT-III
• ViT model scales: ViT-B, L, H
• Norm artifact handlers: registers, attention bias, none

Generalization to detection and qualitative patch feature improvement (PCA, patch artifact reduction) is observed.

Figure 4: DINOv2 w/ regs. Visual feature improvement with register tokens and specialization.

Figure 5: DINOv2. Standard architecture before specialization.

Figure 6: DINOv2 w/ registers. Patch representation improvement in register-enhanced model.

Figure 7: DINOv2 w/ attn. bias. Patch representation improvement in attention-bias-enhanced model.

The approach significantly outperforms class-attention mechanisms (CaiT), where explicit [CLS] aggregation layers degrade dense performance relative to targeted specialization.

Ablations, Analysis, and LoRA Adaptation

Layer-level ablations indicate specialization of QKV and normalization is critical; post-attention projection or MLP specialization provides weak or negative gains. Specializing early blocks yields maximal dense task improvement, supporting the hypothesis that input distribution divergence and final objective proximity drive the effectiveness of specialization depth.

Low-Rank Adaptation (LoRA) is tested as a reduced-parameter variant for QKV specialization: using LoRA at rank 16 and 128 achieves near-maximal segmentation performance at only 0.2%–1.4% parameter increase, suggesting promising directions for scalable deployment.

Figure 8: LoRA design. Low-rank QKV adaptation for parameter-efficient specialization.

Theoretical and Practical Implications

The findings directly implicate the architectural design of ViTs for vision foundation modeling. Indistinguishable processing of global/local tokens is suboptimal, forcing the model to spend representational capacity on implicit separation. Explicit specialization, especially within normalization/QKV layers, enables richer dense patch representations, enhances downstream segmentation and depth prediction, and generalizes across tasks and pre-training regimes.

The results highlight:

• Normative architectural friction: Token-type conflation leads to locality loss and high-norm artifacts. The attention mechanism alone cannot compensate.
• Specialization as a universal principle: Explicitly disentangling token treatment is beneficial across scales and paradigms.
• Minimal parameter cost, no compute penalty: Targeted architectural modifications are practical for deployment.

Future research avenues include parameter-efficient specialization (e.g., LoRA, block-level dynamic splitting), extension to multimodal token-type disambiguation, and adaptive specialization based on downstream task requirements.

Conclusion

This paper establishes that Vision Transformer architectural design must recognize and explicitly accommodate the divergent functional roles of [CLS] and patch tokens. By leveraging normalization- and QKV-layer token specialization, dense feature quality is substantially improved with negligible parameter expansion and no inference cost. The insights are practically impactful for vision foundation model training and theoretically significant for understanding transformer token dynamics. Explicit specialization should be viewed as a fundamental architectural consideration for next-generation scalable and versatile vision models.