---
title: Collapsed [CLS] Embeddings
url: https://www.emergentmind.com/topics/collapsed-cls-embeddings
type: topic
---

# Collapsed [CLS] Embeddings

Collapsed [CLS] embeddings refer to a well-documented phenomenon in Transformer-based language models whereby the special [CLS] token’s output vector, after fine-tuning or in standard BERT-like architectures, occupies a narrow, highly redundant subspace—sometimes essentially a single principal direction or very few dimensions—regardless of the semantic content of the underlying input. This collapse undermines both the expressiveness and the downstream transferability of the [CLS] representation, particularly across languages or in long-context settings, and is consistently observed in empirical studies, theoretical work, and practical deployment analyses [2601.00366, 1910.10832, 2601.21525, 2505.17282].

## 1. Characterization and Diagnostics of Collapsed [CLS] Embeddings

Collapse is rigorously defined as the situation in which the [CLS] vectors for diverse input sequences—semantically unrelated or across disparate languages—exhibit exceedingly high mutual similarity and reside within a low-dimensional manifold. This is quantifiable using several diagnostic metrics:

- **Pairwise Cosine Similarity**: In collapsed spaces, $\cos(h_\text{CLS}(x_i), h_\text{CLS}(x_j)) \gtrsim 0.99$ even for completely unrelated sentence pairs [2601.00366].
- **Principal Component Analysis (PCA)**: Collapsed embeddings exhibit a spectrum dominated by a single principal component, which explains 50–80% of the total variance in models like RoBERTa-Base (47%) or XLM-RoBERTa-Base (78%) [2601.00366]. Finetuned BERT [CLS] vectors exhibit a similar concentration of variance (explained in a few top components) [1910.10832].
- **Effective Rank (e.g., RankMe)**: Spaces with collapse have effective rank ≈1–2, indicating only 1–2 directions carry nearly all informational content [2601.00366].
- **t-SNE and Isotropy Diagnostics**: Embedding visualizations cluster tightly according to language, with little or no overlap between translation equivalents [2601.00366].

## 2. Theoretical Explanations for [CLS] Collapse

The root mechanisms driving collapse differ depending on architecture and training strategy:

- **Softmax-Attention Models**: In one-layer softmax-attention with a trainable [CLS] vector $p$ and word embedding table $E$, gradient descent with logistic loss rapidly aligns $E$ with the output head vector $v$ in proportion to token-class correlations, and gradient flow on $p$ maximizes the selection margin for important features. This converges to a “winner-takes-all” [CLS], where $p$ attends almost exclusively to highly predictive tokens, mathematically saturating the representation and forcing collapse [2505.17282].
- **Transformer Pooling and RoPE**: In BERT-like architectures using rotary positional embeddings (RoPE), the positional bias strongly attenuates attention weights as distance from [CLS] increases, restricting information aggregation to initial tokens. As a result, [CLS] output concentrates on the initial document span, ignoring distributed evidence and collapsing the representation for long or information-dispersed sequences [2601.21525].
- **Finetuning for Classification**: Fine-tuned models often segregate class decision information into a very low-dimensional subspace. PCA and single-dimension “salient neuron” analyses show that after finetuning, [CLS] output nearly always shrinks onto a subspace with dimensionality similar to the number of classes, amplifying collapse [1910.10832].

## 3. Empirical Manifestations and Impact

Empirical studies demonstrate pervasive [CLS] collapse across standard and multilingual BERT-style models:

- **Quantitative Benchmarks**:
  - RoBERTa/XLM-RoBERTa: Cosine similarity $0.991$–$0.998$ between [CLS] vectors of unrelated and translated sentence pairs; first PCA component explains 47–78% of variance [2601.00366].
  - Pairwise cosine trends remain high for positive pairs and only decline for negatives after applying corrective objectives [2601.00366].
- **Long-Context and Multilingual Performance**:
  - For long documents, [CLS] attention weights are heavily local, causing under-representation of distributed evidence; retrieval and classification metrics degrade as input length increases [2601.21525].
  - In multilingual settings, collapsed [CLS] embeddings result in language-specific clusters with poor cross-lingual alignment, impeding zero-shot transfer [2601.00366].

| Diagnostic       | Typical Value (Collapsed)                | Source         |
|------------------|------------------------------------------|----------------|
| Cosine (unrelated)| 0.995–0.998                             | [2601.00366]   |
| PCA PC1 var.     | 47% (RoBERTa), 78% (XLM-RoBERTa)         | [2601.00366]   |
| Effective Rank   | 1–2                                      | [2601.00366]   |

## 4. Mitigation Strategies and Remedies

Several strategies have been proposed to counteract [CLS] collapse, each targeting different underlying mechanisms:

- **Joint Embedding Predictive Architectures (JEPA)**: In BERT-JEPA (BEPA), a JEPA-style InfoNCE alignment loss is added during fine-tuning. This decorrelates [CLS] vectors for unrelated pairs and incentivizes language-invariant, semantically meaningful structure. Empirically, this reduces same/different-language unrelated cosine similarities from ≈0.99 to ≈0.50–0.60 for negatives, lowers the variance explained by PC1 from up to 78% to ≈34%, and raises embedding effective rank from ≈1–2 to ≈5–10 [2601.00366].
- **Landmark (LMK) Pooling**: By partitioning the input sequence into chunks and pooling over multiple landmark ([SEP]) tokens, LMK sidesteps the positional bias inherent in RoPE and distributes representation capacity across the document. This method yields improved retrieval performance, especially for long input sequences, and recovers global coverage without sacrificing local salient evidence [2601.21525].
- **Compression via PCA and Salient Neuron Pruning**: For models where class information is collapsed into a low-dimensional manifold, projecting [CLS] vectors into the top $k$ principal components ($k \approx$ number of classes), or even selecting the top $k$ most salient coordinates, retains near-optimal downstream accuracy while removing vast redundancy ($\leq$ 0.5 pp loss for $k = 5$–$25$) [1910.10832].

## 5. Quantitative Results and Comparative Analyses

Results quantifying the efficacy of different strategies and the severity of collapse include:

- **JEPA/BERT-JEPA**:
  - XNLI zero-shot transfer: Baseline XLM-RoBERTa 72.8% $\to$ BEPA-Bilingual 74.4% accuracy (+1.6%).
  - MLQA F1: Baseline 62.1, BEPA-Bilingual 63.2.
  - Effective Rank (RankMe): Baseline ≈1–2 $\to$ BEPA ≈5–10 [2601.00366].

- **LMK Pooling**:
  - MLDR NDCG@10: CLS 24.9, LMK 35.0 (Δ+10.1).
  - Robust improvement on out-of-domain multilingual retrieval (MLDR: CLS 24.9, LMK 37.2) [2601.21525].

- **Dimensionality Reduction**:
  - For IMDB sentiment analysis, projecting to $k=5$ components: full accuracy (93.7%), $k=25$: 93.7%, full (768-d): 93.7%.
  - Top-1 “salient neuron” recapitulates nearly all accuracy on 2-class datasets; $n$ top neurons recover all accuracy for $n$-class tasks [1910.10832].

## 6. Broader Implications and Recommendations

The collapse of [CLS] embeddings poses challenges for semantic discrimination, cross-lingual generalization, and scalability to long contexts or multi-task settings. The following recommendations follow from the empirical and theoretical literature:

- **For Multilingual and Zero-Shot Tasks**: Use alignment objectives (e.g., InfoNCE) to explicitly decorrelate [CLS] representations across unrelated samples; train with both monolingual and bilingual packaging for best cross-lingual generalization [2601.00366].
- **For Long Sequence Embeddings**: Prefer distributed pooling schemes (e.g., LMK pooling) to avoid biased aggregation of information at the document head [2601.21525].
- **For Storage and Compute Efficiency**: Compress output representations via PCA/SVD or single-dimension pruning, reducing memory and compute with negligible loss in classification accuracy [1910.10832].
- **Model Design**: Question and adapt the exclusive reliance on a single [CLS] vector, especially for tasks requiring wide semantic coverage or robust retrieval beyond short contexts [2601.21525].

## 7. Open Questions and Future Directions

Despite advances in remedies, several open problems remain:

- Theoretical analysis of collapse in deep versus shallow attention models beyond single- or two-layer softmax attention [2505.17282].
- Generalization of landmark pooling to multi-modal or multi-vector architectures [2601.21525].
- Optimal balancing between information bottlenecking (to facilitate compression) and retention of discriminative capacity for highly semantic or reasoning-intensive tasks [1910.10832].
- Systematic study of JEPA-style alignment objectives with flexible predictors for various downstream generalization and transfer scenarios [2601.00366].

A plausible implication is that future embedding architectures may adopt hybrid pooling and alignment strategies to both preserve contextual richness and support efficient, analytically tractable representations for cross-lingual and long-context NLP tasks.

Source: https://www.emergentmind.com/topics/collapsed-cls-embeddings