Why Mean Pooling Works: Quantifying Second-Order Collapse in Text Embeddings

Abstract: For constructing text embeddings, mean pooling, which averages token embeddings, is the standard approach. This paper examines whether mean pooling actually works well in real models. First, we note that mean pooling can collapse information beyond the first-order statistics of the token embeddings, such as second-order statistics that capture their spatial structure, potentially mapping distinct token embedding distributions to similar text embeddings. Motivated by this concern, we propose a simple metric to quantify such a collapse induced by mean pooling. Then, using this metric, we empirically measure how often this collapse occurs in actual models and texts, and find that modern text encoders are robust to this collapse. In particular, contrastive fine-tuned text encoders tend to be less prone to the collapse than their pretrained backbone models. We also find that the robustness of these text encoders lies in the concentration of token embeddings within each text. In addition, we find that robustness to the collapse, as quantified by our proposed metric, correlates with downstream task performance. Overall, our findings offer a new perspective on why modern text encoders remain effective despite relying on seemingly coarse mean pooling.

• The paper introduces the SOCM metric to quantify second-order collapse in mean-pooled text embeddings, offering a principled tool to assess embedding information loss.
• It demonstrates that contrastive fine-tuning induces token embedding concentration, thereby reducing covariance differences and enhancing model performance.
• Empirical analyses reveal that fine-tuned encoders mitigate mean pooling collapse, ensuring robust semantic discriminability across diverse NLP tasks.

Quantifying Second-Order Collapse in Mean-Pooled Text Embeddings

Motivation and Problem Formulation

Mean pooling, i.e., averaging token embeddings to produce text embeddings, has become ubiquitous in NLP applications due to both empirical success and computational efficiency. However, mean pooling inherently summarizes only the first-order statistics (mean) of the token embedding distribution and thus may collapse distinct token embedding distributions into similar representations, especially when higher-order statistics (e.g., covariance structure) are lost. The paper "Why Mean Pooling Works: Quantifying Second-Order Collapse in Text Embeddings" (2604.27398) interrogates this phenomenon, introduces a principled metric for quantifying such collapse, and empirically analyzes whether modern text encoders are susceptible to this limitation.

Figure 1: Mean pooling can map distinct token embedding distributions to similar text embeddings, yet fine-tuned text encoders are empirically robust to this collapse.

Theoretical Characterization of Collapse

Mean pooling fails to distinguish distributions whose means are similar but whose second-order statistics differ. The paper formalizes this intuition by defining the problem: mean pooling collapse arises if texts produce token embedding distributions that have similar means ($d_\mu$ small) but dissimilar covariances ($d_\Sigma$ large). This scenario is systematically illustrated, capturing cases where distinct spatial structures in the token embedding distribution are lost to the pooling operation.

Figure 2: Collapse arises (red) when means are similar but covariance differs; it does not (green) when either means differ or both mean and covariance are similar.

SOCM: A Robust Metric for Collapse Quantification

To operationalize this intuition, the authors propose the Second-Order Collapse by Mean pooling (SOCM) metric, defined formally as:

$$\mathrm{SOCM}(d_\mu, d_\Sigma) := (1 - d_\mu) d_\Sigma$$

where $d_\mu$ is the scaled squared Euclidean distance between means and $d_\Sigma$ is the scaled Bures-Wasserstein distance between covariances. Both are normalized to $[0,1]$ under reasonable assumptions (unit-normed means, bounded trace). SOCM directly quantifies severe collapse: it is maximized when means are indistinguishable yet covariances diverge, and minimized when means diverge or covariances match.

Figure 3: SOCM values across $(d_\mu, d_\Sigma)$ confirm strict satisfaction of monotonicity and boundary properties.

The metric is rigorously motivated via decomposition of the Wasserstein distance between Gaussianized token embedding distributions and is shown to satisfy key properties: boundary conditions, monotonicity with respect to both arguments, and appropriate interaction effects. This guarantees SOCM's theoretical soundness as a collapse quantifier.

Empirical Analysis Across Modern Text Encoders

The authors empirically compute SOCM across a broad suite of modern text encoders—including contrastive fine-tuned models (SimCSE, E5, GTE, Nomic Embed, etc.) and their backbone models (BERT, MiniLM, MPNet)—using large-scale datasets (Wikipedia, MS MARCO). The results demonstrate that contrastively fine-tuned encoders consistently exhibit lower SOCM than their backbones: mathematical collapse due to mean pooling is rare, particularly in modern fine-tuned models. Quantitative and qualitative analyses show that fine-tuned encoders maintain separation even for semantically unrelated texts, whereas backbones may collapse them.

Mechanism: Concentration Induced by Fine-Tuning

To explain this robustness, the paper analyzes token embedding concentration inside texts. Theoretically, conditions in a simplified Transformer block are derived under which the spread of token embeddings contracts through self-attention, residual connection, and per-token transformation (LayerNorm, FFN). Empirical layerwise analysis reveals that contrastive fine-tuning induces stronger concentration of token embeddings in the later layers, suppressing intra-text variance and thus minimizing the covariance difference effect ($d_\Sigma$). This concentration is reflected in lower SOCM.

Figure 4: Layerwise concentration metrics, showing that fine-tuned encoders have lower $\lambda$ (attention contraction) and normalized spread in final layers.

Downstream Task Implications and Correlation

A key empirical finding is that SOCM is negatively correlated with downstream task performance: models with lower SOCM achieve higher scores on standard evaluation benchmarks (MTEB v2, etc.). Notably, SOCM performs better than simple concentration measures ($S(\bm{X})/\|\bm{X}\|_2^2$) for correlating with downstream utility, capturing both intra-text concentration and inter-text mean separation.

Figure 5: Negative correlation between average SOCM and MTEB scores; BERT-based models annotated for clarity.

Discussion and Future Directions

The results assert a contradictory claim: despite theoretical risks of information collapse via mean pooling, empirical evidence shows that modern, fine-tuned encoders are robust against such collapse—primarily due to contrastive fine-tuning which promotes intra-text concentration and inter-text discriminability. SOCM emerges as a principled tool for model analysis, potentially useful for training diagnostics and regularization. The paper opens future directions:

• Mathematical modeling of the mechanisms driving concentration under contrastive fine-tuning.
• Extending SOCM regularization to encoder pretraining.
• Analysis of mean pooling's role in LLM-based generation and context compression.

Conclusion

This work provides a rigorous quantification of the limitations of mean pooling, demonstrates through theoretical and empirical analyses why modern encoders avoid collapse, and connects collapse robustness to downstream effectiveness. It reframes mean pooling not as an inherently coarse operation, but as one that—under appropriate encoder training—remains effective. The SOCM metric is a valuable analytical tool, and further exploration could drive improvements in both model architecture and training protocols for future AI systems.