Your UnEmbedding Matrix is Secretly a Feature Lens for Text Embeddings

Abstract: LLMs exhibit impressive zero-shot capabilities across a wide range of downstream tasks. However, they struggle to function as off-the-shelf embedding models, leading to suboptimal performance on massive text embedding benchmarks. In this paper, we identify a potential cause underlying this deficiency. Our motivation stems from an unexpected observation: text embeddings tend to align with frequent but uninformative tokens when projected onto the vocabulary space. We argue that this excessive expression of high-frequency tokens suppresses the model's ability to capture nuanced semantics. To address this, we introduce EmbedFilter, a simple linear transformation designed to refine text embeddings derived from LLMs directly. Specifically, we uncover that the unembedding matrix within LLMs encodes a latent space that is actively writing these frequent tokens into embedding space. By filtering out this subspace, EmbedFilter suppress the influence of high-frequency tokens, thereby enhancing semantic representations. As a compelling byproduct, this enables an inherent dimensionality reduction, lowering index storage and speedup retrieval while fully preserving the refined embedding quality. Our experiments across multiple LLM backbones demonstrate that LLMs equipped with EmbedFilter achieve superior zero-shot downstream performance even with significantly reduced embedding dimensions. We hope our findings provide deeper insights into the mechanisms of LLM-based representations and inspire more principled designs to improve text embeddings training. Our code is available at https://github.com/CentreChen/EmbFilter.

• The paper presents the EmbedFilter method, a post-processing technique that filters detrimental spectral directions to suppress high-frequency token bias in LLM text embeddings.
• It employs spectral analysis and singular value decomposition to isolate an edge subspace responsible for semantic dilution in zero-shot representations.
• Experimental results on the MTEB benchmark demonstrate up to 14.1% improvement and effective dimensionality reduction across various LLM architectures.

The Unembedding Matrix as a Feature Lens for LLM-Derived Text Embeddings

Introduction

Pretrained LLMs have become central to representation learning, yet they exhibit documented deficiencies as off-the-shelf embedding models for semantic retrieval and related tasks. This paper identifies and mechanistically analyzes a previously unaddressed failure mode: the tendency of LLM hidden text embeddings to align disproportionately with high-frequency but semantically uninformative tokens, especially when projected into the vocabulary space via the unembedding matrix. Through empirical analyses and spectral investigations, the paper demonstrates that the LLM unembedding matrix captures a latent subspace inducing this effect and proposes a novel linear post-processing method, EmbedFilter, to suppress this phenomenon, resulting in both empirical performance improvements and efficient dimensionality reduction.

Empirical Evidence of Representation Collapse

The authors leverage the Logit Lens tool to project LLM-derived text embeddings into vocabulary space, revealing that the resulting logits are highly aligned with uninformative, frequent tokens (e.g., stopwords, punctuation). This observation holds universally across diverse LLM architectures (Qwen, Llama, Mistral). As illustrated in Figure 1, such alignments dominate the decoding probabilities regardless of the input semantic content.

Figure 1: Logit Lens applied to text embeddings from three LLM backbones demonstrates alignment with high-frequency, semantically uninformative tokens.

Concurrent with prior literature on embedding anisotropy [bertflow, ethayarajh-2019-contextual], the analysis here attributes the effect to a "centroid" or "average token" region dominating hidden state geometry, occluding nuanced semantics at the representation level.

Spectral Mechanistic Analysis

To probe the source of this representation collapse, the paper applies spectral analysis to the unembedding matrix. Through singular value decomposition and Logit Spectroscopy, the authors isolate a latent "edge spectrum" subspace—the span of right singular vectors with smallest and largest singular values—as dominating the representation of high-frequency tokens. Experimental manipulation confirms that filtering out projections onto these edge vectors disrupts the logit alignment with frequent tokens, while having little effect on infrequent or random tokens. Figure 2 presents the $\Delta\pi$ distribution, illustrating that the largest logit shifts for frequent tokens map to these spectral edges.

Figure 2: $\Delta\pi$ values across the spectrum for three LLMs, demonstrating edge singular vectors as primary contributors to frequent token alignment.

The theoretical analysis substantiates that the unembedding matrix, via its spectral properties, acts as a "feature lens" that projects semantic content but also injects frequency bias through dedicated subspaces.

EmbedFilter: Spectral Filtering of Edge Subspaces

The core contribution, EmbedFilter, is a simple linear transformation post-processing operation. Given learned SVD of the unembedding matrix, EmbedFilter projects each embedding onto the "bulk" subspace (mid-range singular vectors), explicitly filtering out both ends of the spectrum responsible for encoding high-frequency token alignment. Formally, for filtering ratio $\tau$, the retained singular vectors correspond to indices $[l_\tau:r_\tau]$; embeddings are mapped through this bulk projection, which simultaneously suppresses detrimental spectral directions and achieves principled dimensionality reduction.

Empirical application of EmbedFilter markedly alters the token bias in Logit Lens analysis: as shown in Figure 3, filtered embeddings exhibit higher alignment with semantically salient tokens from the input sequence, reducing the presence of spurious high-frequency outputs.

Figure 3: Re-running Logit Lens with EmbedFilter suppresses frequent tokens and strengthens semantic interpretability in token alignments.

Experimental Validation and Numerical Results

Extensive evaluations on the MTEB benchmark and related retrieval/classification tasks confirm that EmbedFilter provides consistent and significant zero-shot performance improvements across multiple LLM backbones (Qwen, Llama, Mistral), various pooling strategies (PromptEOL, ECHO, MetaEOL, GenEOL), and filtering ratios. Results cite improvements up to 14.1% on aggregate MTEB metrics, and, critically, the performance gains are maintained even when reducing embedding dimensionality to as little as $1/8$ of original size. EmbedFilter achieves superior performance relative to both naive truncation and traditional post-processing schemes (e.g., whitening [whitening]), without requiring access to calibration data or fine-tuning.

Figure 4 further characterizes the specificity of edge spectrum filtering: logit impact for frequent tokens is highly concentrated at the spectrum edges, in contrast to low-frequency or randomly sampled tokens.

Figure 4: $\Delta\pi$ distributions by frequency class confirm that high-frequency tokens are encoded almost exclusively by the edge spectrum.

Theoretical and Practical Implications

This work advances both interpretability and deployment efficiency for LLM-based embeddings. By mechanistically linking unembedding matrix geometry to semantic representation degradation, it exposes a fundamental limitation in vocabulary-dominant hidden state structure. The simple, analytic solution provided (EmbedFilter) outperforms methods reliant on calibration data or prompt engineering and generalizes robustly across model families. The "feature lens" perspective also invites exploration of alternative spectral filtering strategies and task-specific subspace selection.

The dimensionality reduction result is especially notable: by identifying directions that are detrimental for semantic encoding, EmbedFilter achieves state-of-the-art performance at significantly lower storage and retrieval cost, which has immediate applicability in retrieval, search, and real-time AI systems operating in resource-constrained environments.

Speculation on Future Developments

Anticipated future research includes principled spectral regularization during pretraining (rather than post hoc filtering), deeper analysis of the interaction between prompt-based methods and spectral embedding structure, and extension of the feature lens paradigm to LLMs with specialized output spaces (e.g., multimodal architectures). The mechanistic link between the unembedding matrix and semantic collapse offers an avenue for designing next-generation LLMs with intrinsically more isotropic, discriminative, and efficient embedding spaces.

Conclusion

This paper delivers a detailed spectral analysis of LLM unembedding matrices and establishes the edge spectrum subspace as the root cause of semantic dilution in zero-shot text embeddings. The proposed EmbedFilter method utilizes this insight to filter text embeddings, outperforming established methods in both performance and efficiency, and offering a broadly applicable, interpretable, and computationally attractive enhancement for LLM-based retrieval and representation systems (2606.07502).