ACE: Anisotropy-Controllable Embedding for LLM-enhanced Sequential Recommendation

Abstract: Recent advances in the LLM-as-Extractor paradigm leverage LLMs to transfer semantically rich item embeddings into sequential recommendation (SR) backbones. However, LLM-generated embeddings often suffer from strong anisotropy. Most vectors are concentrated in similar directions, resulting in a geometric imbalance that makes it difficult to adapt to collaborative signals during fine-tuning. To address this challenge, we propose Anisotropy-Controllable Embedding (ACE), which explicitly controls the anisotropy of LLM-generated embeddings. Specifically, ACE utilizes a linear autoencoder (LAE) to reshape the embedding distribution while preserving its semantic structure. In this process, the L2-regularization term mitigates the anisotropy by controlling the dispersion of embedding dimensions, while the reconstruction loss maintains semantic relationships among items. That is, ACE balances geometric uniformity and semantic embedding preservation for more stable learning. Extensive experiments demonstrate that ACE consistently outperforms existing LLM-enhanced SR models, yielding improvements of up to 12.4% and 11.8% in Recall@20 and NDCG@20, respectively.

• The paper introduces ACE, which uses a linear autoencoder to continuously control embedding anisotropy while preserving semantic hierarchy.
• ACE achieves robust improvements, with up to 12.4% gain in Recall@20 and 11.8% in NDCG@20 across various sequential recommendation backbones.
• The method offers a plug-and-play solution adaptable to diverse LLM encoders and datasets without compromising semantic structure.

Anisotropy-Controllable Embedding for LLM-enhanced Sequential Recommendation

Motivation and Background

The integration of LLMs into sequential recommender systems (SR) enables the extraction of semantically rich item embeddings, substantially improving performance on diverse recommendation tasks. The LLM-as-Extractor paradigm replaces randomly initialized item representations with LLM-generated embeddings, which are subsequently fine-tuned within SR backbones. However, these embeddings are known to exhibit strong anisotropy: the majority of vectors are concentrated in a few dominant directions, resulting in a geometric imbalance that restricts effective adaptation to collaborative signals. Prior approaches (e.g., PCA, whitening, MLP projections) inadequately address this fundamental issue, either distorting semantic hierarchy or lacking continuous control over the embedding spectrum.

Figure 1: PCA visualization showing original LLM embedding anisotropy on the Beauty dataset; vectors are over-concentrated in semantic directions.

The importance of proper embedding initialization has been overlooked. Aggressive whitening uniformly flattens the embedding spectrum, destroying semantic structure, whereas PCA and MLP projection do not guarantee control over anisotropy, as confirmed by spectral visualizations and empirical performance.

ACE: Methodology

ACE, the proposed Anisotropy-Controllable Embedding, employs a linear autoencoder (LAE) to reshape LLM embeddings, providing both continuous control of anisotropy and preservation of semantic hierarchy. The LAE objective is formulated as follows:

$$\mathcal{L}(\mathbf{B}_\text{ACE}) = \min_{\mathbf{B}_\text{ACE}} \| \mathbf{E}^\top - \mathbf{E}^\top\mathbf{B}_\text{ACE} \|_F^2 + \lambda \| \mathbf{B}_\text{ACE} \|_F^2$$

Here, the reconstruction loss preserves semantic directions, while the $L_2$ regularization (weighted by $\lambda$) modulates the singular value spectrum, attenuating dominant directions. By increasing $\lambda$, ACE flattens the spectrum toward isotropy—yet crucially, this reshaping process is continuous and avoids total semantic distortion.

Figure 2: Normalized eigenvalue spectrum of ACE for different $\lambda$ on the Beauty dataset; smaller $\lambda$ flattens the spectrum toward isotropy.

Through spectral decomposition, ACE applies magnitude-dependent shrinkage to singular values, reshaping item-item geometry while maintaining semantic hierarchy. This yields embeddings that exhibit improved geometric balance, supporting stable learning and effective adaptation to collaborative signals.

Empirical Evaluation

Experiments are conducted on four benchmark datasets (Beauty, Toys, Yelp, ML-20M) using multiple SR backbone architectures (SASRec, GRU4Rec, BERT4Rec). ACE consistently delivers robust improvements over existing LLM-as-Extractor baselines. Specifically, on SASRec, ACE achieves up to 12.4% gain in Recall@20 and 11.8% in NDCG@20 compared to the strongest competitor. The improvement is attributable to ACE's spectral control: whitening-based methods (WhitenRec+) sometimes degrade performance by distorting semantic structure, whereas ACE achieves balance between isotropy and semantic preservation.

In backbone analysis, ACE demonstrates architecture-agnostic effectiveness, providing consistent gains across GRU4Rec and BERT4Rec. Performance gains persist regardless of sequence modeling mechanism, highlighting the generality of ACE's spectral reshaping.

Figure 3: Performance comparison of ACE against LLM-as-Extractor baselines across three LLM encoders on Beauty with SASRec; ACE consistently yields the best Recall@20.

ACE is also evaluated across multiple LLM encoders (F2LLM-4B, Qwen3-Embedding-8B, KaLM-Embedding-Gemma3-12B-2511). In all settings, ACE outperforms alternatives, evidencing that its benefits are not tied to a specific LLM embedding generator.

Hyperparameter analysis shows that performance peaks at moderate values of $\lambda$. Small $\lambda$ values over-flatten the spectrum, leading to loss of semantic hierarchy; excessively large $\lambda$ values fail to sufficiently reduce anisotropy. This underscores the necessity of balanced spectral control.

Practical and Theoretical Implications

ACE's approach directly addresses a core limitation in LLM-enhanced recommendation: geometric imbalance in embedding distributions. By preserving semantic directions and modulating anisotropy, ACE enables SR models to leverage both LLM semantic knowledge and collaborative filtering signals efficiently. The spectral formulation and controllable shrinkage mechanism open avenues for further research into embedding initialization for other domains (e.g., graph representation, multimodal fusion).

Practical deployment is immediately feasible, as ACE operates as a plug-and-play component for embedding initialization, compatible with modern SR architectures and diverse LLM encoders. The method's continuous control over the embedding spectrum enables adaptation to domain-specific requirements and model architectures.

Future directions include exploring non-linear autoencoder variants for enhanced spectral control, integrating ACE with dynamic online learning for personalized recommendations, and examining its impact in high-variance user/item regimes.

Conclusion

ACE introduces anisotropy-controllable embedding reshaping for LLM-enhanced sequential recommendation, using a linear autoencoder to continuously balance geometric isotropy and semantic preservation. This mitigates representational degeneration and enhances SR performance across datasets, backbones, and LLM encoders. ACE's spectral approach offers both theoretical clarity and practical efficacy, and establishes a foundation for further advances in embedding initialization and spectral geometry in recommendation systems.