TF-LLMER: Efficient LLM-Enhanced Recommender

• TF-LLMER is a lightweight, training-friendly framework that integrates offline LLM item embeddings into sequential recommenders to overcome optimization barriers.
• It employs item embedding normalization and Rec-PCA to mitigate norm disparities and semantic-collaboration misalignment, ensuring stable loss descent.
• Empirical evaluations show that TF-LLMER achieves consistent ranking improvements and faster convergence compared to prior LLM-enhanced recommendation methods.

Training-Friendly LLM-Enhanced Recommender (TF-LLMER) is a lightweight, theory-grounded framework for LLM-enhanced recommendation in the “no inference-time LLM” paradigm, where item representations are extracted offline from a LLM and then injected into a conventional recommender backbone. It was introduced to address an optimization barrier observed in prior LLM-enhanced recommenders: although LLM-derived item vectors provide rich semantics from item text, their direct injection can significantly hinder backbone optimization, producing high training losses that quickly stall compared with random initialization. TF-LLMER addresses this barrier through two coupled mechanisms—item embedding normalization and recommendation-aware dimensionality reduction via Rec-PCA—and is designed to integrate with sequential recommenders such as GRU4Rec, SASRec, and Bert4Rec (Zhu et al., 22 Apr 2026).

1. Problem setting and formalization

TF-LLMER is formulated for sequential recommendation over an item set $I$. A training sample $(q, y)$ consists of a historical interaction sequence $q$ and a ground-truth next item $y$. The backbone recommender encodes $q$ into a sequence representation $h \in \mathbb{R}^d$, while item embeddings are arranged in $E \in \mathbb{R}^{|I| \times d}$. Logits over the item universe are computed as

$s = E h \in \mathbb{R}^{|I|}.$

Training uses cross-entropy: $\ell(s,y) = -s_y + \log \sum_{j \in I} e^{s_j}.$

With $p = \mathrm{softmax}(s)$ and $(q, y)$0 the one-hot vector at index $(q, y)$1, the gradient and Hessian with respect to logits are

$(q, y)$2

Because the representation map is $(q, y)$3, the Hessian with respect to the backbone representation is

$(q, y)$4

The framework studies optimization difficulty through the Hessian condition number

$(q, y)$5

A larger $(q, y)$6 indicates worse conditioning and, therefore, more difficult local optimization. However, under long-tail item distributions, curvature concentrates on a small subset of effective items—typically the positive item and hard negatives—so over the full item universe $(q, y)$7, making $(q, y)$8 uninformative. TF-LLMER therefore restricts the analysis to an effective subspace $(q, y)$9, spanned by the gradient support along training. If $q$0, $q$1 is the orthogonal projection onto $q$2, and $q$3, then the restricted Hessians are

$q$4

The analysis assumes a local non-saturation region: for all $q$5, there exist $q$6 such that

$q$7

Under this assumption, $q$8 is positive definite and induces bounded curvature in the effective subspace, allowing conditioning analysis to focus on the injected item representations rather than on vacuous degeneracy from the full catalog (Zhu et al., 22 Apr 2026).

2. Optimization barrier and curvature analysis

The central theoretical result is that the conditioning of $q$9 depends tightly on both item embedding norms and item embedding angles. Writing $y$0, $y$1, and $y$2, the paper derives

$y$3

This isolates two root causes of ill-conditioning. The first is large norm disparity, represented by $y$4, which quadratically worsens conditioning. The second is angular conditioning, represented by the condition number of the effective cosine similarity matrix $y$5. In the empirical setting motivating TF-LLMER, LLM-derived item embeddings exhibit significant norm variance, and their semantic geometry often emphasizes similarities that are not aligned with collaborative recommendation signals (Zhu et al., 22 Apr 2026).

The normalization results make the necessity claim explicit. If item embeddings are normalized, then

$y$6

whereas without normalization

$y$7

Moreover, even when $y$8 is finite, $y$9 can be arbitrarily large if norms are unconstrained. The practical implication is direct: $q$0 normalization at logit computation is not an optional stabilizer but a necessary condition for avoiding arbitrarily ill-conditioned training.

For angular structure, TF-LLMER defines effective coherence

$q$1

If $q$2, then Gershgorin-based analysis yields

$q$3

Under the same condition, this upper bound increases monotonically with $q$4. Hence reducing maximum cosine similarity among effective items directly tightens the conditioning bound. This formalizes the paper’s claim that a semantic-collaboration misaligned angular clustering of LLM representations can make the optimization geometry hard for the backbone to exploit, even when the semantics themselves are rich (Zhu et al., 22 Apr 2026).

3. Framework components: normalization and Rec-PCA

TF-LLMER contains two components that correspond exactly to the two diagnosed causes of optimization failure.

Component	Mechanism	Intended effect
Item embedding normalization	$q$5, $q$6	Eliminates norm-driven instability and bounds conditioning
Rec-PCA	Spectral transformation using $q$7 and $q$8	Aligns semantic variance with collaborative structure and reduces effective coherence

The normalization step is deliberately minimal. Logits are computed with normalized item embeddings,

$q$9

No temperature scaling is required; pure $h \in \mathbb{R}^d$0 normalization is stated to suffice. Empirically, this restores stable loss descent, whereas without normalization losses start extremely high and plateau prematurely.

The second component, Rec-PCA, is a recommendation-aware dimensionality reduction method applied to LLM item representations $h \in \mathbb{R}^d$1. It constructs an item-item co-occurrence graph $h \in \mathbb{R}^d$2 from interaction histories by linking adjacent items within sequences. If $h \in \mathbb{R}^d$3 is the adjacency matrix, $h \in \mathbb{R}^d$4, and

$h \in \mathbb{R}^d$5

is the symmetric normalized Laplacian, then graph alignment is promoted by penalizing total variation: $h \in \mathbb{R}^d$6

Rec-PCA balances semantic retention and collaborative alignment. For transformed data $h \in \mathbb{R}^d$7, the objective is

$h \in \mathbb{R}^d$8

with a diagonal covariance requirement in the transformed dimensions. Defining

$h \in \mathbb{R}^d$9

and requiring $E \in \mathbb{R}^{|I| \times d}$0 so that $E \in \mathbb{R}^{|I| \times d}$1, the closed-form solution is obtained by spectral decomposition: $E \in \mathbb{R}^{|I| \times d}$2 The transformed embeddings have diagonal covariance equal to the top-$E \in \mathbb{R}^{|I| \times d}$3 eigenvalues of $E \in \mathbb{R}^{|I| \times d}$4, thereby maximizing semantic variance under graph alignment. To avoid computing the matrix square root exactly, TF-LLMER uses Chebyshev polynomial approximation. The paper gives first- and second-order approximations, with low-order expansions reported to suffice in practice. The intended geometric effect is to attenuate graph-inconsistent components, reduce recommendation-misaligned cosine similarity, lower $E \in \mathbb{R}^{|I| \times d}$5, and thereby improve the conditioning of the backbone’s optimization problem (Zhu et al., 22 Apr 2026).

4. Integration into backbone recommenders

TF-LLMER is model-agnostic and is integrated as an initialization-and-training protocol rather than as a new backbone architecture. The pipeline is explicitly defined in five stages. First, LLM item representations $E \in \mathbb{R}^{|I| \times d}$6 are extracted; the framework can use any LLM, but the reported experiments use Qwen3-Embedding-8B for fair comparison across methods. Second, an item-item co-occurrence graph is built by connecting consecutive items in all sequences, then $E \in \mathbb{R}^{|I| \times d}$7-sparsified by keeping, for each item, the top-$E \in \mathbb{R}^{|I| \times d}$8 neighbors with highest co-occurrence counts. Third, Rec-PCA is applied to form

$E \in \mathbb{R}^{|I| \times d}$9

compute $s = E h \in \mathbb{R}^{|I|}.$0 as the top-$s = E h \in \mathbb{R}^{|I|}.$1 eigenvectors, and obtain the initialized embedding table

$s = E h \in \mathbb{R}^{|I|}.$2

Fourth, the backbone item embedding table is initialized with $s = E h \in \mathbb{R}^{|I|}.$3, and training proceeds with normalized logits $s = E h \in \mathbb{R}^{|I|}.$4 under standard cross-entropy. Fifth, inference uses the backbone exactly as usual, with no LLM at serving time (Zhu et al., 22 Apr 2026).

The framework retrains the backbone from scratch after initialization. Backbone training complexity remains unchanged except for the cheap normalization at logit computation. The dominant additional costs are forming the $s = E h \in \mathbb{R}^{|I|}.$5 matrix $s = E h \in \mathbb{R}^{|I|}.$6, computing a partial eigendecomposition for the top-$s = E h \in \mathbb{R}^{|I|}.$7 components, and applying low-order graph filtering to $s = E h \in \mathbb{R}^{|I|}.$8. The implementation notes specifically mention Lanczos or ARPACK for the eigendecomposition and Chebyshev polynomials for approximating $s = E h \in \mathbb{R}^{|I|}.$9.

The reproducibility settings reported for the reference implementation use PyTorch, Adam, and cross-entropy for all backbones. The learning rate is $\ell(s,y) = -s_y + \log \sum_{j \in I} e^{s_j}.$0; weight decay is chosen from $\ell(s,y) = -s_y + \log \sum_{j \in I} e^{s_j}.$1; $\ell(s,y) = -s_y + \log \sum_{j \in I} e^{s_j}.$2 is searched over $\ell(s,y) = -s_y + \log \sum_{j \in I} e^{s_j}.$3; $\ell(s,y) = -s_y + \log \sum_{j \in I} e^{s_j}.$4 for graph sparsification is searched over $\ell(s,y) = -s_y + \log \sum_{j \in I} e^{s_j}.$5; training runs for up to 100 epochs with early stopping if validation NDCG@10 does not improve for 10 epochs. The practical guidance narrows these choices further: $\ell(s,y) = -s_y + \log \sum_{j \in I} e^{s_j}.$6 is recommended because too small a graph loses signal and too large a graph adds noise, while $\ell(s,y) = -s_y + \log \sum_{j \in I} e^{s_j}.$7–$\ell(s,y) = -s_y + \log \sum_{j \in I} e^{s_j}.$8 is often best, with $\ell(s,y) = -s_y + \log \sum_{j \in I} e^{s_j}.$9 needed to preserve a well-defined square root. The embedding dimension $p = \mathrm{softmax}(s)$0 should match the backbone’s embedding dimension, and normalization should always be used when computing logits. The paper also notes that TF-LLMER can be useful for cold-start settings because it leverages text-only LLM representations while using Rec-PCA to align them with collaborative structure and normalization to stabilize norm variability across cold items (Zhu et al., 22 Apr 2026).

5. Empirical evaluation, compatibility, and ablations

The reported evaluation covers Yelp, Amazon Sports, and Amazon CDs with three sequential backbones—GRU4Rec, Bert4Rec, and SASRec—and uses Hit Rate@$p = \mathrm{softmax}(s)$1 and NDCG@$p = \mathrm{softmax}(s)$2 for $p = \mathrm{softmax}(s)$3. All methods use Qwen3-Embedding-8B item text embeddings for fairness. The baselines are LLMInit, LLM-ESR, LLMEmb, and LLM2Rec. TF-LLMER is reported to consistently outperform these baselines across datasets and backbones, with statistically significant gains under paired $p = \mathrm{softmax}(s)$4-tests over five runs at $p = \mathrm{softmax}(s)$5 (Zhu et al., 22 Apr 2026).

The SASRec results illustrate the reported gains:

Dataset	Best baseline	TF-LLMER
Yelp	LLM2Rec: H@10 $p = \mathrm{softmax}(s)$6, N@10 $p = \mathrm{softmax}(s)$7	H@10 $p = \mathrm{softmax}(s)$8, N@10 $p = \mathrm{softmax}(s)$9
Sports	Best-baseline gains reported	H@10 $(q, y)$00, N@10 $(q, y)$01
CDs	Best-baseline gains reported	H@10 $(q, y)$02, N@10 $(q, y)$03

On SASRec, TF-LLMER improves over the best baseline by $(q, y)$04 and $(q, y)$05 on Yelp H@10 and N@10, by $(q, y)$06 and $(q, y)$07 on Sports, and by $(q, y)$08 and $(q, y)$09 on CDs. Similar gains are reported for GRU4Rec and Bert4Rec.

The optimization-focused diagnostics are as important as the ranking metrics. Training loss curves decrease steadily under TF-LLMER, whereas prior LLM-enhanced methods show high initial losses that stall. Effective coherence $(q, y)$10 decreases with Rec-PCA compared with vanilla PCA and random initialization. Ablation studies support the two-component design: removing normalization causes high initial loss and early plateau, while replacing Rec-PCA with PCA increases effective coherence and worsens optimization. The framework also composes with prior methods. Replacing vanilla PCA by Rec-PCA and enforcing normalized logits improves methods such as LLMEmb and LLM2Rec; one reported example is SASRec+LLM2Rec, where H@10 rises from $(q, y)$11 to $(q, y)$12, a $(q, y)$13 relative gain. These results support the paper’s main claim that the bottleneck in LLM-enhanced recommendation is not only semantic quality but also the optimization geometry induced by injected representations (Zhu et al., 22 Apr 2026).

6. Position within LLM-enhanced recommendation and open issues

TF-LLMER belongs to the broader class of LLM-enhanced recommender systems that use LLMs offline while avoiding inference-time LLM calls, a design direction identified as especially practical because real-world recommender serving requires low latency and high throughput (Liu et al., 2024). Within that landscape, TF-LLMER is distinctive for treating the injected LLM representation not primarily as a source of knowledge or as a target for distillation, but as an object whose norm distribution and angular geometry can make the downstream recommendation problem hard to optimize.

This focus differs from other training-friendly directions in the literature. Hypergraph-based methods such as HeLLM emphasize higher-order user-user and item-item structure together with graph-enhanced prefix tuning in the LLM fine-tuning stage (Guo et al., 13 Apr 2025). Parameter-efficient architectures such as Laser freeze the LLM and introduce prefix and suffix virtual tokens conditioned on collaborative information (Zhang et al., 2024). Curriculum-based approaches such as SOFT address the optimization gap in LLM-based recommenders by shifting from self-distilled easy data to real recommendation data (Tang et al., 27 May 2025). TF-LLMER addresses a different failure mode: it assumes the recommender itself remains a conventional backbone and asks why injecting fixed LLM item vectors can degrade, rather than help, optimization.

Two misconceptions are explicitly corrected by the framework. One is that richer semantic embeddings should automatically make recommendation training easier. The empirical observation motivating TF-LLMER is the opposite: direct LLM injection can raise loss and stall training. The other is that dimensionality reduction alone is sufficient. TF-LLMER argues that vanilla PCA does not address semantic-collaboration misalignment, while Rec-PCA does so by incorporating an item-item co-occurrence graph.

The framework also has stated limitations. It presumes non-trivial collaborative structure in the interaction data; in domains with weak collaborative signals or poor text, graph alignment may be less effective. Its theory depends on the effective-subspace assumption and on indirect control of $(q, y)$14 through coherence $(q, y)$15. The stated future directions include adaptive graph construction such as sequence-aware weighting, per-domain selection of $(q, y)$16, extending the conditioning analysis to other losses or pairwise ranking objectives, and exploring joint training schemes or lightweight adaptors that preserve LLM semantics while remaining consistent with the conditioning theory. In that sense, TF-LLMER is both a concrete recipe for stabilizing current LLM-enhanced recommenders and a theoretical argument that optimization conditioning should be treated as a first-class design criterion in this line of research (Zhu et al., 22 Apr 2026).