L3AE: LLM-Enhanced Linear Autoencoder

• L3AE is a linear recommendation model that combines collaborative filtering with dense semantic representations from LLM-derived item embeddings.
• It utilizes a two-phase closed-form approach: first constructing a semantic item-to-item correlation matrix, then distilling this into the interaction-based collaborative filter.
• Empirical results show L3AE outperforms standard approaches, achieving higher Recall and NDCG scores on Amazon reviews, especially for long-tail items.

L3AE, abbreviated from LLM-Enhanced Linear Autoencoder, is a linear recommendation model for top-$N$ recommendation in implicit-feedback settings that integrates collaborative signals from user–item interactions with semantic signals from LLM-derived item embeddings while preserving the defining properties of linear autoencoders: closed-form training, global optimality, and efficiency (Moon et al., 19 Aug 2025). It is formulated as a two-phase procedure. First, it constructs a semantic item-to-item correlation matrix from dense item representations obtained from textual attributes such as title, category, brand, and description. Second, it learns the final item-to-item weight matrix from user–item interaction data while regularizing that matrix toward the semantic correlation structure. The model is positioned as a response to two limitations of prior approaches: standard linear autoencoders such as EASE rely only on interaction data, and earlier text-aware linear autoencoders rely on sparse lexical co-occurrence rather than dense semantic representations (Moon et al., 19 Aug 2025).

1. Problem setting and motivation

L3AE addresses recommendation under severe sparsity, especially for long-tail items, in binary implicit-feedback scenarios where the observed data are user clicks or purchases encoded in a matrix

$\mathbf{X} \in \{0,1\}^{m \times n},$

with $m$ users and $n$ items (Moon et al., 19 Aug 2025). In this setting, many items have very few interactions, so collaborative filtering alone can be insufficient for estimating reliable item–item relationships.

The starting point is the standard linear autoencoder view of recommendation. For a user interaction vector $\mathbf{X}_{u*}$, prediction is expressed as

$\hat{\mathbf{X}}_{u*} = \mathbf{X}_{u*}\mathbf{B},$

where $\mathbf{B} \in \mathbb{R}^{n \times n}$ is an item-to-item weight matrix constrained to have zero diagonal. In EASE, $\mathbf{B}$ is learned by reconstructing $\mathbf{X}$ under $L_2$ regularization. The resulting method is analytically solvable, fast, and stable, but it does not natively exploit item text (Moon et al., 19 Aug 2025).

The paper’s motivation is that LLMs can produce rich semantic representations from item text, yet many recent LLM-based recommenders are non-linear neural models that are expensive to train or infer and do not retain the closed-form simplicity of linear autoencoders. At the same time, prior attempts to incorporate text into linear autoencoders use a sparse multi-hot tag or word matrix $\mathbf{X} \in \{0,1\}^{m \times n},$0, which captures lexical co-occurrence but not semantic similarity such as paraphrastic equivalence between terms like “running shoes” and “athletic sneakers” (Moon et al., 19 Aug 2025).

A common misconception is that adding text to a linear recommender is equivalent to concatenating or blending side information with interaction data. L3AE is explicitly not formulated that way. Its design separates semantic structure learning from collaborative reconstruction, then couples them through a distillation-style regularizer. This suggests that the model treats semantic and collaborative information as statistically distinct sources rather than as homogeneous features.

2. Formal model and two-phase optimization

L3AE introduces two matrices beyond the interaction matrix $\mathbf{X} \in \{0,1\}^{m \times n},$1. For each item $\mathbf{X} \in \{0,1\}^{m \times n},$2, an LLM-derived embedding $\mathbf{X} \in \{0,1\}^{m \times n},$3 is computed and stacked column-wise into

$\mathbf{X} \in \{0,1\}^{m \times n},$4

The model then defines:

• $\mathbf{X} \in \{0,1\}^{m \times n},$5: semantic item-to-item correlation matrix.
• $\mathbf{X} \in \{0,1\}^{m \times n},$6: final item-to-item weight matrix used for recommendation (Moon et al., 19 Aug 2025).

The baseline EASE objective is

$\mathbf{X} \in \{0,1\}^{m \times n},$7

Its closed-form solution is

$\mathbf{X} \in \{0,1\}^{m \times n},$8

where

$\mathbf{X} \in \{0,1\}^{m \times n},$9

Prediction for user $m$0 and item $m$1 is

$m$2

L3AE preserves this inference form and changes the training procedure (Moon et al., 19 Aug 2025).

In Phase 1, the semantic matrix $m$3 is learned by applying an EASE-style objective to $m$4: $m$5 The corresponding closed-form solution is

$m$6

with

$m$7

In Phase 2, the final recommendation matrix $m$8 is learned from interactions while distilling semantic correlations: $m$9 Its closed-form solution is

$n$0

or equivalently

$n$1

where

$n$2

and

$n$3

When $n$4, the model reduces to vanilla EASE on $n$5 (Moon et al., 19 Aug 2025).

The central structural idea is therefore straightforward: $n$6 acts as a semantic “teacher,” and $n$7 is a collaborative “student” constrained to fit interaction data while remaining close to that teacher matrix.

3. Semantic representation and the role of the teacher matrix

For each item, L3AE concatenates textual attributes into the prompt Title: <title>; Category: <category>; Brand: <brand>; Description: <description> and feeds this prompt into an LLM embedding model such as NV-Embed-v2, LLaMA-3.2-3B, or Qwen3-Embedding-8B (Moon et al., 19 Aug 2025). The model then obtains token-level hidden states from the final layer and averages those token embeddings to produce a fixed-size item vector $n$8. Stacking all such vectors yields the dense semantic matrix $n$9.

This construction replaces the traditional sparse tag-item matrix $\mathbf{X}_{u*}$0 with a dense semantic representation. The paper explicitly contrasts this with earlier text-aware linear autoencoders that operate on multi-hot tags or words, which only encode lexical overlap. L3AE instead derives an item–item structure from LLM-induced semantics (Moon et al., 19 Aug 2025).

A naive alternative would be to use cosine similarity over the item embeddings: $\mathbf{X}_{u*}$1 The paper reports that this is empirically weaker than learning $\mathbf{X}_{u*}$2 by applying an EASE-style objective directly to $\mathbf{X}_{u*}$3. Under that formulation, each item’s semantic vector is regressed on the semantic vectors of all other items, so $\mathbf{X}_{u*}$4 becomes the weight of item $\mathbf{X}_{u*}$5 when reconstructing item $\mathbf{X}_{u*}$6’s semantic representation (Moon et al., 19 Aug 2025).

The paper further reports a PCA analysis in which $\mathbf{X}_{u*}$7 exhibits a low effective rank, with most variance concentrated in a few principal components, whereas $\mathbf{X}_{u*}$8 has a slower singular-value decay and more noise. This motivates the two-phase architecture: semantics and interactions are not fused at the raw-feature level but are handled separately before semantic structure is distilled into collaborative structure. A plausible implication is that the authors view spectral mismatch between $\mathbf{X}_{u*}$9 and $\hat{\mathbf{X}}_{u*} = \mathbf{X}_{u*}\mathbf{B},$0 as a reason to avoid direct stacking or undifferentiated joint training.

4. Relation to prior linear autoencoders and LLM-based recommenders

L3AE is closely related to the standard item–item LAE paradigm but differs in how side information is introduced. In a conventional LAE such as EASE, the training problem depends only on $\hat{\mathbf{X}}_{u*} = \mathbf{X}_{u*}\mathbf{B},$1, and the core computation is inversion of the regularized covariance matrix

$\hat{\mathbf{X}}_{u*} = \mathbf{X}_{u*}\mathbf{B},$2

L3AE preserves this analytic item–item structure at inference time, because recommendation scores are still computed as

$\hat{\mathbf{X}}_{u*} = \mathbf{X}_{u*}\mathbf{B},$3

The departure lies entirely in the training objective, where $\hat{\mathbf{X}}_{u*} = \mathbf{X}_{u*}\mathbf{B},$4 is injected through the distillation regularizer $\hat{\mathbf{X}}_{u*} = \mathbf{X}_{u*}\mathbf{B},$5 (Moon et al., 19 Aug 2025).

The paper contrasts L3AE with earlier text-aware LAEs, especially CEASE and Add-EASE. In CEASE, a stacked matrix

$\hat{\mathbf{X}}_{u*} = \mathbf{X}_{u*}\mathbf{B},$6

is formed and EASE is applied to $\hat{\mathbf{X}}_{u*} = \mathbf{X}_{u*}\mathbf{B},$7. In Add-EASE, one matrix is learned from $\hat{\mathbf{X}}_{u*} = \mathbf{X}_{u*}\mathbf{B},$8, another from $\hat{\mathbf{X}}_{u*} = \mathbf{X}_{u*}\mathbf{B},$9, and the two are blended as

$\mathbf{B} \in \mathbb{R}^{n \times n}$0

These methods use a sparse lexical representation $\mathbf{B} \in \mathbb{R}^{n \times n}$1 rather than dense LLM semantics, and they do not define an explicit teacher matrix learned in a dedicated semantic phase (Moon et al., 19 Aug 2025).

The paper also compares L3AE with LLM-enhanced non-linear recommenders such as AlphaRec, RLMRec-Con, and RLMRec-Gen, as well as interaction-only non-linear baselines such as LightGCN and SimGCL. L3AE differs from those systems in being purely linear after item embeddings are extracted offline: there is no gradient-based training of a deep collaborative model, and no end-to-end neural integration of LLMs with recommendation (Moon et al., 19 Aug 2025).

An important point of interpretation is that L3AE is not presented as an LLM recommender in the strong sense of using an LLM to score user–item pairs directly. Rather, it uses LLM-derived item embeddings once to construct $\mathbf{B} \in \mathbb{R}^{n \times n}$2, from which a semantic correlation matrix is analytically learned. The resulting recommender remains an item–item collaborative filter.

5. Theoretical properties, optimization, and computational profile

Both phases of L3AE are described as convex quadratic problems with linear constraints. Phase 1 is quadratic in $\mathbf{B} \in \mathbb{R}^{n \times n}$3 with $\mathbf{B} \in \mathbb{R}^{n \times n}$4 regularization and a zero-diagonal constraint, and Phase 2 is quadratic in $\mathbf{B} \in \mathbb{R}^{n \times n}$5 with two $\mathbf{B} \in \mathbb{R}^{n \times n}$6 terms and the same diagonal constraint. The paper states that such problems yield unique global optima, solvable via normal equations and Lagrange multipliers, and provides explicit closed forms for both phases (Moon et al., 19 Aug 2025).

This closed-form character is a defining property of the model. It implies deterministic optimization, no local minima, no mini-batch training, and no backpropagation through a deep recommender. Training consists of computing $\mathbf{B} \in \mathbb{R}^{n \times n}$7 offline, constructing covariance matrices, inverting them, and applying the closed-form formulas.

Under naive dense algebra, the paper gives the following complexity profile. In Phase 1, computing $\mathbf{B} \in \mathbb{R}^{n \times n}$8 costs $\mathbf{B} \in \mathbb{R}^{n \times n}$9, and inverting $\mathbf{B}$0 costs $\mathbf{B}$1. In Phase 2, computing $\mathbf{B}$2 costs $\mathbf{B}$3 in dense algebra, though $\mathbf{B}$4 is sparse in practice, and inverting $\mathbf{B}$5 costs $\mathbf{B}$6. Total training is therefore dominated by $\mathbf{B}$7 matrix inversions and $\mathbf{B}$8 multiplications (Moon et al., 19 Aug 2025).

At inference time, recommendation for user $\mathbf{B}$9 is a single multiplication

$\mathbf{X}$0

Worst-case cost is $\mathbf{X}$1, but the paper notes that the sparsity of $\mathbf{X}$2 can be exploited, reducing cost to $\mathbf{X}$3, with further reduction possible through top-$\mathbf{X}$4 computation strategies (Moon et al., 19 Aug 2025).

The memory requirement is $\mathbf{X}$5, since $\mathbf{X}$6 must be stored. For $\mathbf{X}$7, the paper notes that this is large but feasible on modern servers, and it mentions lower precision and top-$\mathbf{X}$8 sparsification as possible practical reductions. The paper does not fully detail such scaling strategies. This suggests that L3AE is naturally suited to moderate catalog sizes and would require approximation schemes for much larger item universes.

6. Experimental evaluation, empirical behavior, and limitations

The empirical study uses three Amazon 2023 review subsets: Games, Toys, and Books (Moon et al., 19 Aug 2025). Their reported statistics are:

• Games: 5,222 users, 2,676 items, 85,690 interactions, density $\mathbf{X}$9 (0.62%).
• Toys: 14,750 users, 13,358 items, 250,509 interactions, density $L_2$0 (0.13%).
• Books: 25,300 users, 30,966 items, 640,901 interactions, density $L_2$1 (0.082%).

Preprocessing retains ratings $L_2$2, applies 10-core filtering, and uses an 8:1:1 train/validation/test split. Evaluation uses Recall@k and NDCG@k for $L_2$3, under an “Average over all” protocol that ranks each user against all non-interacted items (Moon et al., 19 Aug 2025).

With NV-Embed-v2 as the LLM backbone, the reported R@20 and N@20 values are:

• Games: AlphaRec $L_2$4, EASE $L_2$5, L3AE $L_2$6.
• Toys: AlphaRec $L_2$7, EASE $L_2$8, L3AE $L_2$9.
• Books: AlphaRec $\mathbf{X} \in \{0,1\}^{m \times n},$00, EASE $\mathbf{X} \in \{0,1\}^{m \times n},$01, L3AE $\mathbf{X} \in \{0,1\}^{m \times n},$02 (Moon et al., 19 Aug 2025).

Averaged over datasets, the paper reports that L3AE improves over the best non-linear LLM-based model, AlphaRec, by about 27.6% in Recall@20 and 39.3% in NDCG@20, and improves over plain EASE by about 14.7% in R@20 and 15.3% in N@20 (Moon et al., 19 Aug 2025). The paper also reports stronger long-tail performance on the Games dataset relative to EASE and tag-based variants.

Ablation results further clarify the model’s behavior. When CEASE and Add-EASE are modified by replacing $\mathbf{X} \in \{0,1\}^{m \times n},$03 with the dense semantic matrix $\mathbf{X} \in \{0,1\}^{m \times n},$04, L3AE still outperforms them by about 1.6% in R@20 and N@20 on average, reaching up to about 4.5% on Toys (Moon et al., 19 Aug 2025). This directly supports the claim that the benefit is not merely due to using better text embeddings; it depends on the two-phase distillation architecture.

The hyperparameter study emphasizes a specific regularization strategy. The paper first finds the best EASE regularization $\mathbf{X} \in \{0,1\}^{m \times n},$05 on $\mathbf{X} \in \{0,1\}^{m \times n},$06, then enforces

$\mathbf{X} \in \{0,1\}^{m \times n},$07

in L3AE to avoid over-regularizing $\mathbf{X} \in \{0,1\}^{m \times n},$08. It reports that performance drops sharply if $\mathbf{X} \in \{0,1\}^{m \times n},$09 deviates from this optimum (Moon et al., 19 Aug 2025).

Backbone comparisons among NV-Embed-v2 (7B), LLaMA-3.2-3B, and Qwen3-Embedding-8B show that L3AE consistently outperforms baselines regardless of the embedding model. The paper states that parameter count is not the main factor; instead, domain alignment and training data of the LLM matter more, with NV-Embed-v2 performing best because it is pretrained with e-commerce data such as AmazonReviews (Moon et al., 19 Aug 2025).

The paper identifies several limitations, directly or by implication. First, the method has quadratic memory and cubic time in the number of items because it stores and inverts full $\mathbf{X} \in \{0,1\}^{m \times n},$10 matrices. Second, performance depends on the quality and domain alignment of the item embeddings. Third, the method assumes each item has meaningful textual metadata. Fourth, it remains primarily a collaborative model at inference, so cold-start users are not directly addressed (Moon et al., 19 Aug 2025).

Taken together, these results place L3AE in a specific part of the recommender-systems design space: a semantic-aware item–item linear autoencoder that attempts to retain the robustness of linear collaborative filtering in sparse regimes while using LLM-derived semantics to improve under-observed items. A plausible implication is that its main contribution is not only empirical improvement but also a methodological template for injecting dense semantic structure into analytically solvable recommenders without abandoning closed-form optimization.