Linear Autoencoders (LAEs)

• LAEs are unsupervised models that linearly map inputs to a latent space and back, minimizing squared reconstruction error.
• They recover principal components through closed-form solutions and regularization, providing robust dimensionality reduction and feature extraction.
• Recent enhancements integrate denoising, semantic regularization with LLMs, and relaxed diagonal constraints to boost long-tail performance in recommender systems.

A linear autoencoder (LAE) is a model that performs unsupervised representation learning via a linear mapping from inputs to latent space and back to input space, optimized to minimize the reconstruction error typically measured by the squared Euclidean norm. LAEs are foundational for dimensionality reduction and feature extraction, and form the basis of many collaborative filtering recommenders. They admit closed-form characterizations, deep links to principal component analysis (PCA), and possess favorable optimization properties. Recent advances have extended LAEs to denoising frameworks, PAC-Bayes generalization analysis, integration with LLMs, relaxation of diagonal constraints, and implicit rank selection via overparameterized architectures.

1. Formal Definition, Objective, and Statistical Interpretation

A basic LAE is parametrized by encoder $B \in \mathbb{R}^{k \times n}$, decoder $A \in \mathbb{R}^{n \times k}$, acting on input $x \in \mathbb{R}^n$: $\hat{x} = A B x$ The canonical objective is the empirical mean squared reconstruction error: $$\mathcal{L}(A, B; X) = \frac{1}{m} \sum_{i=1}^m \| x_i - A B x_i \|_2^2 = \frac{1}{m}\|X - ABX\|_F^2$$ where $X \in \mathbb{R}^{n \times m}$ denotes the dataset. In collaborative filtering, the model is typically reframed as learning a square matrix $W \in \mathbb{R}^{n \times n}$ acting on user–item interaction matrix $H$ with a zero-diagonal constraint: $$\min_{W} \| H - W H \|_F^2 \quad \text{subject to } \operatorname{diag}(W) = 0$$ An LAE is equivalently a constrained multivariate linear regression on bounded data; this enables the transfer of PAC-Bayes generalization bounds from regression to LAEs (Guo et al., 15 Dec 2025).

2. Closed-Form Solutions, Principal Components, and Regularization

Fundamental theory establishes that the global minima of the LAE correspond precisely to projections onto principal components. In the mean-centered setting, LAEs span the top-$k$ principal subspace; with suitable $L_2$ regularization and solution symmetry, the encoder/decoder weights align with the ordered principal directions (Kunin et al., 2019, Plaut, 2018):

• The classical Eckart–Young–Mirsky theorem ensures that the optimal rank-$k$ LAE implements the PCA projector.
• With non-uniform regularization or nested dropout, the learned weights are axis-aligned with the principal components (up to order and sign), breaking rotational symmetry (Bao et al., 2020).

Table: Key LAE-PCA Relationships

Property	Reference	Technical Statement
Spanning principal subspace	(Plaut, 2018, Kunin et al., 2019)	LAE recovers PCA subspace
Axis alignment (symmetry break)	(Bao et al., 2020)	Regularization aligns to PCs
Saddle points/Minima	(Baldi et al., 2011)	Only PCA projectors are global minima

Regularization modulates the solution form. L2 regularization shrinks projections along low-variance directions, and nested dropout imposes learnable ordering (Bao et al., 2020).

3. Characterization of Critical Points and Optimization Landscape

The full critical manifold of the LAE objective is well understood:

• For real or complex LAEs, critical points correspond to projectors onto $p$-dimensional eigenspaces of the input covariance; only those aligned with the top eigenvectors yield global minima, all others are saddles (Baldi et al., 2011).
• The optimization landscape is strictly convex for the squared error objective under diagonal constraints (Moon et al., 2023).
• Empirical illustrations confirm rapid alignment to the principal subspace, and with regularization, eventual axis-alignment to the true PC directions (Kunin et al., 2019, Plaut, 2018, Bao et al., 2020).

4. Extensions: Diagonal Constraints, Relaxations, and Denoising

The item–item weight matrix in collaborative filtering LAEs is often subject to zero-diagonal constraints to prevent trivial self-predictions (as in EASE). However, strict diagonal constraints degrade performance on low-variance ("long-tail") items by disproportionately penalizing their directions (Moon et al., 2023):

• Relaxed Linear Autoencoders (RLAE) and Relaxed Denoising Linear Autoencoders (RDLAE) softly enforce $0 \leq W_{ii} \leq \xi$, enabling interpolation between unconstrained ridge regression and strict zero-diag models.
• Empirically, relaxed constraints improve long-tail recall and NDCG, especially in skewed datasets (Moon et al., 2023).
• Denoising regularization (as in EDLAE and RDLAE) further balances head and tail item performance and admits closed-form solutions via eigendecomposition (Steck et al., 2021, Moon et al., 2023).

5. Implicit Rank Selection and Overparameterized Architectures

Deep linear bottleneck sub-networks in autoencoders can perform implicit greedy rank selection via gradient descent. Analysis of training dynamics under balanced orthogonal initialization demonstrates stepwise emergence of singular modes, with the most prominent principal directions learned first (Sun et al., 2021):

• Orthogonal initialization and depth-normalized learning rates stabilize rank selection and mitigate the dependence of solution quality on architecture depth.
• In nonlinear autoencoders, the greedy linear bottleneck aligns latent rank with downstream discriminative or generative tasks, often eliminating the need for explicit architectural search.

6. Recent Innovations: LLM Integration and Semantic Regularization

Recent work integrates LLM embeddings into LAEs for recommendations. Models such as L³AE construct a semantic item-to-item correlation matrix from LLM-derived item representations, then learn a collaborative item-to-item matrix regularized toward these semantic correlations (Moon et al., 19 Aug 2025):

• Both phases are solved in closed form. Phase I implements semantic item correlations via ridge regression on LLM embeddings, enforcing zero diagonals. Phase II learns the LAE weights from user–item co-occurrence, penalized toward the semantic matrix.
• This two-phase closed-form solution achieves global optimality, computational efficiency, and robustness in highly sparse settings.
• Empirical benchmarks show consistent outperformance relative to previous state-of-the-art, with especially marked improvements for infrequent items.

7. Generalization Theory, Design Guidelines, and Practical Considerations

The first PAC-Bayes generalization bound for LAEs was established, demonstrating that LAEs are statistically consistent and can be tuned via risk-driven criteria rather than metric-specific hyperparameter search (Guo et al., 15 Dec 2025):

• The PAC-Bayes bound, computed via analytic eigendecomposition, correlates strongly with Recall@K and NDCG@K.
• Zero-diagonal constraints systematically tighten generalization, rationalizing their adoption in practice.
• Regularization strength should be tailored to dataset skew; for high-popularity bias, increase the ridge penalty; for milder skew, reduce it (Moon et al., 2023).
• Relaxing diagonal constraints ($\xi > 0$ in RLAE/RDLAE) is consistently beneficial for tail-item accuracy.

In summary, LAEs represent a uniquely tractable and theoretically grounded family of unsupervised representation learners, achieving globally optimal dimension reduction, adaptability to semantic regularization, robust long-tail performance under relaxed constraints, and certifiable generalization in large-scale recommender systems (Baldi et al., 2011, Plaut, 2018, Kunin et al., 2019, Steck et al., 2021, Moon et al., 2023, Moon et al., 19 Aug 2025, Guo et al., 15 Dec 2025, Sun et al., 2021).