Regularized Linear Discriminant Analysis

• Regularized LDA is a framework that enhances classical LDA by introducing techniques such as ridge shrinkage and sparse constraints to stabilize covariance estimation in high-dimensional regimes.
• It improves numerical stability and reduces estimation bias by adjusting the covariance matrix using methods like covariance shrinkage and adaptive interpolation between LDA and QDA.
• Extensions of RLDA include robust, deep, and transfer learning variations that tailor discriminant models to challenges such as contamination, limited samples, and complex data structures.

Regularized Linear Discriminant Analysis (RLDA) denotes a family of modifications of classical Linear Discriminant Analysis that stabilize classification when covariance estimation is unreliable, particularly in high-dimensional regimes where the number of features is comparable to or larger than the sample size. Across the literature, regularization appears as covariance shrinkage, ridge loading, adaptive interpolation between LDA and QDA, sparse or low-rank constraints on discriminant structure, robustification against contamination, and, in more recent work, geometric constraints in latent spaces learned by deep networks (Tabassum et al., 2018, Wang et al., 2017, Bing et al., 2024, Tezekbayev et al., 4 Jan 2026).

1. Statistical foundation and classical formulation

Classical LDA assumes class-conditional Gaussian distributions with a common covariance matrix. In a multiclass setting, if $x \in \mathbb{R}^d$ and class $c$ has mean $\mu_c$, global mean $\mu$, and class size $n_c$, the within-class and between-class scatter matrices are

$$S_w = \sum_{c=1}^C \sum_{i=1}^{n_c} (x_i^{(c)} - \mu_c)(x_i^{(c)} - \mu_c)^T, \qquad S_b = \sum_{c=1}^C n_c (\mu_c - \mu)(\mu_c - \mu)^T.$$

The Fisher criterion for a projection matrix $W$ is

$$J(W) = \operatorname{tr}\!\big((W^T S_w W)^{-1} W^T S_b W\big),$$

and the corresponding directions solve the generalized eigenproblem $S_b v = \lambda S_w v$ (Lu, 2021).

In the standard Gaussian classification form, with pooled covariance estimate $\hat{\Sigma}$, class mean $c$0, and prior $c$1, the LDA discriminant is

$c$2

and prediction uses $c$3. In binary form, this is equivalent to comparing

$c$4

to a prior-dependent threshold (Tabassum et al., 2018).

The difficulty is that the pooled sample covariance can be singular or severely ill-conditioned when $c$5, and even when invertible it can induce unstable discriminants. This is the central motivation for RLDA.

2. Core regularization mechanisms

The most classical RLDA device is ridge or shrinkage regularization of the within-class scatter. In high-dimensional or small-sample settings one replaces $c$6 by

$c$7

or by shrinkage toward identity,

$c$8

with $c$9, $\mu_c$0, and $\mu_c$1. The stated motivations are numerical stability, invertibility, and reduction of estimation bias toward large eigenvalues (Lu, 2021).

A second major line is Friedman’s Regularized Discriminant Analysis, which explicitly traces a path between QDA and LDA. For class $\mu_c$2,

$\mu_c$3

Here $\mu_c$4. The special cases are QDA when $\mu_c$5, LDA when $\mu_c$6, and ridge-regularized LDA when $\mu_c$7 (Aerts et al., 2016).

This interpolation clarifies a common misconception: regularized LDA is not a single estimator. Some variants regularize the common covariance of LDA, while others regularize the entire continuum between pooled and class-specific covariance models. Related methods extend this idea through similarity penalties rather than exact equality. Joint Graphical Lasso discriminant analysis, for example, introduces a similarity penalty across class precision matrices and thereby traces a path between QDA-like and LDA-like structures without forcing all covariances to be identical (Aerts et al., 2016).

3. High-dimensional asymptotics, bias, and parameter choice

A substantial part of the RLDA literature studies the regime in which $\mu_c$8 and $\mu_c$9 grow together. In the two-class Gaussian model with common covariance, explicit asymptotic misclassification formulas have been derived both for LDA and for ridge-regularized LDA under $\mu$0. One notable result is that unregularized LDA suffers a dimension effect through the factor $\mu$1, while RLDA replaces this by spectral functionals $\mu$2, $\mu$3, $\mu$4, and $\mu$5 derived from the Marčenko–Pastur equation. The same analysis identifies a bias term caused by unequal sample sizes and proposes bias-corrected LDA and RLDA intercepts with smaller asymptotic misclassification rates (Wang et al., 2017).

Parameter selection is therefore not merely a computational choice. In one line of work, a nonlinear ridge-type estimator of the inverse covariance is introduced,

$\mu$6

where $\mu$7. This yields a nonlinear RLDA classifier together with asymptotic and consistent estimators of the misclassification rate, so that $\mu$8 can be chosen by a one-dimensional grid search minimizing the estimated error (Mahadi et al., 2024).

A more recent structural analysis emphasizes that RLDA performance depends not only on $\mu$9 but also on how the mean difference aligns with the covariance eigenstructure. In that framework, the empirical spectral distribution $n_c$0 and the alignment measure $n_c$1 enter a non-asymptotic approximation of the misclassification rate. This leads to “Spectral Enhanced Discriminant Analysis” (SEDA), which modifies spiked eigenvalues of the covariance and then applies a regularized inverse of the form $n_c$2 rather than $n_c$3 (Zhang et al., 22 Jul 2025).

These results collectively suggest that RLDA is best viewed as a family of spectral filters. The precise filter may be linear shrinkage, nonlinear shrinkage, or spike-specific enhancement, but in each case classification performance is controlled by how the method reshapes unstable directions of the covariance spectrum.

4. Sparse, adaptive, and robust extensions

One major branch of RLDA augments covariance regularization with feature selection. “Compressive Regularized Discriminant Analysis” constructs

$n_c$4

then forms

$n_c$5

where $n_c$6 keeps the $n_c$7 rows with largest $n_c$8-norms. This yields joint sparsity across classes and is explicitly motivated by gene selection in microarray studies (Tabassum et al., 2018).

A related, but structurally different, approach is “L1-Pooled Discriminant Analysis.” In the two-class case it solves

$n_c$9

with $$S_w = \sum_{c=1}^C \sum_{i=1}^{n_c} (x_i^{(c)} - \mu_c)(x_i^{(c)} - \mu_c)^T, \qquad S_b = \sum_{c=1}^C n_c (\mu_c - \mu)(\mu_c - \mu)^T.$$0 and $$S_w = \sum_{c=1}^C \sum_{i=1}^{n_c} (x_i^{(c)} - \mu_c)(x_i^{(c)} - \mu_c)^T, \qquad S_b = \sum_{c=1}^C n_c (\mu_c - \mu)(\mu_c - \mu)^T.$$1. This adaptively pools entries of the precision matrices: $$S_w = \sum_{c=1}^C \sum_{i=1}^{n_c} (x_i^{(c)} - \mu_c)(x_i^{(c)} - \mu_c)^T, \qquad S_b = \sum_{c=1}^C n_c (\mu_c - \mu)(\mu_c - \mu)^T.$$2 recovers QDA, while sufficiently large $$S_w = \sum_{c=1}^C \sum_{i=1}^{n_c} (x_i^{(c)} - \mu_c)(x_i^{(c)} - \mu_c)^T, \qquad S_b = \sum_{c=1}^C n_c (\mu_c - \mu)(\mu_c - \mu)^T.$$3 yields LDA with common covariance equal to the pooled empirical covariance (Simon et al., 2011).

For multiclass Fisher discriminants, sparsity can also be imposed directly on the discriminant directions while avoiding unstable covariance constraints. “Sparse Fisher’s discriminant analysis with thresholded linear constraints” replaces the usual $$S_w = \sum_{c=1}^C \sum_{i=1}^{n_c} (x_i^{(c)} - \mu_c)(x_i^{(c)} - \mu_c)^T, \qquad S_b = \sum_{c=1}^C n_c (\mu_c - \mu)(\mu_c - \mu)^T.$$4 constraints by thresholded proxies based on $$S_w = \sum_{c=1}^C \sum_{i=1}^{n_c} (x_i^{(c)} - \mu_c)(x_i^{(c)} - \mu_c)^T, \qquad S_b = \sum_{c=1}^C n_c (\mu_c - \mu)(\mu_c - \mu)^T.$$5, and proves asymptotic consistency and asymptotic optimality for arbitrary numbers of classes in high dimensions (Luo et al., 2015).

Robustness against contamination motivates another large subfamily. Aerts and Wilms propose cellwise robust regularized discriminant analysis by replacing classical covariance inputs with

$$S_w = \sum_{c=1}^C \sum_{i=1}^{n_c} (x_i^{(c)} - \mu_c)(x_i^{(c)} - \mu_c)^T, \qquad S_b = \sum_{c=1}^C n_c (\mu_c - \mu)(\mu_c - \mu)^T.$$6

and using marginal medians for class means. Their framework covers robust RDA, robust Graphical Lasso discriminants, and robust Joint Graphical Lasso discriminants, remains computable for $$S_w = \sum_{c=1}^C \sum_{i=1}^{n_c} (x_i^{(c)} - \mu_c)(x_i^{(c)} - \mu_c)^T, \qquad S_b = \sum_{c=1}^C n_c (\mu_c - \mu)(\mu_c - \mu)^T.$$7, and is explicitly designed for cellwise outliers rather than only rowwise contamination (Aerts et al., 2016).

5. Alternative formulations: regression, functions, reduced rank, and transfer

RLDA is closely related to multivariate regression. A recent regression-based formulation defines

$$S_w = \sum_{c=1}^C \sum_{i=1}^{n_c} (x_i^{(c)} - \mu_c)(x_i^{(c)} - \mu_c)^T, \qquad S_b = \sum_{c=1}^C n_c (\mu_c - \mu)(\mu_c - \mu)^T.$$8

and establishes the explicit linkage

$$S_w = \sum_{c=1}^C \sum_{i=1}^{n_c} (x_i^{(c)} - \mu_c)(x_i^{(c)} - \mu_c)^T, \qquad S_b = \sum_{c=1}^C n_c (\mu_c - \mu)(\mu_c - \mu)^T.$$9

This identity turns multiclass LDA into a post-processed regularized regression problem and enables excess-risk analysis for $W$0-regularized and reduced-rank estimators in the LDA setting (Bing et al., 2024).

In functional data analysis, regularization is unavoidable because the covariance operator is compact and not invertible in the ordinary sense. “Sparse Functional Linear Discriminant Analysis” therefore solves

$W$1

which induces zero regions in the discriminant function and yields misclassification error converging to the Bayes error under Gaussian assumptions (Park et al., 2020).

Reduced-rank formulations provide another viewpoint. In large-scale high-dimensional settings, reduced-rank LDA can be written as the least-squares problem

$W$2

The randomized Kaczmarz method then converges toward the minimum-$W$3-norm solution and thereby supplies implicit regularization without an explicit penalty term, a property used in “Large Scale High-Dimensional Reduced-Rank Linear Discriminant Analysis” (Chi, 11 Feb 2026).

Transfer learning introduces yet another regularization axis. In “Transfer learning via Regularized Linear Discriminant Analysis,” the target discriminant is estimated as a weighted combination of ridge estimators from target and source populations,

$W$4

with weights chosen either to minimize discriminant estimation error or to minimize classification error under high-dimensional random-effects asymptotics (Zhang et al., 5 Jan 2025).

6. Deep and probabilistic generalizations

Deep extensions reinterpret LDA as a training objective in latent space. “Regularized Deep Linear Discriminant Analysis” replaces categorical cross-entropy by an eigenvalue-based Fisher objective and regularizes the within-class scatter as

$W$5

This attenuates off-diagonal covariance terms while preserving diagonal variances, and the paper reports improvements over both DLDA and cross-entropy baselines on STL-10 and CIFAR-10 (Lu, 2021).

Likelihood-based deep LDA reveals a different failure mode. Joint maximum-likelihood training of an unconstrained deep LDA head can drive class means together and collapse covariance. One remedy is the “Discriminative Negative Log-Likelihood”

$W$6

which adds a penalty proportional to the mixture density and thereby discourages overlap of class-conditional Gaussians. This objective was reported to match softmax accuracy on synthetic and image benchmarks while substantially improving calibration (Tezekbayev et al., 4 Jan 2026).

A more restrictive but geometrically transparent alternative fixes class means to the vertices of a regular simplex and constrains the shared covariance to be spherical: $W$7 Under these constraints, maximum-likelihood training becomes stable, latent clusters remain well separated, and the resulting deep LDA models achieve accuracy competitive with softmax on Fashion-MNIST, CIFAR-10, and CIFAR-100 (Tezekbayev et al., 4 Jan 2026).

The optimization geometry of deep LDA itself can also act as an implicit regularizer. For an $W$8-layer diagonal linear network minimizing the Rayleigh quotient

$W$9

gradient flow conserves $$J(W) = \operatorname{tr}\!\big((W^T S_w W)^{-1} W^T S_b W\big),$$0. The special cases are $$J(W) = \operatorname{tr}\!\big((W^T S_w W)^{-1} W^T S_b W\big),$$1 for $$J(W) = \operatorname{tr}\!\big((W^T S_w W)^{-1} W^T S_b W\big),$$2, $$J(W) = \operatorname{tr}\!\big((W^T S_w W)^{-1} W^T S_b W\big),$$3 for $$J(W) = \operatorname{tr}\!\big((W^T S_w W)^{-1} W^T S_b W\big),$$4, and a nonconvex $$J(W) = \operatorname{tr}\!\big((W^T S_w W)^{-1} W^T S_b W\big),$$5 quasi-norm for $$J(W) = \operatorname{tr}\!\big((W^T S_w W)^{-1} W^T S_b W\big),$$6, which gives a depth-dependent sparsity-like implicit bias (Li, 3 Mar 2026).

A probabilistic parallel appears in speaker verification, where covariance-regularized PLDA back-ends use interpolation and sparsity rather than diagonal pruning. Interpolated PLDA updates the between-speaker inverse covariance by

$$J(W) = \operatorname{tr}\!\big((W^T S_w W)^{-1} W^T S_b W\big),$$7

while sparse PLDA solves an $$J(W) = \operatorname{tr}\!\big((W^T S_w W)^{-1} W^T S_b W\big),$$8-penalized inverse-covariance problem. In domain adaptation experiments, both approaches outperformed diagonal regularization, illustrating that the logic of RLDA extends beyond classical Gaussian discriminants to broader latent-variable discriminant models (Peng et al., 2022).

Regularized LDA is therefore less a single classifier than a methodological umbrella. Its central problem is stable estimation of discriminative structure under limited data, contamination, or model mismatch. Its central idea is to trade variance for structure: by shrinking, sparsifying, robustifying, pooling, geometrically constraining, or spectrally correcting covariance-dependent quantities, RLDA converts Fisher’s classical rule into a flexible framework for modern high-dimensional and representation-learning settings.