Continual Ridge Regression

• Continual ridge regression is an extension of standard ridge regression that enables sequential updates using flexible, matrix-valued penalties and data-driven regularization.
• It adapts to diverse data sources in high-dimensional domains like omics and time-series, effectively mitigating catastrophic forgetting while leveraging past information.
• Recent research shows its optimal, direction-adaptive regularization achieves oracle-like error rates and relates closely to early stopping in gradient descent.

Continual Ridge Regression generalizes standard ridge regression to enable sequential or adaptive updating of linear models in the presence of new tasks, heterogeneous data sources, or high-dimensional feature spaces. The framework accommodates weighted penalization, data-driven regularization toward nonzero targets, and flexible penalty matrices that can reflect domain-specific structure or prior knowledge. This approach is especially pertinent for high-dimensional empirical settings (such as omics, signal processing, and time-series), where empirical non-identifiability is prevalent and catastrophic forgetting must be mitigated. Recent theoretical developments have characterized both the statistical properties and generalization performance of continual ridge regression in high-dimensional regimes, established asymptotic equivalence to oracle estimators under optimal regularization, and elucidated its connection with early stopping procedures.

1. Generalized Ridge Regression Principles

Standard ridge regression augments the least squares loss function by an $\ell_2$ penalty, yielding the solution

$$\hat{\beta}_{ridge} = (X^\top X + \lambda I_p)^{-1} X^\top Y,$$

where $X$ is the $n\times p$ design matrix, $Y$ the response vector, and $\lambda > 0$ the regularization parameter. Continual ridge regression extends this by allowing

• Weighted least squares: $W$ is an $n\times n$ diagonal matrix reflecting reliability or importance of observations.
• Penalization toward a nonzero target: Coefficient shrinkage can be toward an arbitrary $\beta_0$.
• General quadratic penalties: The penalty is governed by a symmetric positive-definite matrix $\Delta$ instead of $\lambda I_p$, permitting element-specific and even correlated shrinkage.

The generalized ridge loss

$$L(\beta) = (Y - X\beta)^\top W (Y - X\beta) + (\beta - \beta_0)^\top \Delta (\beta - \beta_0)$$

is strictly convex in $\beta$, with analytic solution

$$\hat{\beta}(\Delta) = (X^\top W X + \Delta)^{-1}(X^\top W Y + \Delta \beta_0).$$

This loss can be interpreted as constrained minimization over an ellipsoidal region in parameter space, providing a geometric counterpart to the traditional ball constraint of standard ridge regression (Wieringen, 2015).

2. Continual Learning and Iterative Updating

Continual ridge regression is formulated for sequential learning over a series of tasks $(X_t, y_t)$, updating parameters by solving problems of the form

$$w_t = \arg\min_{w} \left\{ \frac{1}{n_t}\|X_t w - y_t\|^2 + (w - w_{t-1})^\top H_t (w - w_{t-1}) \right\},$$

where $w_{t-1}$ encodes past information, and $H_t$ is a positive-definite (possibly diagonal) matrix-weighted penalty. This formalism allows balancing forward knowledge transfer (leveraging previous estimates) and backwards stability (protecting past information from being overwritten).

Theoretical developments (Zhao et al., 10 Jun 2024) derive an explicit formula for the estimation error in each eigen-direction, showing that optimal (direction-adaptive) choice of $H_t$ results in estimation risk decaying at the same rate as the oracle estimator trained on all data at once. Explicitly, the estimator error in direction $j$ after $T$ tasks follows

$$E[e_j^{(T)}] \approx \frac{\sigma^2}{\lambda_j^{(0)} + Y_j^{(1)}/n_1 + \cdots + Y_j^{(T)}/n_T},$$

where $Y_j^{(t)}$ is the eigenvalue of the $t$-th task covariance along $u_j$ and $\lambda_j^{(0)}$ is the initial regularization.

In contrast, continual ridge regression using a scalar penalty ($H_t = \alpha I$) exhibits suboptimality due to its inability to appropriately balance between heterogeneous information across directions. Lower bounds show that the estimation error for this approach cannot attain the oracle rate, especially under variable task covariances or overparameterization.

3. High-dimensional Asymptotics and Risk Metrics

Recent studies (Zhao et al., 21 Aug 2025) have established exact high-dimensional asymptotics for continual ridge regression using random matrix theory. In the regime $p/n_t \to \gamma_t > 0$, the risk of the continual estimator after $T$ tasks decomposes into bias and variance terms:

• Bias: $$B_X(\hat{\beta}_T ; \beta, \Sigma_0) = \beta^\top [A_1 A_2 \ldots A_T \Sigma_0 A_T \ldots A_2 A_1] \beta$$,
• Variance: $$V_X(\hat{\beta}_T ; \beta, \Sigma_0) = \sigma^2 \sum_{t=1}^T \frac{1}{\lambda_t n_t} \operatorname{Tr}[A_T \ldots A_{t+1} (A_t - A_t^2) A_{t+1} \ldots A_T \Sigma_0]$$, with $$A_t = \lambda_t (\hat{\Sigma}_t + \lambda_t I_p)^{-1}$$.

Three metrics precisely characterize performance in continual learning:

• Average Risk: Weighted aggregate of prediction risks across all tasks.
• Backward Transfer (BWT): Measures memory retention—the change in risk for earlier tasks after learning subsequent tasks.
• Forward Transfer (FWT): Measures the benefit conferred by previous knowledge to new tasks, assessed as the difference between the risk of the sequential estimator and that of the standard ridge estimator on the new task.

Simulations validate these asymptotic formulas and exhibit phenomena such as monotonic risk reduction with proper regularization, transitions from backward to forward transfer, and risk increases under insufficient regularization (catastrophic forgetting).

4. Bayesian Interpretation and Structured Penalization

Generalized ridge regression possesses a direct Bayesian counterpart. With prior $$\beta | \sigma^2 \sim N(\beta_0, \sigma^2 \Delta^{-1})$$, the ridge estimator is precisely the posterior mean given flat prior on variance (Wieringen, 2015). This correspondence allows incorporation of prior domain knowledge, block-specific penalties (using block-diagonal $\Delta$), and borrowing of strength across studies or tasks.

Special cases include:

• Fused Ridge: Penalty matrix $\Delta$ with banded structure enforces similarity between adjacent regression coefficients—useful when predictors are ordered spatially or temporally.
• Unpenalized Covariates: Partitioning coefficients into penalized and unpenalized blocks enables mandatory variables (e.g., age, gender in biomedical studies) to remain unsuppressed.
• Continual Update via Targets: By using previous estimates as the new target $\beta_0$ for each update, models can sequentially assimilate new information while penalizing deviation from learned structure.

5. Connections with Early Stopping and Algorithmic Implications

Theoretical analysis demonstrates an equivalence between early stopping in gradient descent and generalized $\ell_2$-regularization in continual learning (Zhao et al., 10 Jun 2024). Specifically, by controlling the learning rate matrix $A_t$ and stopping time $m_t$ in gradient descent

$$w^{(k)} = w^{(k-1)} - \frac{1}{n_t} A_t X_t^\top (X_t w^{(k-1)} - y_t)$$

for $k=1,\dots,m_t$, one recovers the output of the generalized $\ell_2$ estimator with matching regularization. Properly tuned early stopping thus reproduces the effect of explicit matrix-valued penalization.

Algorithmically, continual ridge regression supports exact sequential updates for sufficient statistics ($X^\top X$, $X^\top y$), permitting efficient online adaptation. Extensions include recursive and square-root updates of the ridge inverse in incremental Broad Learning Systems, which significantly reduce computational complexity and maintain generalization performance in distributed and high-volume data settings (Zhu, 2019).

6. Structural Adaptations and Application Areas

Continual ridge regression is amenable to adaptation for:

• Multi-penalty formulations: Assignment of data-type specific or block-specific penalties permits modeling heterogeneity in multi-omics, multi-view, or multi-modal data (Wiel et al., 2020, Maroni et al., 2023).
• Transfer and domain adaptation: Weighted aggregation of sequential ridge estimates from distinct but related tasks/settings, with weights analytically derived from minimization of estimation/prediction risk and informed by inter-task correlation (Zhang et al., 2023).
• Manifold-valued regression: Intrinsic regularization carried to Riemannian manifolds for time-series prediction and forecasting, as in meteorological track analysis (Nava-Yazdani, 27 Nov 2024).
• Functional regression with adaptive templates: Penalization toward nonzero, data-adaptive templates (often piecewise or sparse), solving with alternating optimization and closed-form reduction in the quadratic components (Belli et al., 2020).

These structures enable ridge-type continual estimators to incorporate domain knowledge, handle non-Euclidean observation spaces, and adapt dynamically to changing data environments.

7. Empirical Results and Practical Implications

Empirical studies confirm the theoretical insights regarding generalization performance:

• Optimal generalized penalties achieve oracle-like error rates.
• Scalar/ridge-based penalties fail to capture information heterogeneity, leading to suboptimal or stagnant error decay, especially under covariate shift or overparameterization.
• Continual updating with appropriately calibrated penalization critical for avoiding catastrophic forgetting and for exploiting accumulated information in sequential data streams.

Simulation-based risk curves showcase monotonic risk reduction under optimal regularization, a transition from backward transfer dominance to forward transfer dominance in long task sequences, and sharp risk increase under under-regularization (demonstrating the fragility of continual learning systems when tuning is poor).

The suggested implication is that regularization parameter tuning and structural adaptation of penalization are central to the successful deployment of continual ridge regression in practice. The theoretical and empirical advances described have direct impact for real-time, high-dimensional, multi-modal, and sequentially evolving applications.