Recursive Weighted Least Squares (RWLS)

• Recursive Weighted Least Squares (RWLS) is a family of online algorithms that estimates model parameters by recursively updating weights and covariance matrices.
• It generalizes classical RLS with flexible weighting schemes to handle nonstationary noise, class imbalance, quantized data, and distributed network settings.
• RWLS variants, including multi-forgetting-factor and diffusion methods, offer robust convergence guarantees and efficient computational performance in dynamic environments.

Recursive Weighted Least Squares (RWLS) is a family of online algorithms for parameter estimation in linear models that generalizes the classic recursive least squares (RLS) framework by introducing weighting schemes on the residuals or updates. RWLS maintains an adaptive recursive update of both the parameter vector and the underlying covariance, yielding efficient algorithms for tracking time-varying systems, handling nonstationary noise, managing class imbalance, and modeling quantized or networked data. The RWLS paradigm encompasses classic exponentially weighted RLS, multi-forgetting-factor extensions, robust and distributed modifications, and exact deterministic online solutions for reweighted tasks.

1. Core Methodology and Variants

At its core, RWLS minimizes a weighted least-squares cost, typically of the form

$$V_t(\theta) = \sum_{s=1}^t w_s [y(s) - \phi(s)^\top \theta]^2,$$

where $\{w_s\}$ are recursively updated weights, $\phi(s)$ are regressors, and $y(s)$ are corresponding measured outputs. Different weightings capture prior knowledge, nonstationarity, or class imbalance.

• Classic RWLS/Exponentially Weighted RLS: Uses $w_s = \lambda^{t-s}$, $\lambda \in (0,1]$, to exponentially discount old data, with the update

$$\theta_t = \theta_{t-1} + K_t [y(t) - \phi(t)^\top \theta_{t-1}],$$

$$K_t = \frac{P_{t-1} \phi(t)}{\lambda + \phi(t)^\top P_{t-1} \phi(t)},$$

$$P_t = \lambda^{-1} [P_{t-1} - K_t \phi(t)^\top P_{t-1}],$$

where $P_t$ is the inverse covariance (Yu et al., 2018).

• Multi-Forgetting-Factor RWLS: Introduces componentwise forgetting factors $\{w_s\}$0 for each parameter, generalized via a mapping $\{w_s\}$1 on the covariance, yielding parameter-specific discounting. Notable variants include:
  • Diagonal update (DI): Discards off-diagonal correlations.
  • Tuned/correlated (TC): Shrinks cross-terms by $\{w_s\}$2 for parameters $\{w_s\}$3.
  • Cubic-spline (CS): Employs a kernel-based quadratic shrinkage (Fraccaroli et al., 2015).
• Non-Iterative Reweighted RLS: In online classification under class imbalance, RWLS achieves exact deterministic class-weighted solutions by recursive maintenance of moment statistics and weight matrices, using rank-one and affine updates, to enforce cost-sensitive TER minimization (Jang, 2023).
• Quantized RWLS: For system identification with quantized outputs, RWLS leverages probabilistic modeling to form a surrogate (linearized) WLS cost, using recursive two-stage updates to jointly estimate noise variance and regression parameters (Liu et al., 27 Feb 2025).
• Diffusion RWLS: In distributed multi-agent networks, RWLS recursions are embedded in adapt-then-combine steps, where agents locally adapt via RLS, then combine estimates across neighbors, with optional robustness constraints on update norms for impulsive noise resilience (Yu et al., 2018).

2. Regularization, Penalties, and Forgetting Schemes

RWLS can be interpreted as a sequence of regularized least squares problems. At each step, the update solves

$\{w_s\}$4

with the penalty determined by covariance-weighted quadratic form, e.g., $\{w_s\}$5 in standard RLS. For multi-forgetting-factor RWLS, the penalty is generalized by $\{w_s\}$6, enabling direction-dependent adaptation rates. This framework ensures the positive definiteness of the covariance under broad conditions.

Forgetting schemes are critical for balancing adaptation and stability:

• High forgetting ($\{w_s\}$7): Emphasizes stability and low steady-state error; poor at tracking rapid parameter changes.
• Low forgetting ($\{w_s\}$8): Rapidly adapts to changes but increases noise sensitivity.
• Multiple forgetting ($\{w_s\}$9): Allows different adaptation speeds for subsets of parameters, crucial in time-varying ARX models with both fast- and slow-varying dynamics (Fraccaroli et al., 2015).

3. Algorithmic Implementations and Complexity

RWLS admits several efficient recursive formulations:

• Classic RLS/RWLS: $\phi(s)$0 per update where $\phi(s)$1 is parameter dimension.
• Non-iteratively Reweighted RLS (NR-RLS): Maintains online exact equivalence to batch WLS using constant-memory operations and two $\phi(s)$2 matrix updates per sample, exploiting the Sherman–Morrison–Woodbury formula. The per-step complexity is $\phi(s)$3, regardless of the number of samples, enabling scalability for streaming data (Jang, 2023).

The NR-RLS procedure for class-imbalanced online binary classification is summarized as follows:

Step	Key Operation	Complexity
Class-weight update	Ratio updates	$\phi(s)$4
Inverse covariance update	Matrix inverse	$\phi(s)$5
Weighted moment update	Vector update	$\phi(s)$6
Weight estimate	Matrix-vector	$\phi(s)$7

Empirical evidence shows NR-RLS preserves exactness and efficiency, outperforming iterative reweighting (IR-RLS, $\phi(s)$8 cost per sample) and stochastic alternatives on both accuracy and speed in imbalanced tasks.

4. Robust, Distributed, and Specialized RWLS Extensions

RWLS algorithms are adapted to specialized and adversarial settings:

• Impulsive Noise Robustness: By imposing a time-varying constraint on the squared-norm of local updates (e.g., $\phi(s)$9), the diffusion RWLS can mitigate the effect of rare, large outliers in measurements. The threshold is recursively adapted and diffused among agents, ensuring resilience and collaborative tracking across the network (Yu et al., 2018).
• Reset Strategies: Detecting sudden parameter changes via local or networked statistics facilitates “forgetting” via re-initialization of the covariance and threshold, restoring quick adaptation after abrupt dynamics (Yu et al., 2018).
• Quantized Observations: RWLS enables recursive identification with quantized outputs by (i) modeling the expected value of quantized output products, (ii) estimating noise variance via ML-type recursion, and (iii) applying a standard RWLS to the linearized moment (Liu et al., 27 Feb 2025).
• Extension to Output-Error Systems: RWLS methods can be posed for output-error models with long FIR approximations, then solved via Durbin’s linear equations to recover the original system coefficients (Liu et al., 27 Feb 2025).

5. Theoretical Analysis and Convergence Guarantees

Classic and advanced variants of RWLS have provable performance:

• Exact Online-to-Batch Correspondence: NR-RLS is shown to yield online estimates precisely equal to those from full batch WLS with current weights at every time (Jang, 2023).
• Convergence Rates: For probabilistically weighted and quantized RWLS under suitable conditions (Gaussian regressors and noise), the estimator converges to the true parameter almost surely at $y(s)$0, and in $y(s)$1 at $y(s)$2; asymptotic normality is also established (Liu et al., 27 Feb 2025).
• Positive-Definiteness and Stability: Under mild conditions on mapping $y(s)$3, the covariance updates remain positive definite, ensuring algorithmic stability (Fraccaroli et al., 2015).
• Empirical Superiority: Simulation studies on time-varying ARX models demonstrate that multi-forgetting-factor RWLS outperforms scalar-forgetting RLS and other vector-forgetting schemes, especially when dynamic rates are tailored to parameter components; the TC scheme yields best tracking fit and prediction accuracy (Fraccaroli et al., 2015).
• Exact Class-weighted Classification: Statistical analysis confirms that NR-RLS recovers the Bayes-optimal classifier under cost-sensitive losses in the limit, with empirical rank-1 performance on G-mean metrics across 31 real-world data sets (Jang, 2023).

6. Applications, Limitations, and Extensions

RWLS is widely applicable to:

• Time-Varying System Identification: Adaptive control, online filtering, and signal processing where parameter drift must be tracked (Fraccaroli et al., 2015).
• Distributed Sensor Networks: Multi-agent estimation under local and network-wide adaptation constraints, tolerant to bursty noise and variable topologies (Yu et al., 2018).
• Online Imbalanced Classification: Real-time detection tasks in streaming and large-scale settings, providing both statistical optimality and computational tractability (Jang, 2023).
• Quantized Measurement Environments: Recursive identification in digital and resource-constrained systems (Liu et al., 27 Feb 2025).

Limitations depend on the specific RWLS form. The NR-RLS algorithm is restricted to linear models and binary classes, though extensions to multiclass (via one-vs-all), kernelization, and alternative losses are possible. Multi-forgetting RWLS requires tuning a vector of forgetting rates and a corresponding shrinkage map, which may be nontrivial for very high-dimensional or poorly conditioned systems.

Proposed extensions include incorporation in multiclass settings, low-rank kernel approximations for nonlinear regression, and more general loss-induced reweighting schemes (Jang, 2023).

7. Comparative Overview of Online Reweighted Least Squares

The RWLS landscape can be summarized as follows:

Method	Per-step Complexity	Weight Design	Remarks
Classic RLS	$y(s)$4	Exponential ($y(s)$5)	Single forgetting factor, no reweighting
Iteratively Reweighted RLS (IR-RLS)	$y(s)$6	Arbitrary, iterative	Exact, high cost
Non-iterative RWLS (NR-RLS)	$y(s)$7	Deterministic, exact	Exact, supports online reweighting
Multi-forgetting-factor RWLS	$y(s)$8	Vector $y(s)$9	Parameterwise adaptation, empirically superior
Stochastic first/second-order (PE/PA)	$w_s = \lambda^{t-s}$0/$w_s = \lambda^{t-s}$1	Unweighted/approx.	Lower accuracy in class imbalance
Distributed Diffusion RWLS	varies	Networked, robust	Spreads information, robust to noise

Empirical benchmarks indicate that NR-RLS and multi-forgetting RWLS methods achieve superior accuracy and adaptation at constant per-step cost, particularly in nonstationary, imbalanced, and distributed settings (Jang, 2023, Fraccaroli et al., 2015, Yu et al., 2018). The selection of weighting and forgetting schemes is problem-dependent and typically guided by properties of the underlying data streams.