Linear Least Squares Estimator (LLSE)

• LLSE is a statistical method that minimizes the sum of squared residuals to estimate unknown parameters in linear models.
• It provides unbiased, minimum-variance estimates under standard noise assumptions, making it a cornerstone of regression, signal processing, and system identification.
• LLSE techniques extend to weighted, recursive, and large-scale settings, ensuring robust performance in dynamic and non-i.i.d. error environments.

The linear least squares estimator (LLSE) is the canonical solution to the problem of estimating unknown parameters in linear models by minimizing the sum of squared residuals. LLSEs are fundamental throughout statistical inference, signal processing, and system identification, providing unbiased, minimum-variance linear estimators under mild regularity and noise assumptions. Their algebraic structure enables exact analysis for general noise laws, extensions to large-scale and streaming settings, and robust theoretical guarantees under non-i.i.d. and temporally dependent sampling.

1. Model Formulation and Estimation

The LLSE addresses the problem

$y = Hx + v,$

where $y \in \mathbb{R}^m$ is the observation vector, $H \in \mathbb{R}^{m \times n}$ is a known design (regression) matrix, $x \in \mathbb{R}^n$ is the unknown parameter vector, and $v$ is a disturbance (noise) vector. The LLSE seeks $x$ minimizing

$J(x) = \|y-Hx\|_2^2.$

The minimizer, assuming $H^{\mathsf T}H$ invertible, is obtained via the normal equations:

$\hat{x} = (H^{\mathsf T}H)^{-1}H^{\mathsf T}y.$

For possibly rank-deficient $H$, the Moore–Penrose pseudoinverse, $y \in \mathbb{R}^m$0, yields the unique minimum Euclidean-norm solution (Zhang, 2022).

In weighted settings with noise covariance $y \in \mathbb{R}^m$1, the LLSE generalizes to

$y \in \mathbb{R}^m$2

For linear regression, the estimator

$y \in \mathbb{R}^m$3

gives

$y \in \mathbb{R}^m$4

with standard extensions to multivariate or time-series cases (Bertin et al., 2021, Zhang, 2022).

2. Finite-Sample Distribution and Moments

When the disturbances $y \in \mathbb{R}^m$5 are i.i.d. with mean zero and variance $y \in \mathbb{R}^m$6, the LLSE $y \in \mathbb{R}^m$7 is unbiased ($y \in \mathbb{R}^m$8), and its error covariance is

$y \in \mathbb{R}^m$9

If $H \in \mathbb{R}^{m \times n}$0, this reduces to

$H \in \mathbb{R}^{m \times n}$1

The Gauss–Markov theorem asserts that, among all unbiased linear estimators, the LLSE attains the minimal variance in every direction (Zhang, 2022, Bertin et al., 2021). For non-Gaussian, e.g. uniform noise $H \in \mathbb{R}^{m \times n}$2, the law of the estimator is not Gaussian but can be explicitly characterized via generalized Irwin–Hall convolutions (Jlibene et al., 2021). Unbiasedness and closed-form variances still hold: $H \in \mathbb{R}^{m \times n}$3 with $H \in \mathbb{R}^{m \times n}$4, $H \in \mathbb{R}^{m \times n}$5, $H \in \mathbb{R}^{m \times n}$6.

3. Asymptotic and Robustness Properties

In the classical i.i.d. context, if the design satisfies $H \in \mathbb{R}^{m \times n}$7 is positive definite, the LLSE is strongly consistent and asymptotically normal: $H \in \mathbb{R}^{m \times n}$8 For simple linear regression with uniform noise, with mild design regularity,

$H \in \mathbb{R}^{m \times n}$9

and similar for the intercept (Jlibene et al., 2021).

For models where errors are temporally dependent but strictly stationary, under Hannan-type short-range dependence, the normalized LLSE admits a central limit theorem with an explicitly estimable asymptotic variance involving the spectral measure of the design and the error spectral density (Caron, 2018). Dependent-robust versions of classical $x \in \mathbb{R}^n$0- and $x \in \mathbb{R}^n$1-tests, built from consistent covariance estimators using tapered or kernel-smoothed residual autocovariances, correct for size inflation observed under naively ignoring dependence.

In random and dependent observation patterns, e.g., negatively superadditive dependent random sampling, the LLSE remains strongly consistent under mild moment and dependence assumptions, with rate of convergence $x \in \mathbb{R}^n$2, and yields nondegenerate almost-sure distributional limits that can be made explicit (Bertin et al., 2021).

4. Alternative Characterizations and Interpretations

The LLSE admits geometric, probabilistic, and algebraic interpretations. For equally spaced regressors, the LLSE slope can be obtained as the coefficient which annuls the net signed area under the residual cumulative sum ("data walk"), establishing a direct equivalence between LLSE and area-cancelling random walk slopes (Kostinski et al., 26 Mar 2025). This perspective applies to arbitrary noise distributions and connects LLSE to the theory of random bridges and Brownian motion.

Additionally, LLSE emerges as the maximum likelihood estimator under i.i.d. Gaussian noise but retains optimality (BLUE) properties with merely zero-mean, homoscedastic, uncorrelated noise. Weighted variants, regularization (e.g., Tikhonov), and recursive versions connect LLSE to Kalman filtering and online learning (Zhang, 2022).

5. Computational Strategies and Large-Scale Regimes

For problems where $x \in \mathbb{R}^n$3 (many more samples than parameters), the batch LLSE requires $x \in \mathbb{R}^n$4 operations (forming $x \in \mathbb{R}^n$5), and matrix inversion, which may become computationally intensive. Iterative algorithms, such as steepest descent and sequential least squares, offer reductions in per-iteration computational cost but may require repeated passes over the data.

Approximate Least Squares (ALS) iteratively updates an estimate using only one row $x \in \mathbb{R}^n$6 at a time, reducing per-iteration complexity to $x \in \mathbb{R}^n$7. In the noise-free case, ALS provably converges to the standard LLSE if the step size $x \in \mathbb{R}^n$8 (Lunglmayr et al., 2013). In presence of noise, averaging over $x \in \mathbb{R}^n$9 iterates delivers nearly optimal error (within 10–15% of the batch LLSE variance), while allowing for orders-of-magnitude reductions in total arithmetic operations for large $v$0.

Recursive least squares (RLS) and Kalman-gain-based updates allow efficient online estimation and are widely used in real-time, streaming, or adaptive filtering applications (Zhang, 2022). Regularization strategies, such as adding $v$1 to $v$2, are standard for ill-conditioned or high-dimensional problems.

6. Specialized Applications and Extensions

The LLSE framework extends to autoregressive system identification (e.g., estimation of stable LTI system parameters) and nonlinear parameter estimation via Gauss–Newton or Levenberg–Marquardt algorithms, with the linearized subproblems at each iteration solved via LLSE (Davis et al., 2019, Zhang, 2022). In finite sample identification of stable linear dynamical systems, the ordinary least squares estimator achieves minimax-optimal error rates up to universal constants, with precise non-asymptotic confidence bounds derived via novel concentration of the data Gramian (Jedra et al., 2020).

Application domains include frequency estimation in signal processing, efficient $v$3 direct LLSE-based batch solutions for tone estimation (matching nonlinear LS performance at far lower computational cost), and robust time series modeling (Davis et al., 2019, Caron, 2018).

7. Summary of Key Theoretical and Practical Guarantees

Property	Main Results	Reference
Unbiasedness	$v$4	(Zhang, 2022, Bertin et al., 2021, Jlibene et al., 2021)
Minimum-Variance (Gauss–Markov)	LLSE is BLUE among linear unbiased estimators	(Zhang, 2022, Bertin et al., 2021)
Finite-sample explicit distribution	Exact for uniform errors via Irwin–Hall density	(Jlibene et al., 2021)
Asymptotic normality	CLT under i.i.d. and certain dependent settings	(Jlibene et al., 2021, Caron, 2018)
Strong consistency w/ random or NSD sampling	$v$5 convergence rate, explicit limit	(Bertin et al., 2021)
Computation/large-scale regimes	ALS: $v$6, RLS, batch	(Lunglmayr et al., 2013, Zhang, 2022)
Robustness to error dependence	Dependent-robust inference via tapered covariance	(Caron, 2018)

LLSE provides a versatile, tractable, and theoretically grounded tool for linear statistical inference. It remains a central object in contemporary research, notably in analysis of random walk characterizations, sample-complexity optimality in dynamical systems, and robust inference under temporal dependence and irregular observation patterns (Kostinski et al., 26 Mar 2025, Jedra et al., 2020, Bertin et al., 2021, Caron, 2018, Zhang, 2022).