Analytical Least-Squares Method Overview

• Analytical least-squares method is a collection of techniques that minimize the sum of squared deviations to estimate unknown parameters with rigorous statistical and numerical foundations.
• It incorporates advanced formulations including generalized least squares, iterative block methods, and meshless PDE discretizations to enhance convergence, stability, and accuracy.
• The method is applied across various fields from signal processing to inverse problems, enabling robust error estimation and adaptive tuning in complex, large-scale data environments.

The analytical least-squares method encompasses a family of mathematical approaches in which unknown parameters of models—or roots, coefficients, or functionals—are determined by minimizing the sum of squared deviations between model predictions and observed or sampled values. It is foundational across statistics, numerical analysis, inverse problems, computational physics, signal processing, and dynamical system sensitivity analysis. Recent developments extend the reach of the least-squares principle far beyond the historical linear regression and quadratic minimization context, introducing novel iterative schemes, robust estimation strategies, graph-based decompositions, meshless discretizations for PDEs, and direct analogies to probabilistic processes such as random walks.

1. Fundamental Formulations and Theoretical Frameworks

Analytical least-squares estimation is framed as the minimization of a quadratic cost function—usually of the form $\|m - Jx\|_P^2$ where $m$ is the measurement or data vector, $J$ the design matrix or system operator, $x$ unknown parameters, and $P$ a weighting matrix (often related to the inverse of the noise covariance). The canonical solution is

$x^* = (J^T P J)^{-1} J^T P m$

with associated estimator covariance

$V_x^* = (J^T P J)^{-1} J^T P Q P J (J^T P J)^{-1}$

where $Q$ is the noise covariance (1305.6324). The "generalized least squares" (GLS) case ($P=Q^{-1}$) yields the minimum variance (BLUE) estimator and is central in applications where noise characteristics are heterogeneous or correlated.

Rigorous theoretical treatments address issues such as:

• Transitivity, ensuring sequential applications of the estimator yield results equivalent to a single global optimization—a property guaranteed in GLS but not in the naive ordinary least squares case (1305.6324).
• Numerical Stability, managed through alternative matrix factorizations (e.g., Gram–Schmidt) or rescaling, and highlighted in settings where the normal equations are ill-conditioned or data magnitude ranges span multiple orders (Skala, 2018).

Least-squares procedures naturally align with optimal filtering concepts; the solution to a GLS problem for a signal-in-noise model is shown to be mathematically equivalent to convolution with a frequency-domain matched filter under suitable assumptions (1305.6324). In the Fourier domain, the estimator and its variance become spectral integrals weighted by the noise power spectrum.

2. Iterative and Block Methods for Large-Scale and Inconsistent Systems

Classical direct least‐squares solvers are computationally impractical in large-scale or streaming scenarios. Modern approaches exploit randomized and block iterative methods:

• Randomized Block Kaczmarz with Projection (Needell et al., 2014): Extends the standard Kaczmarz iterative projection onto solution spaces of individual rows (or blocks) to inconsistent (overdetermined, noisy) systems. The introduction of projection steps onto column/row blocks—with updates such as $$z_{k} = z_{k-1} - A_{\tau_k} (A_{\tau_k})^\dagger z_{k-1}$$—and matrix paving ensures geometric (linear) convergence even in the inconsistent case, overcoming the stagnation plateau of traditional row-based methods in the presence of noise.
• Progressive Iterative Approximation and Chebyshev Semi-Iterative Acceleration (Wu et al., 2022): Adaptive step-sizes derived from Chebyshev polynomials are employed to accelerate classical least-squares progressive iterative approximation (LSPIA) schemes for curve/surface fitting, providing near-optimal convergence rates comparable to the conjugate gradient method, and maintaining robustness even when the system matrix is rank-deficient.
• Flexible Krylov Subspace Methods (FMLSMR) (Yang et al., 29 Aug 2024): Iterative solvers for least squares are refined to require only a single linear solve per iteration (via a merged preconditioning strategy), and integrated with flexible preconditioning as in GMRES. This reduces both memory and computational cost relative to earlier implementations (e.g., LSMR, FLSMR), and is effective for large, ill-conditioned systems.

3. Analytical Least Squares in Differential Equations and Inverse Problems

The least-squares methodology has been extended from data/model fitting to the solution of differential, partial differential, and inverse problems with a variety of novel formulations:

• Boundary/Initial Value Problems via Constrained Expression Expansion (Mortari, 2017): The solution to a linear ODE or PDE is expressed as

$$y(t) = g(t) + \sum_i \beta_i(t, t_k)\left[y^{(d_i)}_{t_i} - g^{(d_i)}_{t_i}\right]$$

embedding all constraints exactly via specialized functions $\beta_i$; $g(t)$ is expanded in an appropriate basis (e.g., Chebyshev polynomials), and the coefficients are determined via least squares. This produces uniformly distributed residual error and provides a mechanism for detecting ill-posed or non-unique problems.

• Meshless PDE Discretization with Constrained Least Squares (Ying et al., 9 Jul 2024): The Constrained Least-Squares Ghost Sample Points (CLS-GSP) method introduces auxiliary (ghost) points into the local approximation, and imposes exact constraints ensuring properties such as sum-to-zero of Laplacian weights. This yields a differential matrix with improved diagonal dominance and conditioning, guarantees discrete consistency with the continuous operator, and provides superior eigenvalue and solution accuracy for irregular domains.
• Least-Squares Finite Element Methods (LSFEM) (Bertrand, 2018, Bertrand et al., 2023, Chaudhry et al., 2020): The PDE system is recast in a first-order residual form (e.g., simultaneously in terms of velocity and stress), and the sum of the squared residuals is minimized in a variational setting. This approach ensures coercivity and stability in the norm induced by the sum of the squared residuals, obviates the need for inf-sup conditions, and enables equivalent-order approximation spaces for all variables. Rigorous, computable error estimators facilitate offline-online decompositions in reduced basis settings.
• Least-Squares Shadowing (LSS) for Chaotic Sensitivity Analysis (Chater et al., 2015): For ergodic dynamical systems, the derivative of a long-time average is computed by solving a constrained least-squares problem: minimizing

$\int_{0}^{T} \|v(t)\|^2 + \alpha \eta^2(t)\,dt$

subject to a linearized state constraint encoding the shadowing direction and time-dilation parameter. This methodology provides stable statistical sensitivity analysis in chaotic regimes where tangent/adjoint methods fail.

• Inverse Medium Reconstruction with Mixed Regularization (Ito et al., 2022): A two-stage procedure combines direct sampling (DSM) to localize inclusions, followed by a total least-squares formulation with mixed $H^1$ (smoothness) and $L^1$ (sparsity/prominence of sharp interfaces) regularization, imposed as

$$\psi(q) = \int_\Omega \left(\frac{\alpha}{2} (|\nabla q|^2 + q^2) + \beta |q| \right)\,dx + \chi(q;q_0,q_1)$$

Coupled well-posedness, convergence, and robustness against noise are established analytically and verified in numerical reconstructions.

4. Nonstandard and Generalized Least-Squares Principles

Beyond ordinary least squares, several generalizations and reinterpretations have emerged:

• Root Finding via Analytical Least-Squares Fitting (1302.1966): For a nonlinear function $y=f(x)$, root finding is reformulated as successive fitting of a parametric polynomial $y=a(x-b)^N$ through three equidistant points. Minimizing the sum of squared deviations—without derivatives—yields an iterative process for updating the root estimate $b$; $N$ can be dynamically adapted for fast (quadratic) convergence and to avoid divergence or oscillation.
• Graph-Theoretic Least-Squares Ranking (Csató, 2015): The least squares ranking solution for incomplete, weighted paired comparison tournaments is interpreted as a fixed point of an iterative “Neumann series” process on a multi-graph with possibly attached loops, providing a clear relation to Markov chains and graph balancing. Refinements and modifications are achieved by varying loop weights, thus altering convergence properties and sensitivity to comparison structure.
• Random Walk Interpretation of LLS (Kostinski et al., 26 Mar 2025): For uniformly sampled data, the least squares slope is shown to coincide with the unique value that annuls the net area under the cumulative-sum “data walk” (bridge) of mean-centered observations. The explicit formula

$\alpha = -\frac{12}{N(N+1)} \sum_{j=1}^N z_j$

with $z_j = \sum_{k=1}^j (y_k - \overline{y})$ provides a probabilistic and geometric perspective on linear regression, exact for arbitrary noise when the sampling is equispaced.

• Variations on Least Squares for Regression (Talvila, 2020): Analytically distinct “vertical,” “horizontal,” and “perpendicular” least-squares regression lines are derived and compared. The perpendicular method, minimizing squared orthogonal distances to the line, is provably rotation invariant and yields slopes satisfying

$|m| \leq |\tan\theta| \leq |m_X|$

where $m$ is the vertical method slope, $m_X$ the reciprocal of the horizontal method slope, and $\theta$ the angle in the perpendicular minimization.

• Parameter Auto-Tuning in Data Fitting (Barratt et al., 2019): The least squares problem is itself parameterized (in regularization, weighting, or feature-transformation hyperparameters), and an outer optimization loop adjusts these parameters to minimize an external, application-specific target objective (e.g., validation loss), leveraging proximal gradient methods and automatic differentiation for efficient tuning—especially in overparameterized or overregularized data regimes.

5. Numerical Conditioning, Basis Choice, and Implementation Strategies

Numerical conditioning is critical in large-scale or highly scaled data problems:

• Conditioning via Basis Scaling (Skala, 2018): Replacing orthonormal with scaled orthogonal basis vectors (with scaling factors $q_j$ set to match the maximum column magnitudes) dramatically reduces the condition number of the normal equations, particularly for problems where variable ranges differ by several orders of magnitude. The geometric meaning and solution are preserved, but numerical stability is greatly improved—this applies equally to polynomial regression and radial basis function (RBF) approximation.
• Frame/Object-Oriented Automation in Least Squares (Agbachi, 2018): Automated systems based on frame or object-oriented paradigms provide one-step pipelines for the entire LS estimation process in applications such as geomatics. Tasks including coefficient matrix construction, weighting, cycle analysis, and solution update are encapsulated in modular “frames,” streamlining both dynamic data environments and error/cycle consistency checking.

6. Consistency, Error Estimation, and Rigorous Convergence Analysis

• Consistency and Diagonal Dominance in Meshless Discretizations (Ying et al., 9 Jul 2024): The inclusion of hard constraints in the CLS-GSP method ensures the discrete Laplacian annihilates constants exactly; analytical error bounds demonstrate convergence to the true operator as point cloud resolution increases.
• Error Bounds in Statistical Estimation (Lobos et al., 2015, Chaudhry et al., 2020): Tight analytic bounds for bias and mean-square error of the LS estimator are established via higher-order Taylor expansion, with explicit expressions incorporating a relative curvature residual $\delta$ to define sharp confidence intervals both for small and large signal-to-noise regimes.
• A Posteriori Error Estimators in Reduced Basis LS-FEM (Chaudhry et al., 2020): The norm of the error can be bounded above in terms of the solution to an enriched-space LS problem and a computable residual, leading to robust, certified model reduction for parameterized PDEs.

7. Applications and Impact Across Domains

The analytical least-squares method and its extensions have significant and diverse impacts:

• Signal Processing: GLS is fundamental to matched filtering; in spectral estimation, optimal weighting via noise PSD is required for minimum variance (1305.6324).
• Ranking and Network Analysis: Iterative LS ranking methods provide performance-sensitive, interpretable solutions in incomplete or asymmetrical competition graphs (Csató, 2015).
• PDE and Inverse Problems: LS-based finite element and meshless discretizations enable robust, monolithic treatment of complex, multiphysics systems, including nonlinear sea-ice dynamics (Bertrand, 2018, Bertrand et al., 2023), inverse scattering, and tomography (Ito et al., 2022).
• Dynamical Systems/Chaos: LSS addresses inherently unstable sensitivity questions in high-dimensional ergodic systems (Chater et al., 2015).
• Computational Statistics and Machine Learning: Algorithmic advances in iterative and block LS solvers, coupled with automatic hyperparameter tuning, render least-squares estimators relevant and competitive in large-scale, high-noise, or non-Gaussian data scenarios (Barratt et al., 2019, Wu et al., 2022).

These developments collectively demonstrate the continued evolution and centrality of analytical least-squares methodology in computational mathematics, with ongoing innovation at multiple interrelated mathematical, algorithmic, and application frontiers.