---
title: 'Total Least Squares: Theory and Applications'
url: https://www.emergentmind.com/topics/total-least-squares-tls
type: topic
---

# Total Least Squares: Theory and Applications

Total Least Squares (TLS) is a framework for linear regression and inverse problems in which errors are permitted in both the matrix of explanatory variables (the regression matrix, $A$) and the observation vector ($b$). Unlike ordinary least squares (LS), which assumes the explanatory variables are measured without error, TLS jointly corrects both $A$ and $b$ to achieve a solution that is consistent within the error model, thus providing an "errors-in-variables" estimator with broad relevance in system identification, signal processing, computational biology, control, photonics, and numerous other fields [1406.0189].

## 1. Classical Formulation and Geometric Interpretation

The standard TLS problem for an overdetermined linear system $Ax \approx b$ seeks minimal joint perturbations $(E, f)$, in Frobenius norm, so that the perturbed system $(A+E)x = b+f$ has an exact solution. Mathematically, the problem is
\[
\min_{E,\,f}\:\| [E\ f] \|_F^2~~\text{subject to}~~(A+E)x = b+f.
\]
A classical result shows that the TLS solution corresponds to the right singular vector associated with the smallest singular value $\sigma_{n+1}$ of the augmented matrix $[A\ b]$. Letting $v_{n+1}$ be this singular vector, the solution can be written as
\[
x_{TLS} = - (v_{n+1}(1:n) / v_{n+1}(n+1)),
\]
provided $v_{n+1}(n+1) \neq 0$ [1110.0166].

Geometrically, TLS minimizes the orthogonal (“shortest”) distance from the observed data $(A, b)$ to the set of perfectly consistent data $\{(A', b'): A'x = b'\}$, thus generalizing LS, which measures only the vertical distance in $b$. For line fitting, this reduction leads to a quadratically constrained quadratic program (QCQP) or, after suitable variable elimination, an eigenproblem involving the data covariance matrix about the centroid [2206.05082].

## 2. Generalizations and Modern Algorithms

### Sparsity-Cognizant and Structured TLS

Classical TLS models do not exploit possible sparsity in the parameter vector $x$. The S-TLS (Sparsity-Cognizant TLS) framework introduces an $\ell_1$-norm regularization,
\[
\min_{x,E,e} \|e\|_2^2 + \lambda \|x\|_1\quad\text{s.t.}\quad y+e = (A+E)x,
\]
enabling robust sparse recovery under both $A$ and $y$ perturbations. Solution strategies include a global bisection/branch-and-bound algorithm and a highly efficient alternating descent scheme that alternates between convex Lasso-type subproblems for $x$, and closed-form updates for $E$ [1008.2996].

Structured TLS (STLS) extends the classical problem by incorporating known structure, such as Toeplitz or Vandermonde matrices for $A$, or heteroscedastic noise models. The convex relaxation approach replaces the non-convex rank constraint in TLS with a nuclear norm, leading to efficient optimization methods, particularly when combined with reweighting heuristics that closely approximate rank minimization [1406.0189].

### Randomized, Quantum, and Mixed Precision Methods

Classical algorithms for TLS rely on full SVDs, incurring $O(mn^2)$ computational cost. For large-scale or ill-posed problems, randomized algorithms (e.g., randomized subspace projection, randomized SVD) provide efficient and accurate approximations to the TLS solution, sometimes leveraging additional structure for further acceleration [1401.6832].

Quantum algorithms interpret the TLS solution as finding the ground state of a matrix Hamiltonian, offering at least polynomial speedup over classical SVD-based schemes when overlap between initial and ground state is non-negligible [1906.01194]. Quantum-inspired algorithms, using $\ell_2$ norm sampling and sketched SVD routines, mimic the quantum state preparation/sampling process to produce approximate TLS solutions in sublinear time relative to data size [2205.00455].

Mixed-precision Rayleigh Quotient Iteration (RQI-PCGTLS-MP) leverages hardware support for half and single precision to accelerate TLS solution of large, sparse systems by performing QR factorization and inner linear solves at reduced precision, provided precision bounds derived from both $A$ and $b$ are maintained [2305.19028].

## 3. Condition Numbers, Perturbation, and Error Analysis

Both normwise and refined condition numbers for TLS have been formulated to evaluate solution sensitivity to data perturbations. Notably, modern SVD-based formulas allow the condition number to be computed efficiently using only the singular values and blocks from the SVD of $[A\ b]$ [1101.2342, 1110.0166]. For example,
\[
\kappa_g(A,b) = \sqrt{1 + \|x_{TLS}\|^2}\ \| V_{11}^{-T} S \|,
\]
where $S = \text{diag}(\sqrt{\sigma_i^2+\sigma_{n+1}^2}/(\sigma_i^2-\sigma_{n+1}^2))$.

Componentwise and mixed condition numbers, reflecting data sparsity and scaling, produce more realistic perturbation estimates compared to traditional normwise bounds and reveal that structure-aware analysis yields sharper results [1612.08337, 2004.12082]. Perturbation theory shows the equivalence, at leading order, of several condition number notions previously proposed [1401.6832]. Statistical condition estimation and directional-derivative methods enable a posteriori estimation of these quantities in practical solvers [2004.12082].

A variety of studies confirm that structured condition numbers can be significantly smaller than unstructured ones, particularly when $A$ exhibits known structure, justifying the use of structure-preserving algorithms for TLS [1612.08337, 2004.12082].

## 4. Practical Applications

TLS and its variants have been adopted across technical domains where measurements are contaminated in both predictors and responses:

- **Compressive Sampling/Dictionary Learning**: S-TLS enables robust sparse recovery when both measurement data and the dictionary may be mismatched, outperforming standard Lasso and BP in cognitive radio and direction-of-arrival (DoA) estimation under off-grid and uncalibrated scenarios [1008.2996].
- **Biological Inference**: Structured TLS with nuclear norm relaxations has proved effective in inferring cell-type-specific expression profiles from population averages, recovering mixing fractions accurately even when error structure is complex [1406.0189].
- **Robust Line Fitting and Computer Vision**: The TLS framework for line fitting as a QCQP or SDP supports robustification (against outliers using M-estimation) and optimality certification via Black–Rangarajan duality and Lagrangian methods, uniting classical and modern optimization perspectives [2206.05082].
- **Data-driven Control Design**: Constrained TLS (CTLS) embedded in virtual reference feedback tuning eliminates LS bias in ARX-type controller identification from noisy data, yielding estimates with both low bias and low mean-squared error compared to instrumental variable methods [2009.06787].
- **Phase Retrieval with Operator Noise**: TLS phase retrieval methods account for explicit sensing vector errors, often outperforming LS methods in real optical experiments when the main error is in the measurement operator [2102.00927].
- **Permutation and Matching Problems**: Shuffled TLS enables permutation recovery and robust Procrustes alignment of noisy correspondences, with theoretical upper bounds and algorithmic frameworks based on alternating linear assignment and TLS steps [2209.01066].
- **Tensor-structured Inverse Problems**: The tensor regularized TLS (TR-TLS) generalizes TLS and Tikhonov regularization to higher-order data, yielding efficient solutions for image and video deblurring problems via tensor algebra and the T-product [2211.00217].
- **Biquaternionic and Complex Systems**: The RBMTLS framework solves systems in reduced biquaternion algebra and their complex specializations, providing explicit error-resilient solutions for mixed error distributions across matrix blocks [2311.06464].

## 5. Limitations and Insights from Reduced-Rank and Robust Extensions

Reduced-rank methods, successful in LS settings for variance–bias tradeoffs, do not generally extend to TLS because in the TLS noise model the unknown parameter appears in both mean and variance of the observations. This coupling prevents the direct use of classical bias–variance ordering or order-selection rules, except in restricted or norm-constrained parameter cases. This result highlights intrinsic limitations of transferring low-rank techniques from LS to the TLS noise model [1903.01745].

Robust extensions (e.g., Geman–McClure M-estimation) and convex relaxations using reweighted nuclear norms have been demonstrated to achieve near-optimal solutions in structured/noisy environments where traditional SVD-based or nonconvex lifting strategies may fail or be suboptimal [1406.0189, 2206.05082]. SDP relaxations, while globally reliable, can be computationally slow for large data, motivating the use of IRLS paired with optimality certification strategies.

## 6. Computational Efficiency, Scaling, and Algorithmic Trade-Offs

For high-dimensional or large-scale TLS problems, randomized, quantum-inspired, and sketching-based algorithms provide computational scalability:

- Randomized SVDs, subspace projection, and sketching enable solutions in time linear in number of nonzeros plus a poly$(n/\epsilon)$ term [1401.6832, 1909.12441, 2205.00455].
- These methods rely on leverage score sampling, CountSketch, and iterative refinement to produce solutions within $(1+\epsilon)$ of optimal, without explicit formation of large dense representations.
- Accuracy bounds rely on preservation of SVD structure by random projections and appropriate gap between dominant and non-dominant singular values.
- Mixed precision iterative methods must account for both $A$ and $b$ in spectral gap-based bounds to ensure numerical stability and avoid loss of attainable accuracy [2305.19028].

Algorithmic trade-offs between global optimality (SVD-based, bisection/branch-and-bound), efficiency (alternating descent, randomized sketching), and practical reliability (statistical estimation of conditioning, structured perturbations) are dictated by problem size, data structure, and rigidity of error modeling.

## 7. Future Directions

Ongoing research in TLS explores deeper integration of convex relaxations, structure-exploiting randomization, quantum and quantum-inspired acceleration, robust and componentwise error analysis, and high-order tensor extensions. Automatic error and condition estimation via statistical sampling is increasingly integrated into numerical solvers for reliable error bounds in the presence of sparsity, scaling, and structure [1612.08337, 2004.12082]. Advanced applications continue to drive development in computational efficiency, robustness, and the unification of operator and measurement error models.

Source: https://www.emergentmind.com/topics/total-least-squares-tls