Total Least Squares: Theory and Applications

• Total Least Squares (TLS) is a regression method that permits errors in both the predictor matrix and response vector, yielding accurate results for noisy data.
• TLS computes solutions by minimizing the overall data perturbations using the smallest singular value of the augmented matrix, enhancing the classic LS approach.
• Modern TLS techniques integrate sparsity, structured models, and scalable methods such as randomized and quantum-inspired algorithms for efficient large-scale analysis.

Total Least Squares (TLS) is a framework for linear regression and inverse problems in which errors are permitted in both the matrix of explanatory variables (the regression matrix, $A$) and the observation vector ($b$). Unlike ordinary least squares (LS), which assumes the explanatory variables are measured without error, TLS jointly corrects both $A$ and $b$ to achieve a solution that is consistent within the error model, thus providing an "errors-in-variables" estimator with broad relevance in system identification, signal processing, computational biology, control, photonics, and numerous other fields (Malioutov et al., 2014).

1. Classical Formulation and Geometric Interpretation

The standard TLS problem for an overdetermined linear system $Ax \approx b$ seeks minimal joint perturbations $(E, f)$, in Frobenius norm, so that the perturbed system $(A+E)x = b+f$ has an exact solution. Mathematically, the problem is

$$\min_{E,\,f}\:\| [E\ f] \|_F^2~~\text{subject to}~~(A+E)x = b+f.$$

A classical result shows that the TLS solution corresponds to the right singular vector associated with the smallest singular value $\sigma_{n+1}$ of the augmented matrix $[A\ b]$. Letting $v_{n+1}$ be this singular vector, the solution can be written as

$x_{TLS} = - (v_{n+1}(1:n) / v_{n+1}(n+1)),$

provided $v_{n+1}(n+1) \neq 0$ (Jia et al., 2011).

Geometrically, TLS minimizes the orthogonal (“shortest”) distance from the observed data $(A, b)$ to the set of perfectly consistent data $\{(A', b'): A'x = b'\}$, thus generalizing LS, which measures only the vertical distance in $b$. For line fitting, this reduction leads to a quadratically constrained quadratic program (QCQP) or, after suitable variable elimination, an eigenproblem involving the data covariance matrix about the centroid (Barfoot et al., 2022).

2. Generalizations and Modern Algorithms

Sparsity-Cognizant and Structured TLS

Classical TLS models do not exploit possible sparsity in the parameter vector $x$. The S-TLS (Sparsity-Cognizant TLS) framework introduces an $\ell_1$-norm regularization,

$$\min_{x,E,e} \|e\|_2^2 + \lambda \|x\|_1\quad\text{s.t.}\quad y+e = (A+E)x,$$

enabling robust sparse recovery under both $A$ and $y$ perturbations. Solution strategies include a global bisection/branch-and-bound algorithm and a highly efficient alternating descent scheme that alternates between convex Lasso-type subproblems for $x$, and closed-form updates for $E$ (Zhu et al., 2010).

Structured TLS (STLS) extends the classical problem by incorporating known structure, such as Toeplitz or Vandermonde matrices for $A$, or heteroscedastic noise models. The convex relaxation approach replaces the non-convex rank constraint in TLS with a nuclear norm, leading to efficient optimization methods, particularly when combined with reweighting heuristics that closely approximate rank minimization (Malioutov et al., 2014).

Randomized, Quantum, and Mixed Precision Methods

Classical algorithms for TLS rely on full SVDs, incurring $O(mn^2)$ computational cost. For large-scale or ill-posed problems, randomized algorithms (e.g., randomized subspace projection, randomized SVD) provide efficient and accurate approximations to the TLS solution, sometimes leveraging additional structure for further acceleration (Xie et al., 2014).

Quantum algorithms interpret the TLS solution as finding the ground state of a matrix Hamiltonian, offering at least polynomial speedup over classical SVD-based schemes when overlap between initial and ground state is non-negligible (Wang et al., 2019). Quantum-inspired algorithms, using $\ell_2$ norm sampling and sketched SVD routines, mimic the quantum state preparation/sampling process to produce approximate TLS solutions in sublinear time relative to data size (Zuo et al., 2022).

Mixed-precision Rayleigh Quotient Iteration (RQI-PCGTLS-MP) leverages hardware support for half and single precision to accelerate TLS solution of large, sparse systems by performing QR factorization and inner linear solves at reduced precision, provided precision bounds derived from both $A$ and $b$ are maintained (Oktay et al., 2023).

3. Condition Numbers, Perturbation, and Error Analysis

Both normwise and refined condition numbers for TLS have been formulated to evaluate solution sensitivity to data perturbations. Notably, modern SVD-based formulas allow the condition number to be computed efficiently using only the singular values and blocks from the SVD of $[A\ b]$ (Jia et al., 2011, Jia et al., 2011). For example,

$$\kappa_g(A,b) = \sqrt{1 + \|x_{TLS}\|^2}\ \| V_{11}^{-T} S \|,$$

where $$S = \text{diag}(\sqrt{\sigma_i^2+\sigma_{n+1}^2}/(\sigma_i^2-\sigma_{n+1}^2))$$.

Componentwise and mixed condition numbers, reflecting data sparsity and scaling, produce more realistic perturbation estimates compared to traditional normwise bounds and reveal that structure-aware analysis yields sharper results (Diao et al., 2016, Meng et al., 2020). Perturbation theory shows the equivalence, at leading order, of several condition number notions previously proposed (Xie et al., 2014). Statistical condition estimation and directional-derivative methods enable a posteriori estimation of these quantities in practical solvers (Meng et al., 2020).

A variety of studies confirm that structured condition numbers can be significantly smaller than unstructured ones, particularly when $A$ exhibits known structure, justifying the use of structure-preserving algorithms for TLS (Diao et al., 2016, Meng et al., 2020).

4. Practical Applications

TLS and its variants have been adopted across technical domains where measurements are contaminated in both predictors and responses:

• Compressive Sampling/Dictionary Learning: S-TLS enables robust sparse recovery when both measurement data and the dictionary may be mismatched, outperforming standard Lasso and BP in cognitive radio and direction-of-arrival (DoA) estimation under off-grid and uncalibrated scenarios (Zhu et al., 2010).
• Biological Inference: Structured TLS with nuclear norm relaxations has proved effective in inferring cell-type-specific expression profiles from population averages, recovering mixing fractions accurately even when error structure is complex (Malioutov et al., 2014).
• Robust Line Fitting and Computer Vision: The TLS framework for line fitting as a QCQP or SDP supports robustification (against outliers using M-estimation) and optimality certification via Black–Rangarajan duality and Lagrangian methods, uniting classical and modern optimization perspectives (Barfoot et al., 2022).
• Data-driven Control Design: Constrained TLS (CTLS) embedded in virtual reference feedback tuning eliminates LS bias in ARX-type controller identification from noisy data, yielding estimates with both low bias and low mean-squared error compared to instrumental variable methods (Garcia et al., 2020).
• Phase Retrieval with Operator Noise: TLS phase retrieval methods account for explicit sensing vector errors, often outperforming LS methods in real optical experiments when the main error is in the measurement operator (Gupta et al., 2021).
• Permutation and Matching Problems: Shuffled TLS enables permutation recovery and robust Procrustes alignment of noisy correspondences, with theoretical upper bounds and algorithmic frameworks based on alternating linear assignment and TLS steps (Wang et al., 2022).
• Tensor-structured Inverse Problems: The tensor regularized TLS (TR-TLS) generalizes TLS and Tikhonov regularization to higher-order data, yielding efficient solutions for image and video deblurring problems via tensor algebra and the T-product (Han et al., 2022).
• Biquaternionic and Complex Systems: The RBMTLS framework solves systems in reduced biquaternion algebra and their complex specializations, providing explicit error-resilient solutions for mixed error distributions across matrix blocks (Ahmad et al., 2023).

5. Limitations and Insights from Reduced-Rank and Robust Extensions

Reduced-rank methods, successful in LS settings for variance–bias tradeoffs, do not generally extend to TLS because in the TLS noise model the unknown parameter appears in both mean and variance of the observations. This coupling prevents the direct use of classical bias–variance ordering or order-selection rules, except in restricted or norm-constrained parameter cases. This result highlights intrinsic limitations of transferring low-rank techniques from LS to the TLS noise model (Nagananda et al., 2019).

Robust extensions (e.g., Geman–McClure M-estimation) and convex relaxations using reweighted nuclear norms have been demonstrated to achieve near-optimal solutions in structured/noisy environments where traditional SVD-based or nonconvex lifting strategies may fail or be suboptimal (Malioutov et al., 2014, Barfoot et al., 2022). SDP relaxations, while globally reliable, can be computationally slow for large data, motivating the use of IRLS paired with optimality certification strategies.

6. Computational Efficiency, Scaling, and Algorithmic Trade-Offs

For high-dimensional or large-scale TLS problems, randomized, quantum-inspired, and sketching-based algorithms provide computational scalability:

• Randomized SVDs, subspace projection, and sketching enable solutions in time linear in number of nonzeros plus a poly$(n/\epsilon)$ term (Xie et al., 2014, Diao et al., 2019, Zuo et al., 2022).
• These methods rely on leverage score sampling, CountSketch, and iterative refinement to produce solutions within $(1+\epsilon)$ of optimal, without explicit formation of large dense representations.
• Accuracy bounds rely on preservation of SVD structure by random projections and appropriate gap between dominant and non-dominant singular values.
• Mixed precision iterative methods must account for both $A$ and $b$ in spectral gap-based bounds to ensure numerical stability and avoid loss of attainable accuracy (Oktay et al., 2023).

Algorithmic trade-offs between global optimality (SVD-based, bisection/branch-and-bound), efficiency (alternating descent, randomized sketching), and practical reliability (statistical estimation of conditioning, structured perturbations) are dictated by problem size, data structure, and rigidity of error modeling.

7. Future Directions

Ongoing research in TLS explores deeper integration of convex relaxations, structure-exploiting randomization, quantum and quantum-inspired acceleration, robust and componentwise error analysis, and high-order tensor extensions. Automatic error and condition estimation via statistical sampling is increasingly integrated into numerical solvers for reliable error bounds in the presence of sparsity, scaling, and structure (Diao et al., 2016, Meng et al., 2020). Advanced applications continue to drive development in computational efficiency, robustness, and the unification of operator and measurement error models.