---
title: Truncated Least Squares (TLS)
url: https://www.emergentmind.com/topics/truncated-least-squares-tls
type: topic
---

# Truncated Least Squares (TLS)

Truncated Least Squares (TLS)

Truncated Least Squares (TLS) is an umbrella term encompassing several robust and regularized variants of the classical least squares and total least squares (TLS) frameworks, whose principal aim is to improve estimation accuracy or robustness by selectively ignoring, truncating, or thresholding the influence of large residuals or small singular value modes. These approaches address high-sensitivity to ill-conditioning, outliers, or errors-in-variables, and arise in numerous data-fitting, signal processing, and geometric registration contexts. The two most prominent lines within the "truncated least squares" category are truncated total least squares (TTLS)—which applies hard singular value truncation to the augmented TLS system—and robust TLS estimators that employ per-residual truncation (e.g., via the min function), often leading to combinatorial or non-smooth optimization problems.

## 1. Classical and Truncated Total Least Squares Formulation

Consider a linear regression system $A x \approx b$, where both $A \in \mathbb{R}^{m \times n}$ and $b \in \mathbb{R}^m$ are subject to errors. The total least squares (TLS) approach jointly estimates corrections $E$ and $f$ to $A$ and $b$ to minimize the Frobenius norm of the combined perturbation, subject to $(A+E)x = b + f$. This is equivalent to seeking the minimal perturbation $[E\; f]$ such that the augmented matrix $C=[A\; b]$ drops rank. The standard TLS solution (under a genericity condition on singular values) is given by
\[
x_{\mathrm{TLS}} = -\frac{v_{1:n,n+1}}{v_{n+1,n+1}}
\]
where $v_{:,n+1}$ is the right singular vector of $C$ corresponding to its smallest singular value [1401.6832, 2205.00455, 2004.12082].

Classical TLS is highly sensitive in discrete ill-posed problems where the smallest singular values of $C$ accumulate near zero. To regularize, Truncated TLS (TTLS) restricts the solution to the leading $k$ singular value directions. The TTLS estimate is obtained by forming the truncated SVD
\[
C_k = \sum_{i=1}^{k} \sigma_i u_i v_i^\top
\]
and, after appropriate partitioning of the right singular vectors $V$,
\[
x_k = (V_{11}^\top)^\dagger v_{21}
\]
where $V_{11} \in \mathbb{R}^{n \times k}$ contains the first $k$ columns (excluding the final component) and $v_{21}$ is the trailing component [1401.6832, 2205.00455].

## 2. Robust Truncated Least Squares in Point Cloud Registration

A distinct usage of truncated least squares arises in geometric matching and robust estimation, notably in point cloud registration under high outlier rates [2508.15613]. Here, the per-point squared residual $r_i(R,t)$ is truncated:
\[
\mathrm{TLS}(R,t) = \sum_{i=1}^N \min\left( r_i(R,t),\, \epsilon^2 \right)
\]
where $r_i(R,t) = \| R p_i + t - q_i \|^2$, and $\epsilon > 0$ is a user-defined threshold. This function penalizes only inlier residuals quadratically—outlier residuals saturate at $\epsilon^2$ and do not further influence the cost—yielding extreme outlier robustness. The resulting optimization is combinatorial, as the inlier set is data-dependent and unknown.

Efficient globally optimal solvers employ a combination of convex relaxation (e.g., weighted least squares underestimation), branch-and-bound with pruning (interval contraction based on inlier likelihood), and streamlined active-set routines for nonlinear parameterizations such as rotations. These methods can certify global optimality and solve large-scale registration tasks with high percentages of outliers [2508.15613].

## 3. Truncated SVD and Low-Rank TTLS Algorithms

In large-scale and ill-posed TLS settings, the truncated SVD approach is foundational to TTLS regularization. Instead of directly computing the full SVD of $C$, randomized algorithms generate low-dimensional subspaces that approximate the dominant singular vectors with high probability and low computational cost. For example, given $C = [A\ b]$ and target rank $k$, a typical algorithm [1401.6832, 2205.00455] proceeds as follows:

- Generate a Gaussian random test matrix or use $\ell_2$-norm sampling to rapidly construct a "sketch" $Y = C\Omega$, where the range of $Y$ approximates the top right singular subspace.
- Orthonormalize $Y$ to obtain $Q$, then form a small matrix $Z = Q^\top C$ and compute its truncated SVD.
- Lift the singular vectors back to $C$, partition as before, and compute the TTLS solution $x_k$ in the lower-dimensional subspace.

These techniques reduce computation from $O(m n^2)$ to $O(m n k)$, with accuracy controlled by the singular value gap and the oversampling parameter [1401.6832, 2205.00455]. A quantum-inspired variant (QiTTLS) achieves similar accuracy but leverages advanced sampling data structures for even faster sketch construction [2205.00455].

## 4. Perturbation Analysis and Conditioning of TTLS

The sensitivity of TTLS solutions to perturbations in $A$ and $b$ is captured by condition numbers. Explicit normwise, mixed, and componentwise condition number formulas have been derived. The mapping $[A\ b] \mapsto x_k$ is linearized, and the leading-order derivative is represented by a Kronecker-product based operator $M_k$. The absolute normwise condition number is
\[
\kappa(A,b) = \|M_k\|_2
\]
whereas mixed and componentwise measures provide much sharper and scale-aware bounds, especially significant when data or solution variables are poorly scaled or sparse [2004.12082].

For structured problems (e.g., $A$ restricted to a subspace), structure-preserving condition numbers can be computed; these are always less than or equal to their unstructured counterparts. Small-sample statistical condition estimation (SCE) algorithms provide practical and efficient empirical estimates of all such condition numbers by probing the gradient in random directions, using only the precomputed SVD of $[A\ b]$ [2004.12082].

## 5. Reduced-Rank Analysis, Limitations, and Special Cases

In ordinary least squares (LS), truncating the SVD reduces solution variance at the cost of a controlled increase in bias, leading to a favorable distortion-variance tradeoff that can be exploited using data-dependent ordering of singular directions. In the total least squares case, a naïve application of such truncated projections fails: the optimal truncation level depends on the unknown parameter vector's norm, preventing any adaptive, data-driven scheme from guaranteeing mean square error (MSE) reduction over the full-rank estimator for arbitrary data [1903.01745].

Reduced-rank TLS estimators only arise in special scenarios—for example, when the parameter is known a priori to be norm-bounded, or if side information such as sparsity or prior distributions is available. In these specialized settings, truncation can be meaningfully adapted, but otherwise, the TLS structure precludes effective purely data-driven rank selection [1903.01745].

## 6. Structured and Convex Relaxation Approaches

Structured TLS (STLS) generalizes the standard model to include heterogeneous error variances, block, Toeplitz, or Hankel error constraints, and robust error norms. These structured formulations are nonconvex and do not admit closed-form solutions. Convex relaxation methods, central among them nuclear norm minimization and its reweighted (log-det) variants, provide tractable surrogates for the rank constraint. In standard (unstructured) TLS, nuclear norm relaxation incurs an $O(N)$ factor increase in approximation error, but reweighted relaxations can nearly match the exact TLS SVD solution [1406.0189].

The algorithms can be efficiently implemented using augmented Lagrangian or ADMM schemes. Experimentally, reweighted relaxations outperform both nonconvex optimizers and unweighted convex surrogates, and are highly effective in challenging, application-specific structured TLS problems, robust PCA, and low-rank matrix completion [1406.0189].

## 7. Practical and Algorithmic Considerations

- TTLS effectiveness depends crucially on the truncation index $k$, typically chosen based on the singular value gap or cross-validation/regularization heuristics [1401.6832, 2205.00455].
- For large-scale systems, randomized and quantum-inspired algorithms provide substantial speedups and accurate solutions, as confirmed by benchmark tests on ill-posed problems and applications such as frequency estimation and tomography [1401.6832, 2205.00455, 2004.12082].
- In robust point cloud registration, branch-and-bound solvers exploiting truncation achieve state-of-the-art global optimality and scalability, tolerating outlier rates exceeding 90% [2508.15613].
- Perturbation and condition number analyses, especially those leveraging componentwise or structured counters, offer reliable a posteriori error bounds for practical implementations [2004.12082].

## References

- [1401.6832] Perturbation Analysis and Randomized Algorithms for Large-Scale Total Least Squares Problems
- [2205.00455] Quantum-inspired algorithm for truncated total least squares solution
- [2004.12082] Condition numbers for the truncated total least squares problem and their estimations
- [1406.0189] Convex Total Least Squares
- [2508.15613] Fast globally optimal Truncated Least Squares point cloud registration with fixed rotation axis
- [1903.01745] Reduced-rank Analysis of the Total Least Squares

Source: https://www.emergentmind.com/topics/truncated-least-squares-tls