Faster than Fast-LTS: Robust Regression and Outlier Detection with DC Programming

Abstract: When datasets contain outliers, robust regression is a well-established alternative to Ordinary Least Squares. A commonly employed robust estimator is Least Trimmed Squares (LTS), which computes the regression coefficients from a subset of observations. Determining the exact solution corresponds to a combinatorial problem with prohibitive computational costs, even for instances of moderate dimension. Thus, the most prevalent approach in practice remains a heuristic known as Fast-LTS. Although the heuristic often performs effectively, certain elements of the approach remain open to improvement. In particular, its core procedure provides robust results only when initialized with a large number of starting points. To address the heuristic's limitations, this paper reformulates the LTS problem as a concave minimization problem subject to a capped simplex constraint, and proposes the successive Boosted Difference of Convex Functions Algorithm (sBDCA) as a solution method. Theoretically, we establish via the Łojasiewicz property that sBDCA converges to a local solution with a linear rate in the fastest case. To ensure robustness from a single initialization in practice, we derive and integrate a problem-specific preconditioning matrix into the algorithmic setup. Building on this theoretical foundation, we conduct numerical studies on various synthetic and real-world datasets to demonstrate the effectiveness of sBDCA with preconditioning. Specifically, we show that our approach is up to 3.25 times faster than Fast-LTS and achieves up to 90% lower objective function values, particularly in high-dimensional settings. As all code is openly available, this paper further provides a practical guide to robust regression in Python.

• The paper introduces a rigorous DC programming reformulation of LTS to tackle the computational challenges and statistical robustness in high-dimensional regression.
• It presents the innovative sBDCA algorithm with analytic preconditioning, achieving up to 3.25× speed improvements and lower objective values than Fast-LTS.
• Empirical evaluations on synthetic and real-world datasets demonstrate strong convergence guarantees and enhanced reliability of the method over traditional approaches.

Robust Regression and Outlier Detection via DC Programming: A Technical Analysis of Successive BDCA for Least Trimmed Squares

Introduction

The paper "Faster than Fast-LTS: Robust Regression and Outlier Detection with DC Programming" (2606.28974) presents a comprehensive study addressing the computational and statistical challenges of robust regression in the presence of outliers, focusing on the Least Trimmed Squares (LTS) estimator. The LTS method, while statistically desirable due to its high breakdown point and efficiency, is fundamentally hindered by its computational intractability in moderate and high dimensions. Existing heuristics such as Fast-LTS, though widely adopted in practice, have well-documented limitations regarding initialization sensitivity and lack of robust theoretical guarantees. This work establishes a rigorous mathematical programming reformulation of LTS and introduces a novel Difference of Convex functions (DC) program solved by a Successive Boosted DC Algorithm (sBDCA), explicitly targeting both local optimality and improved computational performance.

Problem Formulation and Limitations of Fast-LTS

Classical OLS regression, despite its attractive computational properties, is highly sensitive to outliers which can induce masking and swamping effects, particularly apparent in multivariate settings.

Figure 1: Masking, Swamping, and Types of Outliers in Regression Analysis.

LTS robustly estimates regression coefficients using the $h$ smallest squared residuals, but computing the optimal subset is combinatorially hard, scaling factorially with both $n$ (samples) and $p$ (predictors). Heuristic solutions like Fast-LTS are iterative: they repeatedly select random $p$-subsets to initialize, iteratively concentrate on the $h$ smallest residuals, and retain the best solution. However, as dimensionality grows, the probability that any random $p$-subset is "clean" (free of outliers) vanishes, causing solution degradation and requiring a prohibitive number of restarts. Moreover, Fast-LTS's core procedure is not provably robust from a single initialization, lacks convergence to stationary points under general conditions, and solution quality is strongly initialization-dependent, particularly in high-dimensional regimes.

Mathematical Programming Reformulation: DC Programming for LTS

The authors rigorously derive a DC programming reformulation of LTS. Binary decision variables select the subset, which upon relaxation results in a capped simplex constraint. By analytical reduction, the inner minimization (for a fixed selection) yields a closed-form in the style of weighted least-squares, resulting in the key concave objective function:

$$f(\mathbf{z}) = \mathbf{y}^\top \mathbf{Z} \mathbf{y} - \mathbf{y}^\top \mathbf{Z} \mathbf{X} (\mathbf{X}^\top \mathbf{Z}\mathbf{X})^{-1} \mathbf{X}^\top \mathbf{Z} \mathbf{y},$$

where $\mathbf{z}$ is subject to box ($0 \leq z_i \leq 1$) and sum ($\sum z_i = h$) constraints. All local and global optima lie on simplex vertices, justifying the relaxation and confirming equivalence to the original combinatorial LTS in terms of solution support.

The paper presents several viable DC decompositions, ultimately adopting:

$n$0

where $n$1 is a tunable convexifying parameter potentially varying across iterations for faster convergence and improved robustness.

Figure 2: Influence of $n$2 on ATD and computation time for fixed/successive DC decompositions.

The Successive Boosted DCA (sBDCA) Algorithm

The sBDCA framework applies a projection-based proximal gradient update in $n$3, projecting onto the capped simplex at each iteration and adapting the convexification parameter $n$4. It features a simplified line search based on the monotonic descent properties of the constructed DC surrogate. The paper includes a detailed convergence analysis leveraging real-analytic properties (Cayley-Hamilton theorem), confirming that the method achieves at least sublinear convergence (and linear for $n$5 within the Łojasiewicz framework), and all limit points are local minima.

Initialization and Preconditioning

Initialization fidelity is critical in nonconvex programming over large polytopes. The authors examine two key strategies: (i) random Dirichlet-distributed interior points and (ii) vertex-based subsets inspired by Fast-LTS ($n$6-subset initializations). Empirical results decisively demonstrate that, absent a large number of restarts, initialization quality dictates final performance, especially in high $n$7 regimes.

Figure 3: Influence of $n$8 on Dirichlet PDF and its effect on subset generation.

Figure 4: Comparison of sBDCA performance under Dirichlet and $n$9-subset initialization.

To mitigate the effect of "good leverage" initialization failures, the authors introduce an analytic preconditioner based on the hat matrix, effectively a Quasi-Newton correction, which rescales gradient steps according to leverage information. This preconditioner targets regression outliers and high-leverage points, enabling the optimization to recover robustly from poor starts—even when initialized at or near combinatorially suboptimal vertices.

Figure 5: Influence of Preconditioner on ATD and computation time.

Empirical Evaluation

Low-dimensional Benchmarks

Thorough evaluations on classical regression datasets demonstrate that sBDCA with preconditioning obtains median ATD (Average Trimmed Distance) values comparable to—and in some cases better than—Fast-LTS, consistently outperforming standard M-estimators, RANSAC, Huber, and Theil-Sen regression approaches.

Figure 6: Simple linear regression fits for various robust estimators.

High-dimensional Synthetic and Real Data

On medium- and high-dimensional synthetic datasets (outlier and leverage contamination under OSCM and DMLP), sBDCA with preconditioning yields up to 90% lower objective function values than Fast-LTS and is up to 3.25× faster, particularly as $p$0 increases. The variability and breakdown rate of Fast-LTS grows in high $p$1/$p$2 regimes due to its random-start dependence, a phenomenon not observed in the DC programming approach.

Figure 7: ATD and infeasibility rates as a function of $p$3 for OSCM synthetic data.

Figure 8: Median iterations and computation time across $p$4 for OSCM datasets.

Figure 9: Performance comparison on DMLP (leverage point) synthetic data.

Figure 10: Optimization trajectories of sBDCA and Fast-LTS in nonconvex LTS landscape.

Figure 11: Iteration count and timing for DMLP datasets.

Evaluation on large real-world datasets (Million Song Dataset, Superconductivity Data) demonstrates retention of superior reliability and solution quality, though the relative gains of preconditioning are reduced in the presence of strong predictor correlation and complex multivariate error structures.

Figure 12: ATD and infeasibility rates on real-world music and science datasets.

Figure 13: Computation time and iteration counts for real-world data.

Theoretical and Practical Implications

From a theoretical perspective, this work establishes that robust LTS regression can be efficiently approximated—even in high-dimensions—by a carefully constructed sequence of DC programs, leveraging tailored convex-concave decompositions and problem-informed preconditioners. The underlying framework is general and applicable to other combinatorial robust regression estimators.

Practically, the DC approach provides strong local descent guarantees, scalability, easy parallelization (due to independence of initializations), and considerably reduces the need for multi-start heuristics. Open-source Python code enhances reproducibility and practical adoption.

Potential Future Developments

Opportunities for further advancements include:

• Automated selection and adaptation of the DC decomposition and preconditioning matrix for problem-specific geometry;
• Systematic design of initialization strategies informed by data geometry (e.g., robust principal components for subset selection);
• Exploration of alternative robust residual measures and extensions to variable selection and high-dimensional penalized models;
• Benchmarking under adversarial contamination and leveraging robust estimation in generalized linear or nonparametric settings.

Conclusion

This work rigorously advances robust regression methodology, showing that DC programming—implemented via sBDCA with analytic leverage-based preconditioning—resolves the major computational and statistical deficiencies of Fast-LTS in both theory and practice. It establishes a new state-of-the-art in the LTS family, particularly in high-dimensional and heavily contaminated regimes, and sets the stage for broad adoption and further inquiry into convex-analytic approaches for robust statistical learning.