Computation of Least Trimmed Squares: A Branch-and-Bound framework with Hyperplane Arrangement Enhancements

Abstract: We study computational aspects of a key problem in robust statistics -- the penalized least trimmed squares (LTS) regression problem, a robust estimator that mitigates the influence of outliers in data by capping residuals with large magnitudes. Although statistically attractive, penalized LTS is NP-hard, and existing mixed-integer optimization (MIO) formulations scale poorly due to weak relaxations and exponential worst-case complexity in the number of observations. We propose a new MIO formulation that embeds hyperplane arrangement logic into a perspective reformulation, explicitly enforcing structural properties of optimal solutions. We show that, if the number of features is fixed, the resulting branch-and-bound tree is of polynomial size in the sample size. Moreover, we develop a tailored branch-and-bound algorithm that uses first-order methods with dual bounds to solve node relaxations efficiently. Computational experiments on synthetic and real datasets demonstrate substantial improvements over existing MIO approaches: on synthetic instances with 5000 samples and 20 features, our tailored solver reaches a 1% gap in 1 minute while competing approaches fail to do so within one hour. These gains enable exact robust regression at significantly larger sample sizes in low-dimensional settings.

• The paper introduces a novel MIO formulation that integrates hyperplane arrangement logic into a branch-and-bound method for penalized LTS regression.
• The proposed approach achieves a polynomial complexity bound for fixed dimensions, significantly improving scalability versus traditional big-M and heuristic methods.
• Empirical evaluations on synthetic and real datasets demonstrate superior optimality and computational efficiency compared to state-of-the-art robust regression algorithms.

Branch-and-Bound Methods for Penalized Least Trimmed Squares with Hyperplane Arrangement Enhancements

Introduction and Problem Setting

The penalized Least Trimmed Squares (LTS) regression framework is a classical centerpiece in robust statistics, notable for its capacity to mitigate the deleterious influence of outliers. Despite its widespread use and statistical desirability, the computation of globally optimal solutions for penalized LTS formulations remains an NP-hard challenge, predominantly due to the combinatorial selection of inliers versus outliers. Existing mixed-integer optimization (MIO) approaches typically suffer from weak relaxations and an exponential dependence on the sample size, which has historically limited their tractability to small- or moderate-scale instances.

This work presents an enhanced and theoretically principled framework for exact penalized LTS regression, combining a new MIO formulation and a tailored branch-and-bound (BnB) algorithm which integrates hyperplane arrangement logic to strengthen the enumeration and relaxation steps. The analysis yields rigorous polynomial upper bounds on the branch-and-bound complexity for fixed $p$, and the presented implementation achieves exact robust regression for substantially larger sample sizes than prior exact algorithms.

Existing Approaches and Limitations

Early MIO formulations for penalized LTS typically rely on big-M constraints, which suffer from severe relaxations and require impractically conservative bounds for correctness. More recent advances leverage perspective reformulations to avoid big-M constants and exploit separability in the loss, yielding stronger convex relaxations. Nevertheless, such formulations nearly always treat observations independently, ignoring the intrinsic combinatorial structure imposed by the outlier-inlier selection, and their best-case complexity remains exponential in $n$.

Classical exact combinatorial algorithms based on hyperplane arrangements, as proposed in the statistical literature, can in principle solve penalized LTS in polynomial time in $n$ for fixed $p$, by partitioning the regression coefficient space into regions associated with unique outlier configurations. However, beyond $p=2$, no practical implementations are known due to combinatorial explosion. Heuristics such as FAST-LTS, based on alternating minimization, are computationally efficient but lack any optimality guarantees and commonly yield suboptimal or inconsistent estimates on contaminated real-world datasets.

Novel Formulation and Theoretical Contributions

The core contribution of this work is an MIO formulation for the penalized LTS problem that incorporates key structural properties of the optimal solution—specifically, logical relations between the regression coefficients and the outlier indicators—via a hyperplane arrangement encoding. By enforcing that outliers must fall sufficiently far from the fit (specifically outside a residual band with width determined by the penalty), the feasible space is aggressively pruned during branching.

This is formalized by embedding disjunctive "optimality logic" (e.g., $z_i = 0 \implies |r_i| \leq \sqrt{2\mu}$; $z_i = 1 \implies |r_i| \geq \sqrt{2\mu}$) as tight constraints without recourse to big-M terms. The resulting formulation is provably the convex hull of the valid imposed region and can be realized either directly in MIO solvers or, more scalably, via custom first-order methods over a projected regression parameter space.

A key theoretical advance is the polynomial complexity bound for the BnB process: when the data dimension $p$ is fixed, the number of BnB nodes is at most $O(n^{p+1})$. The polynomial structure arises from correspondences to the regions of the hyperplane arrangement induced by trimmed squares logic, as each region represents a unique selection of inlier/outlier patterns compatible with the residual thresholds.

Figure 1: Illustration of a depth-5 integer node with four infeasible relatives, highlighting the combinatorial structure of the BnB framework under hyperplane arrangements.

Tailored Branch-and-Bound Algorithm

To leverage the enhanced formulation, the authors develop a custom BnB solver that solves node relaxations over the (typically low-dimensional) regression parameter space using differentiable augmented Lagrangian relaxations. This circumvents the need to solve large-dimension conic programs at each node—an evident limitation of commercial solvers. The algorithm implements three-way branching logic at each node, tailored to the structure of the feasible region partitioning, which aligns naturally with the combinatorial geometry of the residual-based pruning constraints.

Node relaxations admit efficient closed-form solutions for the continuous relaxation of the loss function (see Figure 2—comparison of losses), and the proposed method adheres to reliable dual bounding at each node via strong convexity arguments. Early pruning and warm-start mechanisms are exploited, yielding strong computational efficiency.

Figure 2: Comparison of the original least trimmed loss $\phi_{\text{cap}}$, its perspective-relaxed counterpart, and the Big-M relaxation—the perspective relaxation closely tracks the capped loss for moderate residuals and strictly dominates the Big-M approach.

Empirical Evaluation and Numerical Results

The computational study is extensive, including synthetic and thirteen real benchmark regression datasets. The proposed approach demonstrates substantial improvements over all competing MIO methods:

• On synthetic problems with $n$0 and $n$1, the BnB solver attains a $n$2 optimality gap in 1 minute, while state-of-the-art commercial MIO solvers either fail to close the gap within an hour or solve effectively only for $n$3.
• On real regression benchmarks, the BnB solver and Gurobi with the hyperplane-arrangement formulation each achieve the best runtimes—one solver typically dominates per instance, and both strictly outperform classic perspective or Big-M approaches.

Crucially, the proposed BnB method can close the optimality gap on many instances where even heuristic methods exhibit substantial suboptimality, with relative objective value losses sometimes exceeding $n$4 (see Figure 3—boxplot of suboptimality for heuristic versus optimal computed solutions).

Figure 4: Performance profile aggregating all methods across real datasets, showing the fraction of instances solved within a given time threshold to $n$5 optimality.

Figure 3: Box plot showing that heuristic solutions, such as alternating minimization, are often significantly suboptimal compared to those obtained via the BnB framework, with many cases exhibiting gaps exceeding 1% and some as large as 30%.

Implications and Future Directions

The proposed branch-and-bound framework renders the exact penalized LTS approach accessible at scales that were previously inaccessible to exact algorithms. This unlocks robust regression estimation with guarantees in moderate- to large-$n$6, low- to moderate-$n$7 settings, particularly relevant for modern high-reliability or security-critical applications (e.g., sensor monitoring, medical regressions, and AI systems requiring rigorous outlier insensitivity).

From a theoretical perspective, the work deepens the interface between combinatorial geometry (hyperplane arrangements) and mixed-integer optimization for robust statistics. The embedding of logical-optimality constraints within the MIO paradigm, and the practical algorithmic derivation of convex hull formulations, suggests future research towards other robust or combinatorial estimators (e.g., multivariate LTS, generalized linear models, robust subset selection).

While this approach is presently tailored to $n$8 and moderate $n$9 (with polynomial but still exponential-in-$n$0 scaling), further algorithmic advances in arrangement traversal, constraint generation, or tighter problem-specific relaxations could extend scalability. Conversely, for high-dimensional, sparse regimes, hybrid methods combining branch-and-bound with screening or convex approximations may yield improved practical utility.

Conclusion

This study provides a rigorous, computationally feasible, and empirically validated solution to the penalized LTS regression problem via a logical-tight MIO formulation and a custom BnB algorithm. The theoretical guarantee of polynomial branch-and-bound complexity in fixed dimension $n$1 and the strong empirical dominance over commercial and heuristic alternatives represent a significant advance in robust regression computation. These methods establish the foundation for further developments in robust and statistically motivated combinatorial optimization in statistical machine learning.

(2604.11584)