Hinge Regression Trees and HRT-Boost: Newton-Optimized Oblique Learning for Compact Tabular Models

Abstract: Learning high-quality oblique decision trees remains a significant challenge due to the discrete and non-convex nature of split optimization. We present the Hinge Regression Tree (HRT) framework, which reframes each oblique split as a nonlinear least-squares problem over two linear predictors whose max/min envelope induces ReLU-like representation capacity. We show that the resulting node-level optimization can be interpreted as a damped Newton method, and we establish the monotonic decrease of the node objective for its backtracking line-search variant. We establish, theoretically, that HRT is a universal approximator with an explicit $O(δ^2)$ approximation rate. Building upon this base learner, we propose HRT-Boost, a mathematically synergistic ensemble extension that couples node-level Newton updates with stage-wise functional gradient descent. We show that this ensemble construction admits a stage-wise empirical risk reduction guarantee under the squared loss. Empirical evaluations on synthetic and real-world benchmarks show that HRT is highly competitive with established single-tree baselines, and HRT-Boost compares favorably with strong ensemble baselines and often yields substantially more compact models. The code is publicly available at https://github.com/Hongyi-Li-sz/HRT-Boost.

• The paper presents a novel oblique tree method optimized via damped Newton updates, enabling efficient nonlinear approximations of high-dimensional tabular data.
• The methodology leverages envelope-based hinge functions to achieve global convergence guarantees and compact model structures with fewer leaves compared to traditional trees.
• Empirical results on synthetic and real-world datasets show improved RMSE, MAE, and efficiency, underscoring its potential for scalable and interpretable data modeling.

Hinge Regression Trees and HRT-Boost: Newton-Optimized Oblique Learning for Compact Tabular Models

Introduction and Motivation

The manuscript presents a comprehensive treatment of Hinge Regression Trees (HRT) and their ensemble extension, HRT-Boost, targeting the efficient modeling of nonlinear and high-dimensional tabular data via oblique decision trees optimized with Newton-type methods. The motivation hinges on the trade-off between expressivity and complexity in classical tree methods: axis-aligned trees such as CART suffer from exponential growth in depth and size to model interactions in correlated or high-dimensional settings, while oblique methods, capable of hyperplane splits, face discrete, nonconvex optimization landscapes that hinder practical performance and theoretical analysis.

Hinge Regression Trees: Formulation and Optimization

HRT reformulates node decisions as nonlinear least-squares problems involving the max/min envelope of two linear predictors, producing a locally ReLU-like basis structure. Each internal node is split by adapting the hinge (max/min) between two hyperplanes, where the optimal parameters are obtained via a damped Newton update that alternates between subset re-partitioning and local least squares regression. This makes each update step exactly equivalent to a blockwise Newton or Gauss-Newton update under fixed partitions, leveraging piecewise quadratic structure for computational tractability.

Figure 1: Schematic of HRT and HRT-Boost. HRT splits are optimized as nonlinear least-squares over two linear predictors per node, solved by a damped Newton-type algorithm; the resulting compact oblique tree serves as the base learner for stage-wise boosting in HRT-Boost.

The node objective is proved to decrease monotonically under a backtracking line search variant and converges to the OLS solution once the partition is fixed (Theorem 1). Empirical results (see Figure 2, Figure 3) show that for ill-conditioned or highly oscillatory targets, damping is critical to avoid limit cycles and partition collapse. For well-behaved nodes, unit Newton steps achieve rapid convergence, confirming the method's theoretical dynamics.

Figure 2: Sinc node convergence—monotonicity and stability require proper damping; aggressive steps risk non-monotonicity and poor solutions on nonconvex data.

Figure 3: Twisted sigmoid node convergence—when the objective is well-behaved, even large Newton steps produce rapid and accurate local fits.

Theoretical Properties: Representation and Universal Approximation

The HRT framework is shown to be a universal approximator for $C^2$ targets on compact domains, attaining a partition-dependent rate $O(\delta^2)$, where $\delta$ quantifies the granularity of the partition. This rate matches the best attainable for piecewise linear models and oblique trees, with explicit error scaling in terms of tree depth (Corollary: depth $O(d\log(1/\delta))$ for error $\delta$). The hinge composition yields a hierarchical ReLU circuit, paralleling the expressivity of shallow neural networks and hinging hyperplanes, but with the transparency and explicit structure of decision trees.

HRT-Boost: Functional Gradient Boosting with HRTs

The ensemble extension, HRT-Boost, applies the HRT model as a stage-wise residual fitter within the functional gradient boosting paradigm. This two-level optimization scheme locally employs damped Newton updates at each HRT node, while globally performing gradient descent in function space via ensemble accumulation. Importantly, the authors establish a formal guarantee (Theorem 2) that each boosting stage decreases the empirical risk proportional to the realized fit coefficient $\gamma_m$; if this is uniformly lower-bounded, exponential empirical risk convergence is achieved.

Empirical Results: Synthetic and Real-world Benchmarking

Synthetic Data: Systematic experiments on challenging 2D/3D synthetic functions (oscillatory, highly nonlinear, and with sharp features) confirm that HRT achieves superior or competitive RMSE and MAE relative to strong tree baselines (CART, XGBoost) and that the associated residual distributions are more homoscedastic and less biased. Further, HRT outperforms axis-aligned methods by a clear margin on functions where oblique splits are essential (see Table summaries and synthetic 3D surface visualizations).

Figure 4: Learned HRT fit for a nonlinear 3D function—piecewise linear segments customized by oblique splits effectively model complex response surfaces.

Real Data: For tabular regression benchmarks (Abalone, YearPrediction, CTSlice, etc.), HRT and HRT-Boost consistently exhibit competitive or best-in-class performance versus recent oblique tree models (TAO, DGT, DTSemNet) and boosting frameworks (XGBoost, LightGBM). A notable property is strong structural parsimony: HRT-Boost often requires an order of magnitude fewer leaves and shallower trees to achieve comparable or superior accuracy, attributed to the higher expressivity of each oblique split.

Training and inference efficiency are also analyzed (see FLOPs count tables), showing that HRT-Boost maintains low computational cost per sample due to its compact ensemble size, a point of practical significance in large-scale deployments.

Implications and Outlook

The explicit marriage of Newton-type optimization with piecewise linear partitioning yields robust and analytically tractable oblique trees. The combination of local (node-level) and global (ensemble-level) guarantees makes the framework suitable for high-stakes domains where both interpretability (tree structure) and predictive robustness (compactness/accuracy) are required. The universal approximation rate, formalized monotonicity of optimization, and empirical compactness/efficiency provide clear advancements over both axis-aligned and prior oblique methods.

HRT-Boost presents a compelling direction for further research:

• Theoretical exploration: Extensions to loss functions beyond squared error, statistical analyses under non-i.i.d. or heavy-tailed regimes, and deeper connections to neural ReLU architectures.
• Practical adaptation: Incorporation into streaming, online, or multi-output setups, and integration with hardware-aware or resource-bounded model selection for edge applications.
• Model interpretability: The clear structure and shallow nature of HRT/HRT-Boost facilitate formal post-hoc analysis, opening the door for advances in explainable AI over complex tabular data.

Conclusion

Hinge Regression Trees and HRT-Boost provide a principled and efficient solution to the challenges of oblique tree modeling for tabular data. By recasting oblique splits as Newton-optimized, envelope-based piecewise linear regressions, the method achieves strong theoretical guarantees on convergence and approximation, and translates these into state-of-the-art practical performance on synthetic and real-world benchmarks. The approach successfully unifies the expressivity of hierarchical ReLU structures with the transparency and efficiency of tree models.