Hinge Regression Tree (HRT)

• Hinge Regression Tree (HRT) is an oblique decision-tree model that formulates each split as a nonlinear least-squares problem over two linear predictors using a ReLU-like hinge basis.
• It leverages an alternating optimization procedure equivalent to a Gauss-Newton update with backtracking and optional ridge regularization for robust convergence.
• Empirical results show HRT delivers competitive RMSE with compact, shallow trees, outperforming CART and matching ensemble methods on various regression tasks.

Searching arXiv for the specified HRT papers and closely related work to ground the article. {"query":"arXiv Hinge Regression Tree oblique regression tree splitting HRT (Li et al., 5 Feb 2026, Li et al., 22 May 2026)", "max_results": 10} arxiv_search({"query":"Hinge Regression Tree oblique regression tree splitting", "max_results": 10}) Hinge Regression Tree (HRT) is an oblique decision-tree framework for regression in which each internal split is formulated as a nonlinear least-squares problem over two linear predictors whose max/min envelope defines a hinge basis with ReLU-like expressive power. The method was introduced to address the fact that learning high-quality oblique splits is NP-hard and that practical methods have often relied on slow search or theory-free heuristics. In HRT, the alternating procedure of assigning samples to one of two linear models and refitting those models by least squares is not merely heuristic: within a fixed partition it is exactly equivalent to a damped Newton, specifically Gauss-Newton, method. The framework further admits a backtracking line-search variant with monotonic objective decrease and convergence guarantees, optional ridge regularization for numerical stability, and a universal approximation theorem with an explicit $O(\delta^2)$ rate; a later work extends the same base learner to HRT-Boost (Li et al., 5 Feb 2026, Li et al., 22 May 2026).

1. Formal definition and split geometry

At an internal node $t$, HRT considers training points $\{(x_j,y_j)\}_{j\in D_t}$ with $x_j\in\mathbb R^d$, and uses the augmented feature vector $\tilde x_j=[\,x_j^T,1\,]^T$. Two linear models are maintained: $$\ell_{t1}(x)=\tilde x^T\theta_{t1},\qquad \ell_{t2}(x)=\tilde x^T\theta_{t2},$$ stacked as $$\theta=[\theta_{t1}^T,\theta_{t2}^T]^T\in\mathbb R^{2(d+1)}$$. The node-level objective is

$V(\theta)=\frac12\sum_{j\in D_t}\Bigl(y_j-h(\tilde x_j;\theta)\Bigr)^2, \qquad h(\tilde x;\theta)=\max\bigl\{\ell_{t1}(x),\ell_{t2}(x)\bigr\} \quad\text{(or %%%%6%%%%)}.$

Here $h(\tilde x_j;\theta)$ is a piecewise-linear hinge basis defined as the upper or lower envelope of the two linear functions, and the induced decision boundary is the hyperplane

$\{\tilde x:\;\ell_{t1}(x)=\ell_{t2}(x)\}.$

This makes the split oblique rather than axis-aligned, since the boundary is determined by the equality of two linear predictors rather than by thresholding a single coordinate (Li et al., 5 Feb 2026).

The later HRT exposition makes the ReLU connection explicit through the identity

$t$0

Under this view, each split acts as a ReLU gate on the difference between the two affine responses. Recursively composing such splits up to depth $t$1 yields a nested ReLU circuit of size $t$2 but often with much smaller effective depth. This does not turn HRT into a standard neural network; rather, it characterizes the representational capacity of the hinge envelope used at each node (Li et al., 22 May 2026).

A common misconception is that HRT is simply an oblique-tree search heuristic with an unconventional split score. The defining feature is instead the explicit regression formulation of each split: the split is optimized as nonlinear least squares over two local linear predictors, and the split geometry is part of that optimization problem rather than an external search primitive.

2. Newton interpretation of the node-level algorithm

At iterate $t$3, HRT forms the active partition

$t$4

On a fixed partition, the hinge is inactive and the objective becomes a smooth quadratic: $t$5 With $t$6 the $t$7 design matrix and $t$8 the response vector, the gradient and Hessian are

$t$9

Because $\{(x_j,y_j)\}_{j\in D_t}$0 is locally linear, the Gauss-Newton approximation is exact on that region. The Newton direction $\{(x_j,y_j)\}_{j\in D_t}$1 decouples into two ordinary least-squares problems, so that

$\{(x_j,y_j)\}_{j\in D_t}$2

Equivalently, the damped-Newton update is

$\{(x_j,y_j)\}_{j\in D_t}$3

The case $\{(x_j,y_j)\}_{j\in D_t}$4 is exactly the assign-then-OLS update; smaller $\{(x_j,y_j)\}_{j\in D_t}$5 damps the Newton step (Li et al., 5 Feb 2026).

This equivalence is central to the method’s interpretation. The alternating fitting procedure is not presented as an ad hoc block coordinate routine. Within a fixed partition, it is the full Newton step, and the linearly separable structure of the Hessian explains why the computation reduces to two independent least-squares fits. This also clarifies why the method can remain stable in practice despite the nondifferentiability of the max/min envelope: the optimization proceeds through smooth quadratic regions separated by repartitioning events.

3. Descent guarantees, backtracking, and regularization

HRT supports both fixed damping and an Armijo-style backtracking line search. In the backtracking variant, one starts with $\{(x_j,y_j)\}_{j\in D_t}$6, forms $\{(x_j,y_j)\}_{j\in D_t}$7, accepts the step if $\{(x_j,y_j)\}_{j\in D_t}$8, and otherwise reduces the step, for example via $\{(x_j,y_j)\}_{j\in D_t}$9 with $x_j\in\mathbb R^d$0, until a strict decrease is found or $x_j\in\mathbb R^d$1 becomes tiny. The more formal statement in the extended account uses the condition

$x_j\in\mathbb R^d$2

Under mild regularity assumptions—specifically, that no point lies exactly on the hyperplane at the Newton iterate and that each local normal matrix is uniformly well-conditioned, or in the later statement $x_j\in\mathbb R^d$3—the node-level objective decreases monotonically under backtracking, and the resulting sequence $x_j\in\mathbb R^d$4 converges to some $x_j\in\mathbb R^d$5 (Li et al., 5 Feb 2026, Li et al., 22 May 2026).

The convergence description is sharper once the partition stabilizes. If $x_j\in\mathbb R^d$6 stops changing, the updates reduce to

$x_j\in\mathbb R^d$7

which is a linear contraction to the unique OLS minimizer for that fixed partition. The later formulation states the same conclusion as $x_j\in\mathbb R^d$8 for that partition. The caveat “unless one cycles” is explicit in the earlier account, so the stated convergence mechanism is local to the regime in which repartitioning no longer changes (Li et al., 5 Feb 2026).

Optional $x_j\in\mathbb R^d$9 regularization is incorporated into each node-wise least-squares fit: $\tilde x_j=[\,x_j^T,1\,]^T$0 with closed form

$\tilde x_j=[\,x_j^T,1\,]^T$1

The stated role of this regularization is to raise the minimum eigenvalue of $\tilde x_j=[\,x_j^T,1\,]^T$2, improving numerical stability and guarding against overfitting in small leaves. This suggests that ridge is not an auxiliary add-on but a practical safeguard for the node solver when collinearity or limited sample size makes the local least-squares system ill-conditioned (Li et al., 5 Feb 2026).

4. Approximation theory and representational capacity

The HRT model class is described as a family of continuous piecewise-linear functions with oblique splits. In one formulation,

$\tilde x_j=[\,x_j^T,1\,]^T$3

For $\tilde x_j=[\,x_j^T,1\,]^T$4 on a compact set $\tilde x_j=[\,x_j^T,1\,]^T$5, and for any tessellation of $\tilde x_j=[\,x_j^T,1\,]^T$6 into polytopes $\tilde x_j=[\,x_j^T,1\,]^T$7 of maximum diameter $\tilde x_j=[\,x_j^T,1\,]^T$8, there exists an HRT fit satisfying

$\tilde x_j=[\,x_j^T,1\,]^T$9

The constant $$\ell_{t1}(x)=\tilde x^T\theta_{t1},\qquad \ell_{t2}(x)=\tilde x^T\theta_{t2},$$0 depends on $$\ell_{t1}(x)=\tilde x^T\theta_{t1},\qquad \ell_{t2}(x)=\tilde x^T\theta_{t2},$$1 and on a uniform lower bound $$\ell_{t1}(x)=\tilde x^T\theta_{t1},\qquad \ell_{t2}(x)=\tilde x^T\theta_{t2},$$2 for the smallest eigenvalue of each local normal matrix $$\ell_{t1}(x)=\tilde x^T\theta_{t1},\qquad \ell_{t2}(x)=\tilde x^T\theta_{t2},$$3. The rate $$\ell_{t1}(x)=\tilde x^T\theta_{t1},\qquad \ell_{t2}(x)=\tilde x^T\theta_{t2},$$4 is identified as the classical order for first-order Taylor approximations of $$\ell_{t1}(x)=\tilde x^T\theta_{t1},\qquad \ell_{t2}(x)=\tilde x^T\theta_{t2},$$5 functions, and the cited works use it to establish that oblique piecewise-linear trees are universal approximators (Li et al., 5 Feb 2026).

The later work restates the same theorem for finite-depth HRTs with linear leaves on compact $$\ell_{t1}(x)=\tilde x^T\theta_{t1},\qquad \ell_{t2}(x)=\tilde x^T\theta_{t2},$$6, giving a proof sketch based on first-order Taylor approximation on each region $$\ell_{t1}(x)=\tilde x^T\theta_{t1},\qquad \ell_{t2}(x)=\tilde x^T\theta_{t2},$$7. It adds a depth-error scaling corollary: if a balanced HRT of depth $$\ell_{t1}(x)=\tilde x^T\theta_{t1},\qquad \ell_{t2}(x)=\tilde x^T\theta_{t2},$$8 yields region diameters $$\ell_{t1}(x)=\tilde x^T\theta_{t1},\qquad \ell_{t2}(x)=\tilde x^T\theta_{t2},$$9, then

$$\theta=[\theta_{t1}^T,\theta_{t2}^T]^T\in\mathbb R^{2(d+1)}$$0

to achieve $$\theta=[\theta_{t1}^T,\theta_{t2}^T]^T\in\mathbb R^{2(d+1)}$$1-accuracy (Li et al., 22 May 2026).

These results delimit the meaning of “universal approximator” in the HRT setting. The claim is not that every training procedure will realize the optimal approximation automatically. Rather, it is an existence result for the model class under increasingly fine partitions and appropriate local linear fits. A plausible implication is that the hinge-based oblique partitioning gives HRT a theoretical status closer to that of piecewise-linear neural constructions than to classical axis-aligned regression trees.

5. Empirical performance and compactness

The reported experiments cover synthetic functions, synthetic surfaces, and real-world tabular regression. On 2D synthetic functions, specifically the sinc function

$$\theta=[\theta_{t1}^T,\theta_{t2}^T]^T\in\mathbb R^{2(d+1)}$$2

and the twisted sigmoid

$$\theta=[\theta_{t1}^T,\theta_{t2}^T]^T\in\mathbb R^{2(d+1)}$$3

HRT beats CART in RMSE, with the example $$\theta=[\theta_{t1}^T,\theta_{t2}^T]^T\in\mathbb R^{2(d+1)}$$4 versus HRT $$\theta=[\theta_{t1}^T,\theta_{t2}^T]^T\in\mathbb R^{2(d+1)}$$5, and even outperforms XGBoost on the twisted-sigmoid. In the same experiments, HRT used $$\theta=[\theta_{t1}^T,\theta_{t2}^T]^T\in\mathbb R^{2(d+1)}$$6 leaves at depth $$\theta=[\theta_{t1}^T,\theta_{t2}^T]^T\in\mathbb R^{2(d+1)}$$7, versus CART’s $$\theta=[\theta_{t1}^T,\theta_{t2}^T]^T\in\mathbb R^{2(d+1)}$$8 leaves at depth $$\theta=[\theta_{t1}^T,\theta_{t2}^T]^T\in\mathbb R^{2(d+1)}$$9 (Li et al., 5 Feb 2026).

On 3D synthetic surfaces $V(\theta)=\frac12\sum_{j\in D_t}\Bigl(y_j-h(\tilde x_j;\theta)\Bigr)^2, \qquad h(\tilde x;\theta)=\max\bigl\{\ell_{t1}(x),\ell_{t2}(x)\bigr\} \quad\text{(or %%%%6%%%%)}.$0, described as oscillatory, inflected, and multi-bump landscapes, HRT’s mean RMSE is typically $V(\theta)=\frac12\sum_{j\in D_t}\Bigl(y_j-h(\tilde x_j;\theta)\Bigr)^2, \qquad h(\tilde x;\theta)=\max\bigl\{\ell_{t1}(x),\ell_{t2}(x)\bigr\} \quad\text{(or %%%%6%%%%)}.$1 lower than CART and often matches or slightly betters XGBoost. The HRT trees remain shallow, with $V(\theta)=\frac12\sum_{j\in D_t}\Bigl(y_j-h(\tilde x_j;\theta)\Bigr)^2, \qquad h(\tilde x;\theta)=\max\bigl\{\ell_{t1}(x),\ell_{t2}(x)\bigr\} \quad\text{(or %%%%6%%%%)}.$2 depth and $V(\theta)=\frac12\sum_{j\in D_t}\Bigl(y_j-h(\tilde x_j;\theta)\Bigr)^2, \qquad h(\tilde x;\theta)=\max\bigl\{\ell_{t1}(x),\ell_{t2}(x)\bigr\} \quad\text{(or %%%%6%%%%)}.$3 leaves, whereas CART and XGB often build hundreds of leaves or depth $V(\theta)=\frac12\sum_{j\in D_t}\Bigl(y_j-h(\tilde x_j;\theta)\Bigr)^2, \qquad h(\tilde x;\theta)=\max\bigl\{\ell_{t1}(x),\ell_{t2}(x)\bigr\} \quad\text{(or %%%%6%%%%)}.$4. In the later summary of synthetic experiments, the 2D setups use 1 000 samples with Gaussian noise $V(\theta)=\frac12\sum_{j\in D_t}\Bigl(y_j-h(\tilde x_j;\theta)\Bigr)^2, \qquad h(\tilde x;\theta)=\max\bigl\{\ell_{t1}(x),\ell_{t2}(x)\bigr\} \quad\text{(or %%%%6%%%%)}.$5, and the 3D oscillatory surfaces use 10 000 samples with $V(\theta)=\frac12\sum_{j\in D_t}\Bigl(y_j-h(\tilde x_j;\theta)\Bigr)^2, \qquad h(\tilde x;\theta)=\max\bigl\{\ell_{t1}(x),\ell_{t2}(x)\bigr\} \quad\text{(or %%%%6%%%%)}.$6 (Li et al., 5 Feb 2026, Li et al., 22 May 2026).

On real-world regression, the first paper reports seven UCI and three industrial datasets and states that HRT matches or outperforms single-tree baselines—CART, GUIDE-like linear trees, TAO-oblique, DGT, and DTSemNet—on RMSE in $V(\theta)=\frac12\sum_{j\in D_t}\Bigl(y_j-h(\tilde x_j;\theta)\Bigr)^2, \qquad h(\tilde x;\theta)=\max\bigl\{\ell_{t1}(x),\ell_{t2}(x)\bigr\} \quad\text{(or %%%%6%%%%)}.$7 of cases. Compared to XGBoost, HRT is within $V(\theta)=\frac12\sum_{j\in D_t}\Bigl(y_j-h(\tilde x_j;\theta)\Bigr)^2, \qquad h(\tilde x;\theta)=\max\bigl\{\ell_{t1}(x),\ell_{t2}(x)\bigr\} \quad\text{(or %%%%6%%%%)}.$8 of the best ensemble RMSE on large tasks such as MSLR and YearPred while using a single, interpretable tree. Typical HRT trees have depth $V(\theta)=\frac12\sum_{j\in D_t}\Bigl(y_j-h(\tilde x_j;\theta)\Bigr)^2, \qquad h(\tilde x;\theta)=\max\bigl\{\ell_{t1}(x),\ell_{t2}(x)\bigr\} \quad\text{(or %%%%6%%%%)}.$9 and $h(\tilde x_j;\theta)$0 leaves, versus CART’s $h(\tilde x_j;\theta)$1 depth and hundreds of leaves or TAO’s $h(\tilde x_j;\theta)$2 depth. Training time is described as only a few times slower than CART on medium-sized data and remaining practical, $h(\tilde x_j;\theta)$3 min, on 50k–500k samples (Li et al., 5 Feb 2026).

The later paper reports 12 benchmarks and states that a single HRT yields the best or near-best RMSE on 8/12 datasets, with model complexity approximately depth $h(\tilde x_j;\theta)$4 and leaves $h(\tilde x_j;\theta)$5, substantially smaller than TAO at depth 6–36 and leaves $h(\tilde x_j;\theta)$6. It also reports training times for HRT as 10×–100× faster than gradient-descent oblique methods (Li et al., 22 May 2026).

Setting	Accuracy summary	Complexity summary
2D synthetic functions	HRT beats CART in RMSE; on twisted-sigmoid it even outperforms XGBoost	HRT used $h(\tilde x_j;\theta)$7 leaves at depth $h(\tilde x_j;\theta)$8, vs. CART’s $h(\tilde x_j;\theta)$9 leaves at depth $\{\tilde x:\;\ell_{t1}(x)=\ell_{t2}(x)\}.$0
3D synthetic surfaces	Mean RMSE typically $\{\tilde x:\;\ell_{t1}(x)=\ell_{t2}(x)\}.$1 lower than CART; often matches or slightly betters XGBoost	HRT remains shallow: $\{\tilde x:\;\ell_{t1}(x)=\ell_{t2}(x)\}.$2 depth, $\{\tilde x:\;\ell_{t1}(x)=\ell_{t2}(x)\}.$3 leaves
Real-world regression	Matches or outperforms single-tree baselines on RMSE in $\{\tilde x:\;\ell_{t1}(x)=\ell_{t2}(x)\}.$4 of cases; within $\{\tilde x:\;\ell_{t1}(x)=\ell_{t2}(x)\}.$5 of best ensemble RMSE on large tasks	Typical HRT trees have depth $\{\tilde x:\;\ell_{t1}(x)=\ell_{t2}(x)\}.$6 and $\{\tilde x:\;\ell_{t1}(x)=\ell_{t2}(x)\}.$7 leaves

Taken together, the empirical record is consistent with the framework’s stated goal: accuracy competitive with strong single-tree baselines, and in some settings close to ensemble methods, while preserving the compactness and interpretability associated with a single tree. This suggests that the benefit of oblique hinge splits is not only statistical but also structural, since the gains are repeatedly coupled with shallower trees and lower leaf counts.

6. HRT-Boost and the place of HRT within compact tabular modeling

HRT-Boost extends the HRT base learner to a squared-loss functional-gradient-boosting framework. The ensemble is defined as

$\{\tilde x:\;\ell_{t1}(x)=\ell_{t2}(x)\}.$8

where each $\{\tilde x:\;\ell_{t1}(x)=\ell_{t2}(x)\}.$9 is an HRT of prescribed max-depth. At stage $t$00, the pseudo-residuals coincide with the ordinary residuals,

$t$01

and the new base learner is obtained by

$t$02

using the same node-wise Newton procedure as the single-tree method. The stage-wise update is

$t$03

This construction is accompanied by a stage-wise empirical risk reduction guarantee. If

$t$04

then

$t$05

and recursively

$t$06

under squared loss (Li et al., 22 May 2026).

Empirically, HRT-Boost is compared with RF, Scikit-GBM, AdaBoost, XGBoost, LightGBM, TabNet, and TabM on 12 tasks. It is reported to achieve the lowest RMSE in 6/12 cases and to remain within 1–2% of the best on the others. The total leaf count of HRT-Boost, 200–3 000 leaves, is often 50–80% smaller than XGBoost or LightGBM, reported as 600–12 000 leaves. Inference FLOPs per sample are reported as 3 000–16 000 for HRT-Boost versus 30 000–80 000 for TabNet or TabM, while training-time FLOPs remain in the same order as RF and standard boosting (Li et al., 22 May 2026).

This extension clarifies the role of HRT in the broader tabular-modeling landscape. HRT itself is a single-tree learner whose distinguishing property is Newton-optimized oblique splitting. HRT-Boost reuses that learner inside a stage-wise additive ensemble. A plausible implication is that the same hinge-based node formulation supports two distinct design objectives: compact single-tree models with strong approximation guarantees, and compact boosted ensembles that retain a favorable accuracy–complexity trade-off. The code for the boosting framework is stated to be publicly available at the repository given in the paper abstract (Li et al., 22 May 2026).