Chebyshev Model Tree for Regression

• Chebyshev model tree is a hybrid model that combines decision tree partitioning with local Chebyshev polynomial regressors to capture regime-specific behavior.
• It scales features to [-1,1] within each leaf and applies ridge regression for numerical stability and reduced overfitting.
• The model yields piecewise smooth predictions that balance discrete regime detection with smooth local fits, making it effective for non-continuous targets.

Searching arXiv for the cited paper to ground the article. arxiv_search.query({"search_query":"id:(Gerber et al., 25 Feb 2026)","max_results":5}) A Chebyshev model tree is a model tree for tabular regression whose leaves contain ridge-regularised Chebyshev polynomial regressors. In the formulation introduced in "Revisiting Chebyshev Polynomial and Anisotropic RBF Models for Tabular Regression" (Gerber et al., 25 Feb 2026), a standard decision tree discovers regimes or discontinuities through axis-aligned splits, while each leaf models the response smoothly as a low-degree Chebyshev polynomial in features that are scaled locally to $[-1,1]$. The resulting predictor is therefore piecewise smooth: it is smooth within each leaf region, but different leaf polynomials need not match across region boundaries.

1. Formal definition and model class

The global predictor is defined on a partition of the input space $\mathcal{X}\subset\mathbb{R}^d$ induced by a decision tree regressor. If $\{R_\ell\}_{\ell=1}^L$ denotes the leaf regions, then the model takes the form

$$f(\mathbf{x}) = f_\ell(\mathbf{x}) \quad\text{for}\quad \mathbf{x}\in R_\ell,$$

with leafwise regression functions

$$f_\ell(\mathbf{x}) = \beta_{\ell,0} + \sum_{j=1}^{d} \sum_{k=1}^{c} \beta_{\ell,j,k}\, T_k\!\big(x_j^{(\text{scaled})}\big).$$

Here, $c$ is the per-leaf Chebyshev degree, and each feature is scaled within the leaf to $[-1,1]$ using that leaf’s training subset. The tree component is governed by max_depth and min_samples_leaf, while the leafwise polynomial component is governed by complexity and alpha (Gerber et al., 25 Feb 2026).

This construction places the Chebyshev model tree between two familiar classes. Relative to ordinary regression trees, it replaces piecewise-constant leaf predictions with smooth local regressors. Relative to a global polynomial regressor, it allows regime-specific fits after data-dependent partitioning. The paper explicitly frames it as a smooth-tree hybrid that combines “the tree's capacity for detecting regime boundaries (thresholds, interaction effects, discontinuities) with the smooth local fits provided by Chebyshev polynomials.”

A common misconception is to treat the model as globally smooth because its leaves are polynomial. That is incorrect in the paper’s formulation. The model is only piecewise smooth: within each region the predictor is infinitely differentiable in the inputs, but at leaf boundaries there can be kinks or discontinuities because adjacent leaf models are fit independently.

2. Chebyshev basis and leafwise regression

The leaf models use Chebyshev polynomials of the first kind $T_n(x)$, defined on $[-1,1]$ by the recurrence

$$T_0(x)=1,\qquad T_1(x)=x,\qquad T_{n+1}(x)=2x\,T_n(x)-T_{n-1}(x).$$

These polynomials form an orthogonal basis on $\mathcal{X}\subset\mathbb{R}^d$0 and, as stated in the paper, have much better numerical conditioning than monomials $\mathcal{X}\subset\mathbb{R}^d$1, especially at moderate degrees. The numerical-conditioning argument is central to the model’s design because it enables stable polynomial fitting without the coefficient pathologies often associated with monomial expansions (Gerber et al., 25 Feb 2026).

Given $\mathcal{X}\subset\mathbb{R}^d$2 and degree $\mathcal{X}\subset\mathbb{R}^d$3, the leaf learner builds a feature map $\mathcal{X}\subset\mathbb{R}^d$4 with components

$\mathcal{X}\subset\mathbb{R}^d$5

This yields $\mathcal{X}\subset\mathbb{R}^d$6 basis functions, with $\mathcal{X}\subset\mathbb{R}^d$7 included only for the first feature to avoid redundant intercepts. In the stand-alone ChebyPoly regressor, one may optionally add pairwise interaction terms $\mathcal{X}\subset\mathbb{R}^d$8 and Chebyshev expansions on those products. The Chebyshev model tree omits these interaction terms in the leaves in order “to reduce overfitting risk and to keep the per-leaf basis size tractable.”

The leafwise regression problem is solved by ridge regression in the expanded basis: $\mathcal{X}\subset\mathbb{R}^d$9 where $\{R_\ell\}_{\ell=1}^L$0 is the ridge penalty. Once the basis is constructed, the fit is a single linear solve. Before basis evaluation, each feature is linearly scaled to $\{R_\ell\}_{\ell=1}^L$1 by min–max scaling; inputs outside the training range are clipped (clip_input) for stability. For extrapolation, predictions are clipped to $\{R_\ell\}_{\ell=1}^L$2, where the range and $\{R_\ell\}_{\ell=1}^L$3 are computed from training targets.

The omission of cross-feature interactions at the leaf level is structurally important. It means that the model’s flexibility comes from the combination of tree routing and per-feature univariate Chebyshev expansions, rather than from a large multivariate polynomial basis inside each leaf. This suggests a deliberate bias toward low-variance local models.

3. Training procedure, hyperparameters, and implementation

The paper does not provide explicit pseudocode, but it describes a three-stage conceptual procedure. First, a standard decision tree regressor is fit to the training data using standard regression-tree splitting, with axis-aligned thresholds $\{R_\ell\}_{\ell=1}^L$4, controlled by max_depth and min_samples_leaf. Second, for each leaf $\{R_\ell\}_{\ell=1}^L$5, the training points landing in that leaf are collected. If a leaf has too few samples, no polynomial is fitted; instead, the model falls back to the tree’s constant leaf prediction. The paper states: “Leaves with insufficient samples fall back to the tree's own constant prediction to avoid overfitting degenerate local fits.” Otherwise, each feature is scaled to $\{R_\ell\}_{\ell=1}^L$6 within the leaf, a Chebyshev Vandermonde matrix is built up to degree complexity, and ridge regression with penalty alpha is fitted on the leaf data. Third, prediction proceeds by traversing the tree to the appropriate leaf and then evaluating that leaf’s fitted polynomial or constant value (Gerber et al., 25 Feb 2026).

The benchmark search space exposes four principal hyperparameters. The leaf degree complexity is searched over $\{R_\ell\}_{\ell=1}^L$7. The ridge strength alpha is searched over $\{R_\ell\}_{\ell=1}^L$8. Tree depth uses $\{R_\ell\}_{\ell=1}^L$9. The minimum leaf size is specified as a fraction of training samples, with $$f(\mathbf{x}) = f_\ell(\mathbf{x}) \quad\text{for}\quad \mathbf{x}\in R_\ell,$$0. The authors recommend using lower degree than global ChebyPoly because each leaf sees only a subset of the data, and they recommend omitting interactions inside leaves.

Implementation is provided in the poly_basis_ml package as a scikit-learn–compatible estimator with fit and predict methods. The implementation uses scikit-learn’s tree machinery, NumPy’s Chebyshev routines via numpy.polynomial.chebyshev, and ridge regression from scikit-learn. In the benchmark, hyperparameters are tuned with Optuna under the same preprocessing pipeline and nested cross-validation used for all models.

Typical selected configurations indicate that the fitted model is usually shallow and locally simple rather than deeply partitioned or polynomially elaborate. Appendix Table 15 reports median winning settings of complexity $$f(\mathbf{x}) = f_\ell(\mathbf{x}) \quad\text{for}\quad \mathbf{x}\in R_\ell,$$1 with IQR $$f(\mathbf{x}) = f_\ell(\mathbf{x}) \quad\text{for}\quad \mathbf{x}\in R_\ell,$$2–$$f(\mathbf{x}) = f_\ell(\mathbf{x}) \quad\text{for}\quad \mathbf{x}\in R_\ell,$$3, max_depth $$f(\mathbf{x}) = f_\ell(\mathbf{x}) \quad\text{for}\quad \mathbf{x}\in R_\ell,$$4 with IQR $$f(\mathbf{x}) = f_\ell(\mathbf{x}) \quad\text{for}\quad \mathbf{x}\in R_\ell,$$5–$$f(\mathbf{x}) = f_\ell(\mathbf{x}) \quad\text{for}\quad \mathbf{x}\in R_\ell,$$6, min_samples_leaf $$f(\mathbf{x}) = f_\ell(\mathbf{x}) \quad\text{for}\quad \mathbf{x}\in R_\ell,$$7 with IQR $$f(\mathbf{x}) = f_\ell(\mathbf{x}) \quad\text{for}\quad \mathbf{x}\in R_\ell,$$8–$$f(\mathbf{x}) = f_\ell(\mathbf{x}) \quad\text{for}\quad \mathbf{x}\in R_\ell,$$9, and alpha $$f_\ell(\mathbf{x}) = \beta_{\ell,0} + \sum_{j=1}^{d} \sum_{k=1}^{c} \beta_{\ell,j,k}\, T_k\!\big(x_j^{(\text{scaled})}\big).$$0 with a wide range $$f_\ell(\mathbf{x}) = \beta_{\ell,0} + \sum_{j=1}^{d} \sum_{k=1}^{c} \beta_{\ell,j,k}\, T_k\!\big(x_j^{(\text{scaled})}\big).$$1–$$f_\ell(\mathbf{x}) = \beta_{\ell,0} + \sum_{j=1}^{d} \sum_{k=1}^{c} \beta_{\ell,j,k}\, T_k\!\big(x_j^{(\text{scaled})}\big).$$2. The fitted trees have a median of $$f_\ell(\mathbf{x}) = \beta_{\ell,0} + \sum_{j=1}^{d} \sum_{k=1}^{c} \beta_{\ell,j,k}\, T_k\!\big(x_j^{(\text{scaled})}\big).$$3 leaves, median $$f_\ell(\mathbf{x}) = \beta_{\ell,0} + \sum_{j=1}^{d} \sum_{k=1}^{c} \beta_{\ell,j,k}\, T_k\!\big(x_j^{(\text{scaled})}\big).$$4 parameters per leaf, and median $$f_\ell(\mathbf{x}) = \beta_{\ell,0} + \sum_{j=1}^{d} \sum_{k=1}^{c} \beta_{\ell,j,k}\, T_k\!\big(x_j^{(\text{scaled})}\big).$$5 total parameters summed across leaves.

4. Motivation: piecewise smoothness, generalisation, and surrogate use

The model is motivated by a tension between two families of tabular regressors. Tree ensembles such as Random Forest and XGBoost are strong on raw predictive accuracy, but their prediction surfaces are piecewise constant along splits and can exhibit large generalisation gaps. Smooth-basis models such as global Chebyshev polynomial regression and anisotropic RBF networks produce continuously differentiable surfaces and often tighter generalisation behaviour, but they may struggle with regime changes, threshold effects, phase transitions, or other piecewise structure. The Chebyshev model tree is introduced to address exactly this compromise (Gerber et al., 25 Feb 2026).

The paper links smoothness to generalisation through the background claim that smoothness in function approximators tends to improve algorithmic stability and generalisation, citing the connection between bounded Lipschitz behaviour and reduced sensitivity to small perturbations of the training data. ChebyTree does not provide global smoothness, but it does provide smooth local mappings within each leaf. Empirically, the authors report that this yields smaller generalisation gaps than tree ensembles when predictive accuracy is comparable.

The intended downstream use cases further clarify the design. For surrogate optimisation, sensitivity analysis, and simulation-derived regression problems, ordinary tree ensembles can be problematic because their stepwise surfaces encourage optimisers to chase split artefacts. A Chebyshev model tree instead offers differentiable leaf models, so gradients are well defined within regions even though they are not continuous across all boundaries. A plausible implication is that the model is most appropriate when the response is neither globally smooth nor purely discontinuous, but instead exhibits smooth local variation separated by structural breaks.

5. Empirical performance on 55 regression datasets

The benchmark evaluates eight models across 55 regression datasets: Ridge, single Decision Tree, Random Forest, XGBoost, Chebyshev Polynomial, anisotropic RBF, Chebyshev Model Tree, and TabPFN. Using adjusted $$f_\ell(\mathbf{x}) = \beta_{\ell,0} + \sum_{j=1}^{d} \sum_{k=1}^{c} \beta_{\ell,j,k}\, T_k\!\big(x_j^{(\text{scaled})}\big).$$6, TabPFN ranks first on accuracy across a majority of datasets. Among CPU-viable models, however, ERBF, ChebyTree, XGB, ChebyPoly, and RF form a statistically tied group on average rank. In the full comparison including TabPFN, ChebyTree has mean rank $$f_\ell(\mathbf{x}) = \beta_{\ell,0} + \sum_{j=1}^{d} \sum_{k=1}^{c} \beta_{\ell,j,k}\, T_k\!\big(x_j^{(\text{scaled})}\big).$$7, with ERBF at $$f_\ell(\mathbf{x}) = \beta_{\ell,0} + \sum_{j=1}^{d} \sum_{k=1}^{c} \beta_{\ell,j,k}\, T_k\!\big(x_j^{(\text{scaled})}\big).$$8 and XGB at $$f_\ell(\mathbf{x}) = \beta_{\ell,0} + \sum_{j=1}^{d} \sum_{k=1}^{c} \beta_{\ell,j,k}\, T_k\!\big(x_j^{(\text{scaled})}\big).$$9. Excluding TabPFN, Table 10 gives mean ranks on $c$0 of ERBF $c$1 with $c$2 wins, ChebyTree $c$3 with $c$4 wins, XGB $c$5 with $c$6 wins, ChebyPoly $c$7, and RF $c$8; all five are statistically indistinguishable on accuracy by Nemenyi (Gerber et al., 25 Feb 2026).

The generalisation-gap analysis is one of the clearest empirical differentiators. The paper defines the gap as $c$9, where lower is better. ChebyTree has mean gap rank $[-1,1]$0 and median gap $[-1,1]$1, compared with ChebyPoly at $[-1,1]$2 and $[-1,1]$3, ERBF at $[-1,1]$4 and $[-1,1]$5, RF at $[-1,1]$6 and $[-1,1]$7, TabPFN at $[-1,1]$8 and $[-1,1]$9, and XGB at $T_n(x)$0 and $T_n(x)$1. The paper also reports a matched-accuracy analysis in which, for dataset-model pairs satisfying $T_n(x)$2, the smooth model has the tighter gap in $T_n(x)$3 of comparisons; ChebyTree contributes to that smooth-model group.

Domain-wise results sharpen the interpretation. In the four dataset strata, ChebyTree is second among CPU models in S1 engineering/simulation with mean rank $T_n(x)$4, best CPU model in S2 behavioural/social with $T_n(x)$5, competitive in S3 physics/chemistry/life sciences with $T_n(x)$6, and second CPU model in S4 economics/pricing with $T_n(x)$7. The paper’s interpretation is that ERBF and ChebyPoly tend to lead in smoother physical-process domains, XGB tends to slightly lead in threshold-driven economic settings, and ChebyTree is particularly effective in behavioural or social data with more discrete or step-like structure.

That same pattern appears in the continuous versus non-continuous target analysis. On continuous targets, mean ranks are ERBF $T_n(x)$8, XGB $T_n(x)$9, ChebyTree $[-1,1]$0, and ChebyPoly $[-1,1]$1. On non-continuous targets, ChebyTree becomes the best model with mean rank $[-1,1]$2, followed by ChebyPoly $[-1,1]$3, XGB $[-1,1]$4, and ERBF $[-1,1]$5. The paper offers a specific architectural explanation: ERBF’s localised Gaussian-like basis functions match smooth targets but are a poor match for step-like structure, whereas ChebyTree’s routing partitions the feature space into regions of roughly homogeneous target behaviour and lets smooth leaf polynomials model within-region variation.

Computationally, ChebyTree is positioned as a strong CPU-side model. Table 13 reports mean tuning time of $[-1,1]$6 s under nested CV with Optuna, compared with $[-1,1]$7 s for ChebyPoly, $[-1,1]$8 s for XGB, $[-1,1]$9 s for ERBF, and $$T_0(x)=1,\qquad T_1(x)=x,\qquad T_{n+1}(x)=2x\,T_n(x)-T_{n-1}(x).$$0 s for RF. Prediction time per $$T_0(x)=1,\qquad T_1(x)=x,\qquad T_{n+1}(x)=2x\,T_n(x)-T_{n-1}(x).$$1 instances is $$T_0(x)=1,\qquad T_1(x)=x,\qquad T_{n+1}(x)=2x\,T_n(x)-T_{n-1}(x).$$2 ms for ChebyTree, $$T_0(x)=1,\qquad T_1(x)=x,\qquad T_{n+1}(x)=2x\,T_n(x)-T_{n-1}(x).$$3 ms for XGB, $$T_0(x)=1,\qquad T_1(x)=x,\qquad T_{n+1}(x)=2x\,T_n(x)-T_{n-1}(x).$$4 ms for ERBF, $$T_0(x)=1,\qquad T_1(x)=x,\qquad T_{n+1}(x)=2x\,T_n(x)-T_{n-1}(x).$$5 ms for ChebyPoly, $$T_0(x)=1,\qquad T_1(x)=x,\qquad T_{n+1}(x)=2x\,T_n(x)-T_{n-1}(x).$$6 ms for RF, and $$T_0(x)=1,\qquad T_1(x)=x,\qquad T_{n+1}(x)=2x\,T_n(x)-T_{n-1}(x).$$7 ms for TabPFN. On eight full-scale datasets with up to $$T_0(x)=1,\qquad T_1(x)=x,\qquad T_{n+1}(x)=2x\,T_n(x)-T_{n-1}(x).$$8 and $$T_0(x)=1,\qquad T_1(x)=x,\qquad T_{n+1}(x)=2x\,T_n(x)-T_{n-1}(x).$$9, mean $\mathcal{X}\subset\mathbb{R}^d$00 is $\mathcal{X}\subset\mathbb{R}^d$01 for ChebyTree versus $\mathcal{X}\subset\mathbb{R}^d$02 for XGB, indicating near-parity at larger scale without feature selection or subsampling.

6. Relation to adjacent models and practical deployment

Relative to ChebyPoly, the Chebyshev model tree trades a single global smooth surface for multiple lower-degree local surfaces routed by a tree. ChebyPoly can optionally include interactions and may exploit all data more effectively when the target is globally smooth and continuous. ChebyTree instead addresses thresholded or piecewise structure and, on the paper’s non-continuous subset, improves over ChebyPoly by a mean-rank shift of $\mathcal{X}\subset\mathbb{R}^d$03. The empirical trade-off is explicit: ChebyPoly has the smallest generalisation gap among the competitive models, while ChebyTree is more robust to regime changes (Gerber et al., 25 Feb 2026).

Relative to anisotropic RBF (ERBF), ChebyTree is less purely smooth but more tolerant of discrete or step-like targets. ERBF performs best on the continuous subset and tends to lead in smooth engineering and physics settings. ChebyTree, however, is clearly better on non-continuous targets and remains competitive overall. Relative to tree ensembles, especially XGB and RF, ChebyTree is statistically tied on accuracy, exhibits a much smaller gap than XGB and a somewhat better gap than RF, and provides piecewise-smooth rather than piecewise-constant predictions. Relative to TabPFN, ChebyTree is not the top-accuracy method overall, but it is a CPU-friendly alternative when GPU dependence, inference latency, or the $\mathcal{X}\subset\mathbb{R}^d$04 dataset-size limit of TabPFN constrain deployment.

The paper’s practical guidance recommends ChebyTree when the regression function plausibly has regime structure or thresholds together with smooth variation within regimes, when smoother surrogates than RF or XGB are needed, when compute budget is limited, or when targets are ordinal, count-like, or otherwise non-continuous. It recommends more caution when the target is known to be globally smooth and continuous, in which case global ChebyPoly or ERBF may be preferable, or when globally smooth derivatives are required everywhere, which ChebyTree cannot provide because it is only piecewise smooth.

For tuning, the paper suggests starting around max_depth $\mathcal{X}\subset\mathbb{R}^d$05–$\mathcal{X}\subset\mathbb{R}^d$06, min_samples_leaf near $\mathcal{X}\subset\mathbb{R}^d$07, complexity $\mathcal{X}\subset\mathbb{R}^d$08 or $\mathcal{X}\subset\mathbb{R}^d$09, and alpha in $\mathcal{X}\subset\mathbb{R}^d$10, then increasing degree only if within-leaf fits underfit and leaves are large enough. It also warns against simultaneously using high depth and high degree, since this creates many complex local models and tends to overfit. Because leaf models are fitted independently with ridge regression, training time grows roughly linearly in the number of leaves times the number of polynomial terms, which the paper characterises as manageable for moderate depth and degree.

In the paper’s specific sense, the Chebyshev model tree is therefore best understood not as a replacement for either smooth global regression or tree ensembles in all settings, but as a hybrid predictor class for tabular regression problems where smooth local behaviour and explicit regime detection are both materially important.