Ridge-Regularised Chebyshev Poly Regressor

• The paper introduces a Chebyshev regressor that scales inputs to [-1,1], expands them via Chebyshev polynomials, and fits the design matrix with ridge regression.
• It delivers continuously differentiable prediction surfaces ideal for surrogate optimization and sensitivity analysis while maintaining numerical stability.
• Empirical benchmarks show that the model achieves competitive accuracy and notably tighter generalisation gaps compared to tree ensemble methods.

Searching arXiv for papers on Chebyshev polynomial regression and ridge-regularised Chebyshev models. A ridge-regularised Chebyshev polynomial regressor is a globally smooth linear regressor in an expanded Chebyshev basis in which each input feature is scaled to $[-1,1]$, evaluated through Chebyshev polynomials of the first kind, optionally augmented with pairwise interaction features, and then fit with ridge regression (Gerber et al., 25 Feb 2026). In the benchmark that introduced the model under the name chebypoly, it was positioned as a smooth-basis alternative to tree ensembles for tabular regression, with emphasis on continuously differentiable prediction surfaces, CPU-viable deployment, and tighter generalisation gaps than several competing models at matched accuracy (Gerber et al., 25 Feb 2026).

1. Definition and model class

The regressor is implemented as chebypoly in the poly_basis_ml package and is described as a globally smooth linear regressor in an expanded Chebyshev basis (Gerber et al., 25 Feb 2026). Its construction follows a fixed sequence: scale each input feature to $[-1,1]$, evaluate Chebyshev polynomials of the first kind on the scaled features, optionally add pairwise interaction features, and fit the resulting design matrix with ridge regression. Because the model is linear in the expanded basis coefficients, training reduces to a single regularised least-squares solve.

Within the benchmark, the model belongs to a family of smooth-basis models that also includes an anisotropic RBF network and a smooth-tree hybrid called the Chebyshev model tree (Gerber et al., 25 Feb 2026). The paper’s motivation is that smooth-basis models are well established in numerical analysis and that their continuously differentiable prediction surfaces suit surrogate optimisation, sensitivity analysis, and other settings where the response varies gradually with inputs. At the same time, the paper notes that such smooth models seldom appear in tabular regression, where tree ensembles dominate. The ridge-regularised Chebyshev polynomial regressor is therefore framed not as a replacement for all tabular methods, but as a candidate model class whose numerical and generalisation properties merit routine inclusion in benchmark pipelines.

The paper explicitly contrasts the Chebyshev basis with the monomial basis $(1,x,x^2,\dots)$, stating that Chebyshev polynomials are an orthogonal basis on $[-1,1]$ and are much better conditioned than monomials at comparable degree (Gerber et al., 25 Feb 2026). A common misconception is that polynomial regression in tabular settings is synonymous with a monomial expansion; the model described here instead uses a Chebyshev basis precisely to improve conditioning and practical feasibility at higher degree.

2. Basis construction and expanded feature space

The model uses Chebyshev polynomials of the first kind $T_n$, defined by

$$T_0(x) = 1,\qquad T_1(x) = x,\qquad T_{n+1}(x) = 2x\,T_n(x) - T_{n-1}(x).$$

These form an orthogonal basis on $[-1,1]$ (Gerber et al., 25 Feb 2026).

Before basis expansion, each feature is mapped to $[-1,1]$ with a min-max scaler, and values outside the training range are clipped for stability (Gerber et al., 25 Feb 2026). If $x_j$ denotes the $j$-th raw feature, the scaled value used in the polynomial basis is $[-1,1]$0.

For $[-1,1]$1 features and polynomial degree $[-1,1]$2, called complexity in the paper, the univariate design matrix includes

$[-1,1]$3

This gives $[-1,1]$4 basis terms, with the constant term $[-1,1]$5 included only once to avoid duplicate intercept-like terms (Gerber et al., 25 Feb 2026). The single inclusion of $[-1,1]$6 is a small but explicit structural choice aimed at preventing redundant constant columns in the expanded design matrix.

When include_interactions=True, the basis is augmented with pairwise product interactions of the scaled features,

$[-1,1]$7

The paper explicitly states that these are not cross-polynomials such as $[-1,1]$8, since such terms would create a combinatorial explosion (Gerber et al., 25 Feb 2026). If max_interaction_complexity = 2, each product term is further expanded by appending its Chebyshev evaluation $[-1,1]$9; at the default value $(1,x,x^2,\dots)$0, only the raw products are used. This construction makes the interaction mechanism intentionally limited. A plausible implication is that the model seeks additional cross-feature expressivity without constructing a full tensor-product polynomial basis.

3. Ridge regularisation, preprocessing, and hyperparameter selection

The expanded feature matrix is fit with ridge regression (Gerber et al., 25 Feb 2026). The paper presents the objective as the standard ridge-penalised least-squares problem,

$(1,x,x^2,\dots)$1

or equivalently in matrix form as a ridge problem on the Chebyshev-expanded design. The penalty parameter is $(1,x,x^2,\dots)$2, and ridge regularisation is described as the central control on model complexity: it shrinks coefficient magnitudes in the Chebyshev-expanded feature space and combats overfitting, especially when the polynomial degree is high or interaction terms are enabled.

The paper reports that elastic net was tested but incurred substantial computational overhead from coordinate descent at low $(1,x,x^2,\dots)$3 and did not improve accuracy over ridge; ridge was therefore kept as the regulariser for chebypoly (Gerber et al., 25 Feb 2026). This addresses another common misconception, namely that a richer basis necessarily requires sparse regularisation to be viable. In the reported benchmark, the chosen design instead pairs a well-conditioned orthogonal basis with $(1,x,x^2,\dots)$4 shrinkage.

The model inherits the benchmark’s fold-level preprocessing and also applies its own internal scaling. Relevant benchmark-level steps include median imputation for missing numeric values, target encoding for categorical variables, dataset-level feature filtering that drops features with more than $(1,x,x^2,\dots)$5 missingness, drops quasi-constant features, optionally removes low-correlation features and retains some via mutual information, caps features at 50 via mutual information if needed, and subsamples datasets above 50,000 rows (Gerber et al., 25 Feb 2026). Inside chebypoly itself, features are min-max scaled to $(1,x,x^2,\dots)$6, and out-of-range test values are clipped through clip_input=True to keep basis evaluation numerically stable.

Hyperparameters are selected by nested cross-validation with Optuna (Gerber et al., 25 Feb 2026). The search space for chebypoly consists of complexity in $(1,x,x^2,\dots)$7, alpha in $(1,x,x^2,\dots)$8 on a log scale, include_interactions in $(1,x,x^2,\dots)$9, and max_interaction_complexity in $[-1,1]$0 when interactions are enabled. Across 55 datasets, the median selected degree was 9 with interquartile range 4 to 13, while the median selected $[-1,1]$1 was 0.69 with IQR 0.07 to 2.2. The paper explicitly notes that the high degree is feasible because ridge regularisation stabilises the large basis. This suggests that the practical operating regime of the model is not low-degree polynomial fitting in the classical sense, but relatively expressive basis expansion controlled by shrinkage.

4. Numerical stability and implementation profile

The paper makes several explicit numerical-stability design choices for the regressor (Gerber et al., 25 Feb 2026). First, Chebyshev polynomials are preferred because the monomial basis becomes severely ill-conditioned at moderate degrees, whereas Chebyshev polynomials remain far better conditioned. Second, features are scaled to $[-1,1]$2 and values outside the fitted range are clipped, avoiding basis evaluation far outside its stable domain. Third, the constant $[-1,1]$3 is included only once, preventing redundant intercept-like columns. Fourth, ridge regularisation itself improves numerical stability by making the normal equations better conditioned.

Interaction handling is also constrained for stability and tractability. Only pairwise product features are used; no full tensor-product basis is constructed; and for high dimension $[-1,1]$4, interactions are restricted to the top half of features ranked by variance (Gerber et al., 25 Feb 2026). In the benchmarking protocol, predictions are clipped to $[-1,1]$5 of the training target distribution to suppress extrapolation artefacts that could distort metrics. These design choices delimit the model’s expressivity in a way that is explicitly numerical rather than purely statistical.

Implementation is in Python as part of poly_basis_ml, using NumPy, SciPy, and a scikit-learn-compatible estimator API, with the design matrix built through numpy.polynomial.chebyshev (Gerber et al., 25 Feb 2026). The estimator exposes fit/predict. The paper does not give a single closed-form complexity for chebypoly, but states that basis construction is linear in $[-1,1]$6 times the number of features in the expanded design and that fitting is a ridge regression solve in the expanded space.

The hyperparameter search used 30 Optuna trials per dataset in the nested cross-validation inner loop (Gerber et al., 25 Feb 2026). Among competitive CPU-viable models, chebypoly was reported as one of the cheapest to tune, with mean tune time 37 s, training time 0.33 s, and prediction time 16 ms per 1,000 instances. It is explicitly positioned as a CPU-viable model that does not require GPU inference and runs on commodity CPU hardware.

5. Empirical behaviour in benchmark evaluations

On the full 55-dataset benchmark, chebypoly is presented as competitive among CPU-viable models, though not the top scorer overall (Gerber et al., 25 Feb 2026). Its summary statistics were a mean rank on adjusted $[-1,1]$7 of 4.45 with TabPFN included and 3.55 among CPU-viable models, a mean $[-1,1]$8 of $[-1,1]$9, and a median $T_n$0 of 0.791. Among CPU-viable models, the top cluster was statistically tied and comprised ERBF, ChebyTree, XGBoost, ChebyPoly, and Random Forest.

The broader benchmark conclusion was that the transformer ranked first on accuracy across a majority of datasets, but that its GPU dependence, inference latency, and dataset-size limits constrain deployment in the CPU-based settings common across applied science and industry (Gerber et al., 25 Feb 2026). Against that background, the ridge-regularised Chebyshev regressor is not singled out as the best method in absolute accuracy terms; rather, it is one member of a statistically tied CPU-viable group.

Its most distinctive empirical property is its reported generalisation-gap behaviour. From the gap ranking table, chebypoly had mean gap rank 3.27, described as best among competitive models, with median gap 0.014 and mean gap $T_n$1 (Gerber et al., 25 Feb 2026). The paper reports that smooth models, especially chebypoly, often have tighter generalisation gaps than tree ensembles at matched accuracy. When accuracy is comparable $T_n$2, smooth models beat tree ensembles on gap in 87% of pairwise comparisons overall; specifically, chebypoly versus xgb favored chebypoly on gap in 22/23 datasets, and chebypoly versus rf favored chebypoly in 18/21 datasets. The paper further states that chebypoly has a significant gap advantage over xgb.

Behaviour varies with target type and application stratum. The model does particularly well on non-continuous or discrete-like targets relative to ERBF: its mean rank on non-continuous targets was 2.85, compared with 3.76 on continuous targets (Gerber et al., 25 Feb 2026). The paper suggests this may be because a global polynomial can approximate piecewise structure through oscillation more flexibly than localized Gaussian bumps. In domain strata, chebypoly is described as especially competitive in S2 (behavioural/social), competitive in S3 (physics/chemistry/life sciences), and less dominant in S4 (economics/pricing) than XGBoost or ChebyTree. The model is therefore not presented as uniformly strongest across domains, but as consistently competitive with a distinctive generalisation profile.

6. Relation to broader Chebyshev and ridge-based methodology

The ridge-regularised Chebyshev polynomial regressor belongs to a broader technical landscape in which Chebyshev constructions and ridge subroutines are used for numerical efficiency and stability (Gerber et al., 25 Feb 2026). In tabular regression, the role of the Chebyshev basis is to provide a well-conditioned polynomial expansion on $T_n$3, after which estimation is a regularised linear solve.

A separate line of work uses Chebyshev polynomials in matrix algorithms rather than in feature-basis regression. In “Faster Principal Component Regression and Stable Matrix Chebyshev Approximation” (Allen-Zhu et al., 2016), principal component regression is solved up to a multiplicative accuracy $T_n$4 by reducing the problem to $T_n$5 black-box calls of ridge regression, in contrast to a previous result requiring $T_n$6 such calls. That paper attributes the improvement to a general stable recurrence formula for matrix Chebyshev polynomials and a degree-optimal polynomial approximation to the matrix sign function (Allen-Zhu et al., 2016).

The two settings are distinct: one concerns a tabular regression estimator built from Chebyshev basis expansion, and the other concerns iterative matrix methods for principal component regression. Even so, a methodological connection is visible. This suggests that Chebyshev structures and ridge-based primitives can interact in multiple ways: as a basis-and-regulariser pair in a predictive model, and as approximation-theoretic machinery in large-scale linear-algebraic algorithms. In both cases, stability considerations are central, but the objects being stabilized differ—the expanded feature design in one setting and matrix polynomial recurrences in the other.