---
title: 'CatBoost: Unbiased Gradient Boosting'
url: https://www.emergentmind.com/topics/catboost
type: topic
---

# CatBoost: Unbiased Gradient Boosting

CatBoost (Categorical Boosting) is a gradient-boosted decision tree (GBDT) machine learning library designed to deliver unbiased, high-accuracy predictions on heterogeneous datasets containing both numerical and high-cardinality categorical variables. CatBoost’s core algorithmic innovations, notably ordered boosting and permutation-driven target statistics, systematically address target leakage and gradient bias, setting it apart from classical GBDT approaches such as XGBoost and LightGBM. CatBoost supports both CPU and GPU acceleration and has demonstrated state-of-the-art performance and efficiency across domains including astrophysics, insurance, finance, and cybersecurity [1810.11363, 1706.09516, 2211.09492, 2307.07771, 2411.06860, 2106.07954].

## 1. Algorithmic Foundations: Ordered Boosting and Oblivious Trees

The CatBoost learning strategy centers on the minimization of an empirical loss functional
\[
J(F) = \sum_{i=1}^n L(Y_i, F(X_i)),
\]
where \( F \) is a GBDT ensemble composed of a sequence of base learners (trees) fit to pointwise gradients and, in second-order mode, Hessians of the loss [1706.09516, 1810.11363].

Standard GBDT algorithms suffer from “prediction shift”—bias introduced when gradient (and categorical encoding) calculations for each example \( x_k \) rely on models that have already seen \( x_k \) [1706.09516]. CatBoost eliminates this shift through **ordered boosting**:
  - Draws random permutations \( \sigma \) of the data;
  - For each data point at position \( k \) in \( \sigma \), its pre-fit gradient and categorical encoding are computed using only the preceding \( k-1 \) examples;
  - Maintains logarithmic-sized supporting models per permutation to ensure unbiased, “out-of-fold” gradient and encoding computations while retaining \( O(s n) \) per-iteration cost.

CatBoost’s base learners are **oblivious trees**, i.e., symmetric binary trees of fixed depth \( d \), wherein every node at depth \( l \) splits on the same feature and threshold across all paths. This yields extremely efficient evaluation: the path to each leaf corresponds to a unique \( d \)-bit integer and all leaf values are stored in contiguous arrays [1810.11363, 2211.00391].

## 2. Categorical Feature Encoding: Ordered Target Statistics

Handling categorical variables without information leakage is a central challenge. CatBoost encodes categorical features using **ordered target statistics (“CTR” features)** computed as follows, for feature \( k \), category value \( x_{i,k} \), and permutation \( \sigma \) [1706.09516, 1810.11363]:

\[
x'_{\sigma_p, k} = \frac{ \sum_{j=1}^{p-1} [x_{\sigma_j, k} = x_{\sigma_p, k}] \cdot Y_{\sigma_j} + a \cdot P }{ \sum_{j=1}^{p-1} [x_{\sigma_j, k} = x_{\sigma_p, k}] + a }.
\]

Here, \( a \) is a prior-weight and \( P \) the prior (e.g., the global mean). Multiple permutations (usually \( s = 1 \) or \( 2 \)) stabilize the encoding. This approach enables robust use of very high-cardinality categoricals and dynamic greedy feature combinations. The encoding is strictly “out-of-fold” with respect to model fitting and all downstream splits, eliminating target leakage [1706.09516, 1810.11363, 2106.07954].

## 3. Boosting Optimization and Tree Construction

At each iteration, CatBoost fits an oblivious tree to negative gradients (and Hessians for Newton mode). After the tree structure is built, optimal leaf values \( \gamma_\ell \) are set using a regularized Newton step:

\[
\gamma_\ell = - \frac{ \sum_{i \in \ell} g_i }{ \sum_{i \in \ell} h_i + \lambda }
\]

where \( g_i \), \( h_i \) are the loss gradient and Hessian for each sample in leaf \( \ell \) and \( \lambda \) is the L2-leaf regularization parameter [1810.11363, 1706.09516, 2106.07954]. This procedure generalizes to arbitrary convex loss functions including regression, classification, and distributional objectives (see CatBoostLSS below).

Key hyperparameters include: number of trees (iterations), depth, learning rate (\( \eta \)), L2_leaf_reg (\( \lambda \)), feature bin count, random strength, and bagging temperature [1810.11363, 1706.09516]. CatBoost employs Bayesian bootstrap gradient weights and random permutations to enhance regularization.

## 4. System Engineering and Computational Performance

CatBoost implements aggressive optimizations for both CPU and GPU backends [1810.11363, 2211.00391]:
- **CPU Inference:** All features are pre-binarized; leaf index calculation is performed via bitwise operations; oblivious tree structure enables vectorized, branchless code. AVX2 and AVX-512 kernels with FP16 leaf value support accelerate prediction by 20–70% with negligible numerical error [2211.00391].
- **GPU Training:** Histogram-based split search; features packed into compact representations; composite CTR encoding mapped via hashing and processed using sort/scan primitives; multi-GPU training achieved via feature-parallelism [1810.11363].

Quantitatively, CatBoost is 2.6–15× faster than its own CPU scoring, and outperforms XGBoost and LightGBM in both accuracy and runtime, especially at high tree depths or large ensemble sizes [1810.11363, 2211.00391].

## 5. Extensions: Probabilistic Forecasting and Zero-Inflated Modeling

**CatBoostLSS** is an extension for distributional and probabilistic forecasting that directly predicts parameters of a user-specified distribution (e.g., Normal, Poisson, Generalized Beta, Zero-inflated Poisson) [2001.02121]. For K-parameter distributions, CatBoostLSS alternates Newton boosting over each parameter, using the full negative log-likelihood as the loss:

\[
\ell(y; \theta_1,\ldots,\theta_K) = -\ln p(y; \theta_1, \ldots, \theta_K).
\]

Gradients and Hessians for each parameter are propagated and fit with regular boosting rounds and backfitting [2001.02121, 2307.07771].

Zero-inflated Poisson CatBoost models have also been constructed for count data with heavy excess zeros, enabling simultaneous modeling of mean and inflation probability within or across tree ensembles [2307.07771].

## 6. Interpretability and Feature Analysis

CatBoost computes feature importances by aggregating the loss increase when a split uses a given feature, supporting robust variable selection and diagnosis [1706.09516, 1810.11363, 2411.06860]. Support is native for the SHAP (Shapley Additive Explanations) algorithm, which provides consistent local and global interpretation:

- SHAP summary plots and interaction scatter plots expose non-linear and synergistic effects among features, as applied in insurance telematics and phishing detection [2307.07771, 2411.06860, 2211.09492].
- CatBoost’s permutation-based feature importance avoids over-reliance on single variables or overfitting seen in other GBDT models [2411.06860].

## 7. Practical Applications and Benchmarking

CatBoost has been deployed in diverse high-data domains:
- **Astrophysics:** Regression for photometric redshift estimation in large surveys (DESI Legacy, Euclid), yielding σ_NMAD as low as 0.0156, with error fractions under 1%, and outperforming MLP and Random Forest in both accuracy and AUC [2211.09492, 2504.13020].
- **Insurance:** Superior pseudo-R² and deviance for auto claim frequency via zero-inflated Poisson boosting, with advanced SHAP-based interpretation [2307.07771].
- **Finance:** Enhanced loan risk modeling using target-guided synthetic feature generation, with AUC reaching 98.80% [2106.07954].
- **Cybersecurity:** Robust phishing URL detection, maintaining near-perfect accuracy under aggressive feature selection and outperforming XGBoost/EBM in situations with complex feature interactions [2411.06860].

Empirical evaluations across domains confirm that CatBoost outperforms or matches alternate tree-based approaches in most settings, even at default hyperparameters. Notable limitations include degraded prediction quality at out-of-distribution data (e.g., high-redshift galaxies with training set gaps) and the need for careful permutation scaling for massive datasets [2211.09492, 2504.13020]. 

---

**References:**
- CatBoost: gradient boosting with categorical features support [1810.11363]
- CatBoost: unbiased boosting with categorical features [1706.09516]
- Optimization of Oblivious Decision Tree Ensembles Evaluation for CPU [2211.00391]
- CatBoostLSS: An extension of CatBoost to probabilistic forecasting [2001.02121]
- Enhanced Gradient Boosting for Zero-Inflated Insurance Claims… [2307.07771]
- Euclid preparation. Estimating galaxy physical properties using CatBoost… [2504.13020]
- Photometric redshift estimation… DESI Legacy Imaging [2211.09492]
- Enhancing Phishing Detection… CatBoost, XGBoost, and EBM Models [2411.06860]
- Classification of Fermi-LAT unidentified gamma-ray sources using CatBoost… [2207.04725]
- CatBoost model with synthetic features in application to loan risk… [2106.07954]

Source: https://www.emergentmind.com/topics/catboost