A General Framework for Decision Trees via Bregman Divergences

Abstract: Decision trees are one of the fundamental tools in statistical learning due to their interpretability, flexibility, and their ability to adapt to nonlinear structures. Among them, the Classification and Regression Trees, introduced by Breiman, Friedman, Olshen, and Stone in 1984, became one of the most influential algorithms and remains one of the most widely used methods for classification and regression problems. On the other hand, Bregman divergences, introduced by Lev Bregman in 1967 in the context of convex optimization, provide a broad family of loss functions that naturally generalize the squared Euclidean distance. This family includes, among others, the Kullback-Leibler divergence, the Poisson divergence, and the Itakura-Saito divergence, as well as several losses associated with distributions belonging to the exponential family. Moreover, Bregman divergences possess a rich geometric structure and deep connections with convex analysis and information geometry. In this work, we propose a generalization of the CART paradigm based on Bregman divergences, thereby obtaining a broader family of decision trees adapted to different statistical models and underlying geometries. Although algorithms such as CART or classical implementations such as rpart incorporate different impurity criteria, these are usually introduced in an ad hoc manner for each specific model. In contrast, the Bregman divergence approach provides a unified framework that allows these criteria to be derived and interpreted from common convex and geometric principles. Beyond the algorithmic construction, we also investigate theoretical properties of these trees. In particular, we study how properties of the generating convex function -- such as strong convexity or smoothness -- influence impurity gains between parent and child nodes, as well as stability and consistency properties of the estimator.

• The paper presents a unified convex-geometric approach by defining decision tree splits using Bregman divergences, subsuming classical impurity measures.
• Empirical evaluations demonstrate that Bregman Trees often outperform standard CART, with notable improvements in deviance measures and mean squared error under diverse distributions.
• The framework offers robust theoretical guarantees, including nonnegative split gain, consistency, and explicit lower bounds, which enhance model interpretability and selection.

A Unified Convex-Geometric Framework for Decision Trees: Bregman Divergences in Recursive Partitioning

Introduction

The proposition of decision trees as a nonparametric, interpretable tool for statistical learning is well-established through the influential CART methodology. However, impurity criteria in classical tree algorithms are traditionally selected in an ad hoc, model-specific fashion, lacking a unified principled foundation. The framework developed in "A General Framework for Decision Trees via Bregman Divergences" (2606.13984) addresses this limitation by systematically generalizing impurity measures in tree-based learning to the family of Bregman divergences—a class of statistical discrepancies underpinned by convex analysis and deeply connected to exponential family models. This establishes an invariant geometric and analytic basis, providing rich theoretical links, practical implications in terms of model adaptation, and empirical advantages.

Bregman Divergences: Convex Analytic Foundation

Bregman divergences, defined from strictly convex, differentiable generators $\psi$, encapsulate many key loss functions: squared error, Kullback–Leibler, Poisson, Itakura–Saito, etc. Specifically:

$$D_\psi(x, y) = \psi(x) - \psi(y) - \langle \nabla \psi(y), x - y \rangle$$

For $\psi(x) = x^2$, $D_\psi$ reduces to the Euclidean distance; for $\psi(\mathbf{p}) = \sum_i p_i \log p_i$, it yields the KL divergence. The geometry is visualized by the gap between the function and its tangent at a reference, as illustrated for both quadratic and Poisson cases.

Figure 1: Examples of Bregman divergences—quadratic (left), Poisson (right)—visualized as the gap between a convex function and its tangent at the reference.

Crucially, Bregman divergences do not require symmetry or the triangle inequality, but they are always non-negative and zero iff arguments coincide. From a probabilistic perspective, Bregman divergences naturally align with maximum likelihood estimation in exponential family models.

These divergences are fundamentally linked to Legendre convex conjugacy. The convex conjugate $\psi^*$ plays a central role in expressing dual parameterizations (natural vs mean), with $D_{\psi^*}(x, y)$ providing the divergence in the mean-parameter space. Figure 2 outlines the mechanics of the Legendre transform.

Figure 2: Convex Legendre conjugate, mapping the graph of $\psi$ to its dual via supremizing over affine shifts.

Decision Trees and Bregman Divergences

Classical decision tree impurities—Gini, entropy, variance—have latent interpretations as Bregman divergences, but previous algorithms instantiated these in an uncoordinated manner. The present framework elevates Bregman divergence as the primitive impurity, and thus the split gain at every node is interpreted as the Jensen gap:

$$\Delta_\phi(t) = p_{L} D_\phi(\mu_L, \mu_t) + p_{R} D_\phi(\mu_R, \mu_t)$$

where $\phi = \psi^*$ is the convex generator, and $$D_\psi(x, y) = \psi(x) - \psi(y) - \langle \nabla \psi(y), x - y \rangle$$0, $$D_\psi(x, y) = \psi(x) - \psi(y) - \langle \nabla \psi(y), x - y \rangle$$1, $$D_\psi(x, y) = \psi(x) - \psi(y) - \langle \nabla \psi(y), x - y \rangle$$2 are the (possibly multi-dimensional) means within the parent and each child, weighted by cardinalities.

This perspective unifies existing impurity criteria as follows:

• Variance reduction is induced by $$D_\psi(x, y) = \psi(x) - \psi(y) - \langle \nabla \psi(y), x - y \rangle$$3, yielding classical ANOVA splitting.
• Cross-entropy/entropy is generated by $$D_\psi(x, y) = \psi(x) - \psi(y) - \langle \nabla \psi(y), x - y \rangle$$4.
• Poisson deviance corresponds to $$D_\psi(x, y) = \psi(x) - \psi(y) - \langle \nabla \psi(y), x - y \rangle$$5.

The framework is therefore inherently compatible with any exponential family for which the canonical link is available, as the divergence and impurity are defined by the Legendre duals of the cumulant function ($$D_\psi(x, y) = \psi(x) - \psi(y) - \langle \nabla \psi(y), x - y \rangle$$6).

Theoretical Properties of Bregman Trees

The paper establishes several kernel theoretical results:

1. Split Gain Nonnegativity and Lower Bound: For any split, impurity reduction is always non-negative by convexity of $$D_\psi(x, y) = \psi(x) - \psi(y) - \langle \nabla \psi(y), x - y \rangle$$7. Moreover, if $$D_\psi(x, y) = \psi(x) - \psi(y) - \langle \nabla \psi(y), x - y \rangle$$8 is $$D_\psi(x, y) = \psi(x) - \psi(y) - \langle \nabla \psi(y), x - y \rangle$$9-strongly convex, the gain is lower-bounded by the squared norm of the child means' separation:

$\psi(x) = x^2$0

1. Jensen Gap Characterization: The impurity reduction for a split is precisely the Jensen gap of $\psi(x) = x^2$1 evaluated on the mixture of child means, giving a variational interpretation of split optimality.
2. Consistency: Under additional assumptions on the smoothness of $\psi(x) = x^2$2 and the shrinking of node diameters, the Bregman tree estimator is universally risk-consistent in the sense that the Bregman excess risk vanishes, i.e.,

$\psi(x) = x^2$3

where the population risk is defined via the mean Bregman divergence between predictions and true conditional response.

These properties generalize classical results for tree regressors to arbitrary convex geometry, governed by the chosen Bregman divergence.

Empirical Evaluation and Numerical Results

The framework is empirically validated via comprehensive simulation studies spanning Exponential, Gamma, Inverse Gaussian, Poisson, and Gaussian generative models. The protocol contrasts standard CART (using quadratic loss or Poisson deviance) with Bregman Trees tailor-adapted to the data's exponential family by selecting the corresponding divergence as the impurity.

• On the Exponential and Gamma models, Bregman Trees (using the Itakura–Saito divergence) yield substantive improvements in both distribution-appropriate deviance and MSE, e.g., reducing mean IS deviance by approximately $\psi(x) = x^2$4 and $\psi(x) = x^2$5, respectively, while also outperforming standard CART on MSE despite not optimizing for it directly.
• Advantage under the Inverse Gaussian (using its canonical divergence) is moderate, reflecting increased tail risk.
• As a negative control, on Poisson and Gaussian data where the classical CART and Bregman Tree optimize the exact same criterion, no significant differences are observed, confirming theoretical predictions.

Results are succinctly captured in box plots contrasting empirical distributions of deviance and MSE for both methods:

Figure 3: Empirical distribution of test deviance and MSE for Exponential data; Bregman Trees consistently yield lower IS deviance.

Figure 4: Empirical results on the Poisson setting, where both algorithms coincide in loss, demonstrating nearly identical performance.

Application to the real-world insurance cost dataset further corroborates the framework. The Bregman Tree, optimizing the Gamma-appropriate IS divergence, yields the lowest out-of-sample IS deviance among all models evaluated, and offers a favorable trade-off between parsimony (number of leaves) and predictive accuracy across three metrics (IS deviance, Poisson deviance, MSE).

Implications and Future Directions

The Bregman Tree methodology enables principled adaptation of decision trees to the statistical geometry induced by the data's generative process—especially crucial for distributions with non-constant variance or asymmetric error, e.g., Gamma, Exponential, or Inverse Gaussian outcomes. This generalization subsumes classical variants as special cases and offers flexibility for new loss functions as novel data modalities arise. The explicit dependence of split gain and risk behavior on the generator’s convexity and smoothness facilitates precise theoretical characterization, such as convergence rates and stability analysis.

Future research directions opened by this framework include (i) formal convergence rates beyond consistency, (ii) extension to classification with non-standard impurity, (iii) ensemble models (random forests, boosting) using non-Euclidean divergences as aggregation criteria, and (iv) integration with optimal transport tools exploiting the Bregman-Wasserstein metric. The approach could also augment model selection procedures and foster interpretability in complex models by aligning impurity with inferential or operational objectives.

Conclusion

A fully general framework for decision tree construction via Bregman divergences is established, unifying diverse impurity measures, providing geometric and statistical insight, and delivering empirical gains—particularly under model misspecification—when classical CART is inapplicable. The proposed formalism resolves longstanding ad hoc choices in impurity selection and grounds all splitting and pruning procedures in convex geometric principles, thereby extending the scope and robustness of tree-based statistical learning.