---
title: 'GOSDT: Optimal Sparse Decision Trees'
url: https://www.emergentmind.com/topics/generalized-optimal-sparse-decision-trees-gosdt
type: topic
---

# GOSDT: Optimal Sparse Decision Trees

Generalized Optimal Sparse Decision Trees (GOSDT) encapsulate a comprehensive, scalable framework for constructing globally optimal decision trees subject to explicit sparsity constraints. The approach is central to interpretable machine learning, providing models that jointly optimize accuracy and interpretability by explicitly controlling tree complexity through regularization. Recent extensions further enable principled handling of weighted data, which is crucial for settings such as optimal policy design or inverse propensity scoring. GOSDT combines advanced combinatorial optimization—dynamic programming, branch-and-bound pruning, and incremental bound refinement—to efficiently search the space of tree structures under a broad array of objective functions and dataset regimes [2006.08690, 2210.06825].

## 1. Mathematical Formulation and Losses

Given training data $\{(x_i, y_i)\}_{i=1}^N$ with $x_i \in \mathbb{R}^M$ and $y_i \in \{0,1\}$ (or, for weighted settings, per-sample nonnegative weights $w_i$), GOSDT seeks to identify a decision tree $d$ minimizing the regularized risk:
\[
R(d) = \ell(d; x, y) + \lambda H_d,
\]
where $\ell$ is a monotonic loss function (e.g., weighted misclassification, balanced error, $F_1$, AUC), $H_d$ is the number of leaves, and $\lambda \geq 0$ explicitly regularizes model complexity [2006.08690]. The weighted extension generalizes the empirical risk:
\[
\mathcal{L}_w(t) = \frac{1}{\sum_{i=1}^N w_i} \sum_{i=1}^N w_i\,\mathbf{1}\,[y_i \neq \hat{y}_i^t]
\]
with optimization objective
\[
R_w(t) = \mathcal{L}_w(t) + \lambda H_t, \quad \text{s.t.}\ \text{depth}(t) \leq d.
\]
Losses supported include cost-weighted misclassification, balanced error, area under the ROC convex hull (AUC$_{\rm ch}$), and partial AUC constraints, consistently extended for weighted settings [2006.08690, 2210.06825].

## 2. Core Algorithmic Mechanisms

GOSDT’s framework is underpinned by a branch-and-bound architecture that combines dynamic programming over subsets of the training data with priority-queue–driven search and aggressive subproblem pruning [2006.08690]. Each subproblem corresponds to building a subtree for a specific support set of samples, efficiently cached using bit-vector or hash-based representations. Multiple lower and upper bounds—hierarchical objective bounds, one-step lookahead, leaf-permutation (symmetry), equivalent-points, and incremental similar-support bounds—guide the pruning strategy and accelerate convergence.

In the context of continuous variables, GOSDT eschews pre-bucketing by constructing binary features for all candidate thresholds, with sub-exponential pruning in practice due to the incremental similar-support bound. For weighted data, GOSDT admits three complementary extensions:
- **Direct branch-and-bound with weighted loss**: Fully supports weighted objectives but introduces $O(N)$ cost per node due to general dot-product operations [2210.06825].
- **Integer-weight transformation (data duplication)**: Approximates real weights with integer scaling and sample duplication, preserving bit-operation speedups and providing provably small approximation error.
- **Randomized weighted sampling**: Draws a subsample with probabilities proportional to $w_i/\sum_j w_j$, maintaining unweighted GOSDT optimizations while bounding risk deviation via Hoeffding’s inequality [2210.06825].

## 3. Scalability, Complexity, and Implementation

The worst-case time complexity is combinatorial (for $M$ features, $\sum_{k=0}^M 2^k\binom{M}{k}k! = O(2^M M!)$), but core innovations—memorization via support-set hashing, prefix-sum bitvector computations, lower/upper bounds propagation, and parallelization—yield dramatically superior empirical performance for moderate-to-large $N$ and $M$ [2006.08690]. Bitwise operations enable sublinear scaling in practice for common settings. The branch-and-bound process is guaranteed to converge either with an exact optimum or a certified optimality gap.

For weighted GOSDT, the integer-weight transformation yields dataset expansion to $M \leq pN$, where $p$ is the integer scaling factor; practical runtimes are often two orders of magnitude faster than direct weighted optimization. Weighted sampling scales to very large datasets, with an error bound for empirical risk difference that decays exponentially in sample size [2210.06825].

## 4. Integration of Diverse Objectives and Policy Design

GOSDT is designed to optimize not just classic misclassification error but any loss monotonic in false positives and false negatives. This includes:
- Weighted accuracy: explicit cost for false negatives via parameter $\omega$.
- Balanced accuracy: averaging class-conditional errors.
- $F_1$-loss and AUC: via leaf-level statistics and leaf-label assignments.
- Partial AUC: restricting attention to leaves corresponding to low false-positive regions.

The explicit incorporation of per-sample weights enables optimal policy design, such as inverse propensity–weighted treatment rules in causal inference and cost-sensitive policy learning [2210.06825]. This cannot be realized by traditional discrete-loss, unweighted optimal tree methods.

## 5. Comparative Performance and Empirical Evaluation

Extensive experiments demonstrate that GOSDT and its weighted extensions attain higher accuracy at fixed sparsity than greedy (CART), mixed-integer programming (MIP-based ODT), and itemset DP (DL8.5, BinOCT) baselines. For example, on datasets such as FICO ($N \approx 10^4$), the data-duplication and sampling approaches achieve near-optimal solutions ($<1\%$ objective gap) in under a minute versus hours for direct weighted optimization. GOSDT dominates the test accuracy versus complexity Pareto frontier, especially when compared at fixed leaf-counts [2210.06825, 2006.08690].

| Algorithm         | Objective Support | Handles Weighting | Practical Scaling |
|-------------------|------------------|-------------------|------------------|
| GOSDT             | monotonic losses | Yes (extensions)  | $\sim 10^4$–$10^5$ samples |
| OSDT/DL8.5/BinOCT | 0–1 loss         | No                | Varies           |
| MIP ODT           | Flexible         | Yes (IP)          | Small $N$, small trees |

## 6. Extensions, Limitations, and Theoretical Guarantees

All weighted GOSDT variants inherit the core framework’s guarantees: convergence to the (possibly approximate) global optimum, certification of optimality, and tree interpretability. Integer-weight duplication yields a proven bound on error relative to the optimal weighted objective, with control via the scaling parameter and empirical weight discrepancy [2210.06825]. Weighted-sampling methods guarantee that, for any fixed tree, empirical risk on the sample is close to the true weighted risk with high probability.

Limitations include exponential scaling in the number of effective binary features and sample equivalence classes; memory usage may grow with increased $U$, the number of distinct sample support sets. Non-monotonic, highly non-additive loss functions are not directly supported without deriving new lower bounds. In practice, GOSDT is most effective for moderate-dimensional, structured tabular data.

## 7. Relationship to Other Optimal Tree Methods

GOSDT advances over OSDT and integer-programming–based ODTs by combining scalable DP/B&B with tight, leaf-aware lower bounds and efficient data representations. In contrast to "Branches" (AO*-style AND/OR graph search), GOSDT identifies subproblems via data-support sets, imposes user-set maximum depth (or leaf) constraints, and leverages bitwise computation for speed. GOSDT requires discretized (binary) feature representations for continuous variables but efficiently handles vast numbers of thresholds; methods such as Branches can natively support multi-valued features without one-hot encoding, potentially yielding further efficiency in certain regimes [2406.02175, 1612.03225]. 

A plausible implication is that GOSDT’s generalizations via weighting and objective flexibility establish it as a key algorithmic paradigm for interpretable model construction whenever sample importance, treatment regimes, or complex policy objectives are central concerns.

Source: https://www.emergentmind.com/topics/generalized-optimal-sparse-decision-trees-gosdt