Nearly Optimal Bounds for Computing Decision Tree Splits in Data Streams

Abstract: We establish nearly optimal upper and lower bounds for approximating decision tree splits in data streams. For regression with labels in the range ${0,1,\ldots,M}$, we give a one-pass algorithm using $\tilde{O}(M<sup>2/ε)$ space that outputs a split within additive $ε$ error of the optimal split, improving upon the two-pass algorithm of Pham et al. (ISIT 2025). Furthermore, we provide a matching one-pass lower bound showing that $Ω(M<sup>2/ε)$ space is indeed necessary. For classification, we also obtain a one-pass algorithm using $\tilde{O}(1/ε)$ space for approximating the optimal Gini split, improving upon the previous $\tilde{O}(1/ε<sup>2)$-space algorithm. We complement these results with matching space lower bounds: $Ω(1/ε)$ for Gini impurity and $Ω(1/ε)$ for misclassification (which matches the upper bound obtained by sampling). Our algorithms exploit the Lipschitz property of the loss functions and use reservoir sampling along with Count--Min sketches with range queries. Our lower bounds follow from careful reductions from the INDEX problem.

• The paper introduces one-pass algorithms for regression and classification split selection, achieving nearly optimal additive error using sublinear space.
• It employs reservoir sampling and dyadic range sketches to compute approximate losses with bounded error per candidate split.
• Tight lower bounds via reductions from the Index communication complexity problem demonstrate the optimality of the streaming split selection methods.

Nearly Optimal Bounds for Computing Decision Tree Splits in Data Streams

Problem Statement and Context

This paper provides a definitive analysis of the sample and space complexity for approximating optimal decision tree splits in the streaming model, for both regression and classification objectives. The computational bottleneck in sequential and ensemble-based tree learners is split selection: efficiently identifying the threshold for a real-valued or discretized feature that minimizes an impurity or regression loss, using minimal space and passes over data. While the batch setting is well-studied, streaming split selection presents unique challenges due to harsh space constraints and the need for additive- or multiplicative-approximate solutions. Existing streaming tree methods either used strong distributional assumptions, multiple passes, or had suboptimal space bounds.

Main Contributions

The paper’s contributions consist of (1) one-pass algorithms for regression (MSE), misclassification, and Gini impurity split objectives that achieve nearly optimal additive error using sublinear space, (2) tight one-pass lower bounds for space usage via reductions from the Index communication complexity problem, and (3) algorithmic innovations in using reservoir sampling and efficient dyadic range sketches for loss estimation.

Regression Split Objective

For regression tasks with discrete labels in $\{0, \ldots, M\}$, the authors present the first truly one-pass algorithm with space $\widetilde{O}(M^2/\epsilon)$ to find a split whose squared loss is within $\epsilon$ additive error of optimal. Previous work required two passes or used at least $1/\epsilon^2$ space. The improvement is significant in practical use cases (ensemble training, multi-feature tabular data) where memory footprints per tree must be minimized.

The algorithm proceeds by:

• Maintaining a reservoir sample of candidate split points of size $O((M^2/\epsilon) \log N)$.
• Simultaneously updating three (count, sum, sum-of-squares) Count–Min sketches with dyadic range queries for accurate interval statistics estimation.
• For each candidate, computing an approximate regression loss by querying the sketches.
• Outputting the candidate with minimal estimated loss.

The correctness is ensured by showing (i) the optimal loss function’s approximate Lipschitz continuity w.r.t. feature order, so a near-optimal split is well-approximated in the sample; and (ii) the estimation error from the sketches is tightly bounded. The algorithm does not require knowledge of the stream length in advance and supports high-probability guarantees.

Gini Impurity and Misclassification

For binary classification:

• For the Gini impurity criterion, the paper gives a one-pass ${O(1/\epsilon)}$ space algorithm, strictly improving the prior art which needed $O(1/\epsilon^2)$ space. This is achieved via a label re-encoding technique that equates Gini impurity minimization to regression over $\{0,1\}$ labels, reducing the problem to the MSE case.
• For misclassification loss, the authors note sampling can already achieve $O(1/\epsilon)$ space and complete the theoretical picture by providing a matching lower bound.

Lower Bounds

The authors provide lower bounds for one-pass streaming algorithms:

• Any randomized one-pass algorithm for MSE split (regression) with labels in $[0,M]$ and error $\widetilde{O}(M^2/\epsilon)$0 must use $\widetilde{O}(M^2/\epsilon)$1 bits.
• For Gini impurity and misclassification, $\widetilde{O}(M^2/\epsilon)$2 bits are necessary. These are proved by reductions from the Index communication complexity problem; the reductions construct hard instances where any algorithm violating these bounds would solve Index with insufficient communication.

Notably, these lower bounds match the upper bounds up to polylogarithmic factors, demonstrating the algorithms are optimal.

Technical Analysis

The article rigorously analyzes the algorithm's error, showing all approximation steps preserve the additive error target. Central to the regression algorithm is a stability bound for the regression loss function over contiguous splits, which enables uniform sampling to yield a candidate set covering all near-optimal splits: for intervals with enough mass, reservoir sampling covers them with high probability. For each candidate, the Count–Min sketch is shown to provide sufficiently high-precision estimates for the first three moments (count, sum, sum of squares) in all required intervals, with failure probability $\widetilde{O}(M^2/\epsilon)$3.

For Gini impurity, the key technical observation is the transformation $\widetilde{O}(M^2/\epsilon)$4 under label re-encoding, so the regression machinery fully applies to impurity minimization.

The lower bounds are constructed via careful adversarial streams to manifest two adjacent split candidates whose difference in loss encodes a hidden bit of information, then rely on Index communication complexity to argue space hardness.

Empirical and Numerical Results

The paper is primarily theoretical and does not report empirical experiments, but all complexity statements are concrete. For regression, the one-pass algorithm works in $\widetilde{O}(M^2/\epsilon)$5 update time and $\widetilde{O}(M^2/\epsilon)$6 space—an exponential improvement in $\widetilde{O}(M^2/\epsilon)$7 versus baseline sketching approaches. For Gini impurity, the $\widetilde{O}(M^2/\epsilon)$8 space matches the trivial coupon-collector lower bound and constitutes a strong claim that improves prior art’s resource consumption by a quadratic factor.

Implications and Future Directions

The presented results close fundamental gaps in the streaming decision tree literature:

• The space complexities for one-pass split selection for all standard objectives (regression, Gini, misclassification) are now fully characterized.
• The techniques are practical for large-scale ensemble learning (e.g., gradient boosting, random forests) and for distributed and streaming systems where per-feature memory budgets are tight.
• The reduction-based lower bounds clarify the limits of all possible streaming approaches, not just for decision trees but also for related segmentation and change-point detection primitives.

Possible directions include:

• Extending these ideas to multiway splits or multiclass classification, where impurity measures become more complex.
• Exploring adaptivity to concept drift, temporal feature dependencies, or other streaming nonstationarities.
• Engineering highly optimized open-source implementations for popular streaming ML frameworks and benchmarking these algorithms in dataflow engines.

Conclusion

This work establishes tight bounds—both algorithmic and lower—for the space complexity of optimal split approximation in streaming decision tree induction, for both regression and classification. The proposed one-pass solutions using range-aware sketching and non-iid reductions unify previously disparate approaches and characterize when and how split selection can be performed at scale. The matching lower bounds solidify the theoretical landscape and offer clear guidelines for the design of resource-efficient learning under streaming constraints.

Reference:

"Nearly Optimal Bounds for Computing Decision Tree Splits in Data Streams" (2604.20394)