BudgetRF: Feature-Budgeted Random Forest

• The paper introduces BudgetRF, a novel algorithm that integrates per-feature cost constraints into random forest learning to optimize prediction accuracy.
• BudgetRF employs a GreedyTree splitting strategy with minimax cost-weighted impurity measures, achieving a logarithmic approximation guarantee on feature cost.
• Empirical evaluations across diverse datasets demonstrate that BudgetRF outperforms state-of-the-art cost-sensitive methods with significantly lower feature usage.

Feature-Budgeted Random Forest (BudgetRF) is an algorithmic framework designed for learning and inference under explicit, per-feature acquisition budget constraints. Unlike classical Random Forests, BudgetRF is engineered to optimize prediction accuracy subject to a user-specified average feature cost limit at prediction time, accounting for heterogeneous feature costs and redundancy across ensemble members.

1. Prediction-Time Feature Cost Constraints

Standard Random Forests do not incorporate the cost of acquiring features during prediction; all trees may query every available feature for every sample. In numerous applications, such as surveillance or large-scale retrieval, feature computation and measurement can incur significant monetary, computational, or latency expenses. In these contexts, the objective shifts to maximizing prediction accuracy while not exceeding a prespecified average feature acquisition budget. Feature costs can be heterogeneous—some features may be prohibitively expensive for a marginal gain. Moreover, diversity among trees in RF generally amplifies cost, since features are acquired independently per tree without feature reuse coordination.

BudgetRF formalizes the risk minimization problem for hypothesis class $\mathcal{F}$ and loss $L(\cdot)$:

$$\min_{f \in \mathcal{F}} \frac{1}{n}\sum_{i=1}^n L(y_i, f(x_i)) ,\quad \text{subject to} \quad \frac{1}{n}\sum_{i=1}^n C(f, x_i) \leq B$$

where $C(f, x_i)$ is the total feature cost incurred for prediction $f(x_i)$ and $B$ is the average cost budget.

2. Minimax Cost-Weighted Impurity Splits and Admissible Impurity

At BudgetRF's core lies the "GreedyTree" algorithm for constructing budget-optimized decision trees. Unlike classic impurity-based tree growth, GreedyTree introduces a splitting criterion that directly incorporates feature costs using a minimax (worst-case) formulation:

Let $F$ be an admissible impurity function, $c(t)$ the cost for feature $t$, and $\mathcal{G}_t$ the split functions for $t$. For each available feature, compute:

$$R(t) = \min_{g_t \in \mathcal{G}_t} \max_{i} \frac{c(t)}{F(S) - F(S^i_{g_t})}$$

where $S$ is the current set of samples at a node and $S^i_{g_t}$ are samples routed to branch $i$ by split $g_t$. The algorithm selects $(t^*, g_{t^*})$ minimizing $R(t)$: this is the split that, in the worst-case (across branches), most efficiently reduces impurity per unit cost.

An impurity function $F$ is admissible if it is monotonic, supermodular, non-negative, and zero for pure (single-label) sets. BudgetRF employs the "threshold-Pairs" impurity:

$$F_\alpha(G) = \sum_{i\neq j}\left[\left[n^i_G-\alpha\right]_+\left[n^j_G-\alpha\right]_+-\alpha^2\right]_+$$

with $[x]_+ = \max(x,0)$ and $n^i_G$ the count of class-$i$ samples in group $G$. These admissibility properties are essential for the theoretical analysis of cost guarantees.

3. Forest Construction under Feature Budgets

The BudgetRF ensemble is assembled iteratively:

• Bootstrap training data to grow a tree via GreedyTree.
• Accumulate trees one-at-a-time until average feature cost on a (held-out) validation set reaches the budget $B$.
• Remove the last added tree (which exceeds $B$).

This approach ensures both tree construction and ensemble expansion honor the specified prediction-time feature budget. During inference, each test sample is pushed through every tree; the model only incurs the cost for each feature once per sample, even if reused across trees.

Simplified construction pseudocode:

def BudgetRF(F, B, C, y_tr, X_tr, y_val, X_val): T = [] while average_cost_on_validation(T) <= B: X_sample, y_sample = bootstrap(X_tr, y_tr) tree = GreedyTree(F, C, y_sample, X_sample) T.append(tree) return T[:-1] # Remove tree that exceeded budget

4. Theoretical Cost Guarantees

BudgetRF provides rigorous bounds on tree evaluation cost using admissible impurity functions. For a sample set of size $n'$, let $OPT(S)$ denote the minimum max-cost achievable by any impurity-purifying tree. Let $Cost_F(S)$ be the max-cost of GreedyTree's output. The algorithm guarantees:

$Cost_F(S) \leq O(\log n') \cdot OPT(S)$

This logarithmic approximation bound arises by induction over tree depth, leveraging supermodularity and monotonicity of admissible impurity, and the greedy selection of splits. The effective result is that BudgetRF delivers trees that are nearly minimax-optimal in cost.

5. Empirical Performance and Comparative Evaluation

Extensive experiments evaluate BudgetRF against then state-of-the-art cost-sensitive inference methods such as ASTC and CSTC across datasets with unit and non-uniform costs:

Datasets and Results:

• Yahoo! Learning to Rank (519 features): BudgetRF exceeds the precision of ASTC/CSTC at drastically lower cost (e.g., precision@5 at cost 70 surpasses that of competitors at cost >450).
• MiniBooNE (50 features, binary): At every budget level, BudgetRF attains lower test error than ASTC/CSTC. With only 6 features, it matches or exceeds their accuracy.
• Forest Covertype (54 features, 7-class): Error decreases rapidly with budget. BudgetRF outperforms others, especially for mixed feature types.
• CIFAR-10 (400 binarized features): At lower budgets, ASTC has lower error, but BudgetRF overtakes, maintaining superior accuracy at higher budgets while competitors overfit.

Comparison to unbudgeted RF:

Random Forests limited to $k$ trees cannot achieve similar budget/accuracy trade-offs; BudgetRF uses a much lower fraction of features per example while retaining competitive accuracy. For instance, on Forest data, BudgetRF uses only 23% of features versus RF’s 63% (10 trees), incurring just a slight increase in error.

6. Related Methods and Extensions

BudgetRF marked a transition from cost-agnostic ensemble learning toward explicit budgeted prediction-time control. It inspired subsequent research on feature selection under cost constraints, optimal ensemble pruning, feature sharing coordination, and adaptive regularization.

Notably, bottom-up approaches for RF pruning (Nan et al., 2016), LP-solvable ensemble pruning (Nan et al., 2016), and multivariate feature selection for cost-constrained RFs (Jagdhuber et al., 2020) build on, complement, or extend BudgetRF's philosophy. However, BudgetRF distinguishes itself by its greedy, cost-weighted impurity reduction per tree and log-approximation optimality guarantee for tree cost, which are not matched by pure pruning or filter-based methods.

Method	Feature Cost Awareness	Feature Sharing Across Trees	Theoretical Cost Guarantee
BudgetRF	Yes	No	$O(\log n')$ to optimum per tree
Ensemble Pruning	Yes	Yes	LP-relaxation is exact (Nan et al., 2016)
Cost-Aware Filter	Yes	No	Data dependent

A plausible implication is that BudgetRF serves as a principled basis for ensemble generation where hard prediction-time budgets are paramount, and extensions that introduce cross-tree feature sharing or stronger global optimization can further enhance real-world efficiency.

7. Summary and Impact

Feature-Budgeted Random Forest (BudgetRF) is a theoretically principled and empirically validated algorithm for random forest learning under feature acquisition budgets. By leveraging a greedy cost-weighted impurity reduction criterion with admissible impurity measures, it produces tree ensembles that attain near-optimal cost-accuracy trade-offs. BudgetRF consistently outperforms previous methods on benchmark datasets, achieving strong predictive power with substantially reduced feature usage. The approach's theoretical cost bounds and empirical results underpin its relevance in cost-sensitive and resource-constrained predictive analytics, setting a foundation for ongoing research in budgeted ensemble methods.