Optimal Policy Trees for Decision-Making

• The paper presents optimal policy trees that integrate machine learning, stochastic programming, and reinforcement learning to yield interpretable, near-optimal decision rules.
• Policy trees enable personalized prescriptions and scalable optimization by leveraging causal inference, scenario tree approximations, and mixed-integer programming.
• The approach balances constraints, multi-objective trade-offs, and computational efficiency to support data-driven, high-stakes sequential decision-making.

An optimal policy tree is a data-driven, tree-structured policy for sequential and/or multistage decision-making under uncertainty, designed to maximize expected reward (or minimize loss) within explicit representational or operational constraints. Recent advances in optimal policy tree methodology span multistage stochastic programming, causal inference, combinatorial optimization, reinforcement learning, model checking for control, and interpretable machine learning. These approaches produce policies in the form of decision trees whose internal structure conveys interpretable rules, explicit prescriptions, or compact coverage over families of models, while providing formal or empirical guarantees on optimality, performance, or tractability.

1. Policy Trees in Multistage Stochastic Programming

One foundational approach leverages scenario tree approximations to address the intractability of full multistage stochastic programs. The scenario tree discretizes the uncertainty at each stage, producing a tree in which each node represents a possible state history and encodes constraints for nonanticipative, adaptive decision-making. The discrete tree is solved, yielding a sequence of recourse decisions $\{x_t^k\}$ per scenario. Crucially, the mapping from observed information history to optimal actions in the tree ($D_t = \{(\xi_{(t)}^k, x_t^{k*})\}_k$) is then used to "lift" the solution to a continuous policy via supervised machine learning—specifically, nonparametric models such as Gaussian process regression:

$$\lambda_t^\theta(\xi_{(t)}) = m_t(\xi_{(t)}) + C_t^\theta(D_t, \xi_{(t)})^\top [C_t^\theta(D_t, D_t) + \sigma_w^2 I]^{-1}(x_t^*(D_t) - m_t(D_t)),$$

where $m_t(\cdot)$ is a mean function and $C_t^\theta$ is a kernel.

This machine-learned policy enables rapid out-of-sample simulation across independent realizations for accurate, unbiased policy evaluation and confidence quantification:

$$v^{(TS)}(\pi) = \frac{1}{N} \sum_{n=1}^N [f_1(\pi_1) + f_2(\pi_2(\xi_{(2)}^n), \xi_2^n) + \dots + f_T(\pi_T(\xi_{(T)}^n), \xi_T^n)].$$

Candidate scenario trees with potentially random structures are solved in parallel, each giving rise to a candidate policy; simulation is used to select the policy that is empirically best for the “true” problem, rather than the best on any one scenario tree (Defourny et al., 2011). This decouples the offline approximation burden of the scenario tree from the final policy quality, achieving excellent trade-offs between runtime and solution performance in numerical tests.

2. Prescriptive Policy Trees from Data and Causal Inference

Optimal policy trees have been adapted for prescription directly from data, with extensive use in personalized medicine, resource allocation, and pricing. A unifying approach first estimates counterfactual outcomes using state-of-the-art causal inference estimators:

• For discrete treatments, doubly robust estimation combines propensity score estimation (random forests, boosting) and outcome regression;
• For continuous treatments, outcome regression across discretized candidate doses.

With per-unit, per-treatment outcome estimates $\Gamma_{it}$, tree learning is posed as the minimization of (estimated) total cost partitioned by leaf:

$$\min_{\tau(\cdot)} \sum_i \sum_t \mathbb{1}\{\tau(x_i) = t\}\Gamma_{it},$$

where the optimal action in each leaf is

$$z_\ell = \arg\min_t \sum_{i: v(x_i) = \ell} \Gamma_{it}.$$

This approach is operationalized with globally-optimized (not greedy) tree learning, often via coordinate descent (Amram et al., 2020), or mixed-integer optimization (MIO) (Jo et al., 2021). Such frameworks yield interpretable, scalable prescription policies robust to observational data and capable of incorporating complex constraints (e.g., fairness, budgets). Asymptotic optimality versus the true policy is established under weak conditions by direct optimization of consistent causal estimators, extending guarantees even to shallow, interpretable trees.

3. Optimization and Scalability of Policy Tree Learning

Efficient construction of optimal policy trees in the generic setting is a combinatorial optimization problem. Canonical recursions break the search across tree depth by optimally partitioning a data set $N$: $$R^*_{d,N} = \max_{s \in S} [R^*_{d-1, s_L(N)} + R^*_{d-1, s_R(N)}], \quad R^*_{0,N} = \max_{a \in A} \sum_{i \in N} r(i, a),$$ where the base case assigns all units in $N$ to a single action (Cussens et al., 18 Jun 2025).

Recent algorithmic advances deploy recursive decomposition with discrete optimization bounds for search-space pruning: when observations are shifted between branches, subtrees’ optimal reward contributions are upper/lower bounded without recomputation, e.g.,

$$R^*_{d, N_1 \cup N_3} \leq R^*_{d, N_1} + \sum_{i \in N_3} \max_{a \in A} r(i, a).$$

Efficient set representations (including counting or radix sorts for branch allocation) and aggressive memoization—caching subproblem solutions—enable runtime reductions by nearly 50-fold versus previous "policytree"-type methods, facilitating training on larger datasets or deeper trees (Cussens et al., 18 Jun 2025). These methods have been released in open-source packages suitable for high-dimensional and policy-scale applications.

4. Policy Trees in Reinforcement Learning and Control

Interpretability-driven methods in RL and control have focused on synthesizing decision tree policies with provable performance. Mixed-Integer Linear Programming (MILP)-based frameworks jointly encode the tree structure and MDP dynamics: $$\max \sum_{s}{\sum_{a}{x_{s,a} \cdot \left(\sum_{s'}P_{s,s',a}R_{s,s',a}\right)}}$$ with flow conservation and routing constraints ensuring every feasible state traverses the correct path in the tree and is mapped to an action at a prescribed leaf (Vos et al., 2023). This formulation allows direct maximization of expected discounted return subject to constraints on the number of internal nodes, producing interpretable, small-depth trees that in many empirical settings approach the optimal reward of unrestricted policies.

Alternative approaches optimize tree structure using policy gradient updates (DTPO), combining regression tree heuristics and advantage-weighted gradient signals, with the tree representation updated to best fit gradient-improved predictions (Vos et al., 21 Aug 2024). These methods can outperform standard imitation or extraction (e.g., VIPER) when the underlying optimal policy is ill-suited to small trees, and are competitive with deep RL where interpretability is required.

For deterministic black-box systems, policy trees can be synthesized by exhaustive search in the space of discretized predicates, with pruning provided by trace-based observations: candidate predicates that cannot yield an improved or different execution trace (admissible path) are cut, preserving completeness and optimality within the discretized candidate set (Demirović et al., 5 Sep 2024).

For families of MDPs, e.g., parameterized by environmental or hardware uncertainty, recursive abstraction-refinement with game-based quotient MDPs produces a hierarchical "policy tree" mapping MDP instances to robust memoryless policies or to an “unsat” label. This compactly covers very large numbers of model variants, providing a scalable tool for robust policy synthesis and model checking (Andriushchenko et al., 17 Jul 2024).

5. Handling Constraints, Multi-Objective Trade-Offs, and Weighted Data

Optimal policy trees can encode problem-specific constraints and trade-offs:

• Mixed-integer programming formulations allow intra- and inter-rule constraints to be enforced directly in the tree induction process, enabling practical prescription under operational, regulatory, and fairness requirements (Subramanian et al., 2022).
• Multi-objective policy learning frameworks construct Pareto frontiers of policy trees with different weightings of multiple outcomes using Bayesian optimization, exposing the inherent trade-offs between competing policy goals and supporting selection of policies aligned with stakeholder priorities (Rehill et al., 2022).
• For weighted samples (e.g., incorporating importance sampling, inverse propensity, or other data-dependent priorities), new optimization algorithms enable bit-parallel or sampling-based reductions to unweighted tree induction, maintaining accuracy but with improved scalability (Behrouz et al., 2022). Theoretical error control and scalability are essential for large-scale deployment.

6. Analytical Characterization, Heuristics, and Cognitive Rationality

In resource-constrained planning, analytical work demonstrates that the optimal allocation of search or "imagination" capacity in deep, stochastic decision trees is almost always achieved by exploring a small number (typically two) branches at each depth while maximizing tree depth, rather than balancing width and depth per level—establishing "deep imagination" as nearly optimal in realistic environments (Moreno-Bote et al., 2021). This formalizes widely observed heuristics in human and animal planning and guides the construction of bounded-optimal decision tree policies in large, uncertain domains.

7. Applications and Empirical Performance

Optimal policy trees have demonstrated utility and practical benefits across domains:

• Multistage inventory, resource allocation, energy system design, and swing option pricing (Defourny et al., 2011).
• Personalized medicine (drug dosing, diabetes management), fair resource assignment, and pricing in large retail or financial platforms (Amram et al., 2020, Jo et al., 2021).
• Adaptive model selection or rejection for machine learning ensembles in high-stakes prediction (Bertsimas et al., 30 May 2024).
• Strategic planning under uncertainty and wargaming, logistics, and cyberdefense (Ozturk et al., 24 Apr 2025).
• Model checking for families of MDPs and interpretable reinforcement learning policies for control, navigation, and autonomous systems (Vos et al., 2023, Andriushchenko et al., 17 Jul 2024, Andriushchenko et al., 17 Jan 2025).

Scalability, interpretability, and performance close to the optimal limit are repeatedly empirically validated, with advances in discrete optimization, constraint handling, and generalization to families of problems accelerating both theoretical and applied contributions. In high-stakes decision environments—where transparency and control are non-negotiable—optimal policy trees serve as a bridge between data-driven optimization, formal guarantees, and actionable, succinct rules.