A Hierarchy of Policy Learning Problems

Abstract: Policy learning has received substantial attention with the goal of learning policies from observational data for decision-making. A majority of work in this space has focused on developing algorithms for computing policies that minimize regret compared to the optimal policy. However, in many practical settings, there is insufficient data to obtain low regret. As a result, recent work has shifted attention to alternative objectives, most notably, studying whether it is possible to learn an improving policy that statistically significantly outperforms baseline policies. We argue that there is substantial merit in studying a broader range of policy learning problems. When there is insufficient data to learn an improving policy, there may still be useful questions that can be answered. To this end, we provide a mathematical framework for studying the relationships between policy learning problems. We formalize three problems within our framework: beyond the optimal policy problem and the improving policy problem, we also propose the policy existence problem, which aims to determine if an improving policy exists. Within our framework, we show that the policy existence problem reduces to the improving policy problem, which in turn reduces to the optimal policy problem; these reductions prove that each problem is at least as easy as the next one (in sample complexity). A key question remains: is this hardness strict? We provide partial answers. First, the gap between the optimal policy and improving policy problems is strict. For the improving policy and policy existence problems, we prove that a sublinear polynomial gap exists under natural conditions on improving policy learning algorithms. Thus, we may be able to answer questions about the existence of an improving policy even when we cannot find one. These results highlight the value in studying a broader range of policy learning problems.

• The paper introduces a rigorous hierarchy for offline policy learning, formalizing optimal, improving, and existence problems via clear mathematical reductions.
• It establishes concrete sample complexity bounds, such as n ≈ (8σ_max² log(2k/δ))/τ_min², highlighting that detecting policy existence may require far fewer samples than constructing improvements.
• The work offers practical guidance for low-data decision-making and sets the stage for future research on efficient algorithm design and refined statistical validations.

Succinct Summary and Problem Hierarchy

The paper "A Hierarchy of Policy Learning Problems" (2607.03385) introduces a rigorous mathematical framework for analyzing the sample complexity and relationships between various policy learning objectives in offline, observational data settings. It exposes and formalizes three distinct classes of policy learning problems, each corresponding to progressively weaker desiderata:

1. Optimal Policy Problem: Find a policy that maximizes expected reward, minimizing regret against the unknown optimal policy.
2. Improving Policy Problem: Find a policy that statistically significantly improves over simple baselines (e.g., constant or random policies).
3. Policy Existence Problem: Detect whether there exists any policy that outperforms the best constant policy, even if explicit improvement cannot be constructed.

The paper organizes these into a hierarchy, showing that each problem is inherently at least as difficult as the next in terms of sample complexity. It establishes rigorous reductions between these problems, enabling direct translation of sample complexity bounds.

Formal Framework and Problem Reductions

The authors develop a generalized, notion-agnostic formalism for policy learning problems. Each problem is defined by a specific output space and validator function. Sample complexity is characterized by the number of samples required for an algorithm to achieve a target false negative rate (FNR), subject to a false positive rate (FPR) constraint, where algorithms must abstain unless they are statistically confident.

Reductions between problems ($\mathcal{V}$ to $\mathcal{V}'$) are guaranteed via mappings between output spaces that preserve correctness for instances. The paper proves that the optimal policy problem reduces to the improving policy problem, which in turn reduces to the existence problem, thus establishing their relative hardness. Sample complexity upper bounds for the "harder" problem are automatically upper bounds for the "easier" one.

Theoretical Results and Sample Complexity Analysis

The analysis includes strong numerical results establishing tight sample complexity upper and lower bounds for each problem using standard concentration inequalities. For $k$ unit types, bounded treatment effects $\tau_{\min}$, and outcome variance $\sigma_{\max}$, the optimal policy and improving policy problems require:

$$n \approx \frac{8\sigma_{\max}^2 \log(2k/\delta)}{\tau_{\min}^2}$$

samples to achieve FNR and FPR below $\delta$, assuming all types have effects above $\tau_{\min}$. The improving policy problem is easier in the sense that the bound holds over broader instances, e.g., only some types need to be confidently classified.

A central and novel result is identifying strict separation between the improving policy and policy existence problems. For non-pathological instances with many types and only sublinear (in $k$) number of positive-effect types, their existence can be reliably detected with as few as one sample per type, by aggregating across types. In contrast, any monotone algorithm for improving policy learning requires at least a sublinear polynomial number of samples per type. This demonstrates a pronounced gap—in some scenarios, existence of an improving policy can be detected long before an improved policy can be constructed.

Practical and Theoretical Implications

The framework systematically clarifies what can be answered about data-driven decision-making under severe data limitations. It exposes the utility of asking weaker questions—such as policy existence—when optimizing or even identifying improving policies is infeasible. This has practical implications for advising resource allocation, experiment design, and deciding whether to pursue further data collection or abandon personalization.

Theoretically, the hierarchy deepens understanding of the landscape of policy learning, situating recent advances (e.g., high-confidence policy selection, robust policy improvement) within a formal context and suggesting new directions. The notion of abstention and correct statistical decision-making under varying constraints is made precise for comparing heterogeneous classes of learning objectives.

Future Directions

The paper indicates several avenues for further research:

• Sharper characterization of strict sample complexity gaps, especially for intermediate instance spaces.
• Algorithmic advances for existence detection under varied treatment effect heterogeneity.
• Extensions to settings with missing data, structured covariates, or unknown propensity scores.
• Exploration of additional problems, such as actionable heterogeneity detection and permutation tests for fine-grained decision-making.

Conclusion

This work delivers a unified, abstract mathematical lens for comparing policy learning problems and their sample complexities, formalizing a strict hierarchy with practical and theoretical separation. The results challenge the sufficiency of regret minimization and highlight the value of studying weaker objectives, such as policy existence detection. Systematic reductions and rigorous complexity analysis enable robust guidance in low-data regimes, and set the stage for broadening the menu of questions answerable by offline machine learning for automated and human-in-the-loop decision-making.