First-Best Optimal Policy Learning Explained

• First-Best Optimal Policy Learning (OPL) is a framework for selecting policies that maximize expected welfare by mapping covariates to optimal treatments.
• It integrates risk-neutral and risk-averse criteria, utilizing methods such as regression adjustment, inverse-probability weighting, and doubly robust estimation to estimate policy value.
• The approach extends to constrained settings, censored survival outcomes, and offline reinforcement learning, ensuring robust decision rules under practical limitations.

First-Best Optimal Policy Learning (OPL) is the problem of selecting a policy that maximizes a welfare criterion over a policy class, often the unrestricted class of measurable maps from observed covariates or states to actions. In the potential-outcome treatment setting, the first-best or unconstrained optimal policy assigns each covariate profile to the treatment with the larger conditional mean outcome; in multi-action treatment settings, the same optimization can be posed under risk neutrality or under risk-averse criteria that penalize conditional variance or second moments (Cerulli, 2020, Cerulli, 8 Sep 2025). Across contextual bandits, constrained allocation, censored survival analysis, episodic reinforcement learning, and partially identified structural models, the term denotes the welfare-maximizing benchmark relative to the admissible policy class and feasibility constraints, not a single algorithmic template (Cerulli, 12 May 2026, Kubota et al., 24 Mar 2026, Wu et al., 2021, Russell, 2020).

1. Formal welfare criteria and oracle characterizations

In the multi-action treatment formulation, observed covariates are denoted by $X\in\mathcal X\subseteq\mathbb R^p$, treatment by $A\in\mathcal A=\{0,1,\dots,M-1\}$, and observed outcome by $Y$. Using potential outcomes $Y(a)$, the maintained identifying conditions are conditional independence,

$\{Y(a):a\in\mathcal A\}\perp A\mid X,$

and overlap,

$e(a\mid x)=P[A=a\mid X=x]>0$

for all $a\in\mathcal A$ and $x$ in the support of $X$. A deterministic policy is a mapping $\pi:\mathcal X\to\mathcal A$, and first-best OPL chooses $A\in\mathcal A=\{0,1,\dots,M-1\}$0 from the unrestricted class $A\in\mathcal A=\{0,1,\dots,M-1\}$1 of all measurable functions $A\in\mathcal A=\{0,1,\dots,M-1\}$2 (Cerulli, 8 Sep 2025).

Under risk neutrality, the objective is

$A\in\mathcal A=\{0,1,\dots,M-1\}$3

The same framework permits risk-sensitive criteria. With conditional mean $A\in\mathcal A=\{0,1,\dots,M-1\}$4 and conditional variance $A\in\mathcal A=\{0,1,\dots,M-1\}$5, a linear risk-averse decision maker with aversion parameter $A\in\mathcal A=\{0,1,\dots,M-1\}$6 uses

$A\in\mathcal A=\{0,1,\dots,M-1\}$7

and the induced first-best policy solves

$A\in\mathcal A=\{0,1,\dots,M-1\}$8

A quadratic risk-averse criterion instead penalizes the second moment,

$A\in\mathcal A=\{0,1,\dots,M-1\}$9

so that

$Y$0

with

$Y$1

in practice (Cerulli, 8 Sep 2025).

In the binary-treatment case, let $Y$2 for $Y$3 and $Y$4. The oracle first-best policy is

$Y$5

and the maximized welfare is

$Y$6

This is the omniscient benchmark obtained when the policymaker knows the conditional mean outcomes (Cerulli, 2020).

Setting	First-best criterion	Source
Multi-action treatment	$Y$7 or risk-averse variants	(Cerulli, 8 Sep 2025)
Binary treatment	$Y$8	(Cerulli, 2020)
Budget and coverage	$Y$9 subject to budget and coverage	(Cerulli, 12 May 2026)
Censored survival	maximize $Y(a)$0 subject to $Y(a)$1	(Kubota et al., 24 Mar 2026)
Episodic RL	policy minimizing $Y(a)$2 for all $Y(a)$3	(Wu et al., 2021)
Partial identification	$Y(a)$4	(Russell, 2020)

This taxonomy indicates that “first-best” is defined relative to the feasible policy space and welfare functional. In unrestricted treatment OPL it is the measurable-policy optimum; in constrained or robust settings it is the optimum after imposing budget, coverage, censoring, or ambiguity structure.

2. Value-function estimation and implementation in multi-action treatment OPL

With i.i.d. observations $Y(a)$5, first-best OPL in the Stata implementation opl_ma_fb proceeds by estimating the conditional mean and, when needed, the conditional variance. The algorithm fits a linear model of $Y(a)$6 on $Y(a)$7 and $Y(a)$8 interactions, or separate regressions by action, to obtain $Y(a)$9; computes squared residuals and regresses them on the same design to obtain $\{Y(a):a\in\mathcal A\}\perp A\mid X,$0; forms an individual-level predicted welfare score $\{Y(a):a\in\mathcal A\}\perp A\mid X,$1; and assigns

$\{Y(a):a\in\mathcal A\}\perp A\mid X,$2

saving the result by default as _opt_policy (Cerulli, 8 Sep 2025).

The predicted welfare score depends on the chosen model:

$\{Y(a):a\in\mathcal A\}\perp A\mid X,$3

The implementation can also compare $\{Y(a):a\in\mathcal A\}\perp A\mid X,$4 to a benchmark policy, save match indicators, and produce summary frequencies. It then computes and reports the estimated value-function under $\{Y(a):a\in\mathcal A\}\perp A\mid X,$5 using the regression-adjustment formula, reported as e(V_opt_train) and e(V_opt_new) (Cerulli, 8 Sep 2025).

The companion command [opl_ma_vf](https://www.emergentmind.com/topics/opl_ma_vf) evaluates the maximal welfare after policy construction: $A\in\mathcal A=\{0,1,\dots,M-1\}$68 It returns three estimators of $\{Y(a):a\in\mathcal A\}\perp A\mid X,$6: regression adjustment (RA), inverse-probability weighting (IPW), and doubly robust (DR), stored in e(RA), e(IPW), and e(DR). The paper gives the example “RA = 124.19, IPW = 109.50, DR = 125.31” (Cerulli, 8 Sep 2025).

The RA and DR estimators are

$\{Y(a):a\in\mathcal A\}\perp A\mid X,$7

and

$\{Y(a):a\in\mathcal A\}\perp A\mid X,$8

The DR estimator remains consistent if either $\{Y(a):a\in\mathcal A\}\perp A\mid X,$9 or $e(a\mid x)=P[A=a\mid X=x]>0$0 is correctly specified (Cerulli, 8 Sep 2025).

The same implementation includes graphical output. opl_ma_fb can produce gr_action_train, a bar-chart comparing the frequency of actual vs. optimal actions in the training sample; gr_reward_train, a scatter or line-plot of observed vs. maximal expected reward; and gr_reward_new, a plot of predicted reward $e(a\mid x)=P[A=a\mid X=x]>0$1 across the new sample. Alternatively, opl_plot_best displays side-by-side a scatter of “observed” vs. “maximal” predicted rewards and a categorical plot of “observed treatment” vs. “optimal treatment” for each $e(a\mid x)=P[A=a\mid X=x]>0$2 (Cerulli, 8 Sep 2025).

3. Restricted policy classes and empirical welfare maximization

Although first-best OPL is defined over unrestricted measurable policies, applied work often imposes a lower-dimensional structure. A prominent restriction is the threshold policy

$e(a\mid x)=P[A=a\mid X=x]>0$3

where $e(a\mid x)=P[A=a\mid X=x]>0$4 is a one-dimensional score such as an estimated conditional treatment effect $e(a\mid x)=P[A=a\mid X=x]>0$5, a linear index, or a nonparametric score derived from domain knowledge (Cerulli, 2020).

The threshold-based formulation separates specification from policy selection. First, one estimates $e(a\mid x)=P[A=a\mid X=x]>0$6, often using regression adjustment or machine-learning methods such as causal forest, random forest, or boosted trees. Second, one chooses the scalar threshold $e(a\mid x)=P[A=a\mid X=x]>0$7 that maximizes welfare within the restricted class. With estimated outcome models $e(a\mid x)=P[A=a\mid X=x]>0$8, the empirical welfare for threshold $e(a\mid x)=P[A=a\mid X=x]>0$9 is

$a\in\mathcal A$0

The unrestricted first-best threshold is

$a\in\mathcal A$1

and the resulting policy $a\in\mathcal A$2 converges to $a\in\mathcal A$3 as $a\in\mathcal A$4 tracks the true $a\in\mathcal A$5 (Cerulli, 2020).

The empirical implementation protocol is a five-step “empirical welfare-maximizing” procedure: input $a\in\mathcal A$6; estimate $a\in\mathcal A$7 and $a\in\mathcal A$8 and compute $a\in\mathcal A$9; compute the unconstrained first-best welfare

$x$0

evaluate $x$1 on a fine grid; and select

$x$2

To obtain valid out-of-sample welfare estimates and confidence intervals, the procedure recommends sample splitting into two folds, cross-fit welfare evaluation, and bootstrap over the entire procedure, including sample splitting (Cerulli, 2020).

Under unconfoundedness, overlap, bounded outcomes, uniform consistency of $x$3 and $x$4, and Lipschitz continuity of $x$5, the paper states a uniform regret consistency result:

$x$6

and any sequence $x$7 satisfies

$x$8

The formal statement is framed as a proposition using Assumptions A1–A3: unconfoundedness, overlap, bounded $x$9; uniform consistency of $X$0 and $X$1; and Lipschitz continuity of $X$2 (Cerulli, 2020).

This restricted-policy literature clarifies a central distinction within first-best OPL: the oracle target is the unrestricted welfare maximizer, whereas operational policies may be threshold rules chosen because they are easy to interpret, easy to implement, or compatible with treatment-share constraints.

4. Constraint-driven and outcome-specific extensions

A major extension studies first-best OPL under explicit resource constraints. In the budget-and-coverage setting, the planner chooses binary decisions $X$3, where $X$4 if unit $X$5 is treated, with expected gain $X$6, cost $X$7, total budget $X$8, and minimum coverage requirement $X$9. The optimization problem is

$\pi:\mathcal X\to\mathcal A$0

The paper shows that this problem admits a knapsack-type structure and that the optimal policy can be characterized by an affine threshold rule involving budget and coverage shadow prices. With dual variables $\pi:\mathcal X\to\mathcal A$1 for the budget constraint and $\pi:\mathcal X\to\mathcal A$2 for the coverage constraint, the LP relaxation yields

$\pi:\mathcal X\to\mathcal A$3

under a non-degeneracy assumption, equivalently treating unit $\pi:\mathcal X\to\mathcal A$4 iff

$\pi:\mathcal X\to\mathcal A$5

The LP relaxation has at most two fractional components, and if $\pi:\mathcal X\to\mathcal A$6 for all $\pi:\mathcal X\to\mathcal A$7, then

$\pi:\mathcal X\to\mathcal A$8

implying asymptotic equivalence in per-unit welfare. Two implementable algorithms are analyzed: Greedy–Lagrangian with Coverage (GLC), which is asymptotically exact in per-capita regret under boundedness and non-degeneracy, and rank-and-cut (RC), which is exact if costs are constant or the coverage multiplier is zero, but may misallocate when cost heterogeneity interacts with a binding coverage constraint (Cerulli, 12 May 2026).

A separate extension addresses right-censored survival outcomes. With logged data

$\pi:\mathcal X\to\mathcal A$9

the first-best OPL problem becomes

$A\in\mathcal A=\{0,1,\dots,M-1\}$00

where

$A\in\mathcal A=\{0,1,\dots,M-1\}$01

To correct censoring bias, the framework introduces IPCW-IPS and IPCW-DR. Under independent censoring, with $A\in\mathcal A=\{0,1,\dots,M-1\}$02 and importance weight $A\in\mathcal A=\{0,1,\dots,M-1\}$03,

$A\in\mathcal A=\{0,1,\dots,M-1\}$04

With any estimator $A\in\mathcal A=\{0,1,\dots,M-1\}$05 and residual

$A\in\mathcal A=\{0,1,\dots,M-1\}$06

the doubly robust estimator is

$A\in\mathcal A=\{0,1,\dots,M-1\}$07

The paper states that IPCW-IPS is unbiased if $A\in\mathcal A=\{0,1,\dots,M-1\}$08 is correct and $A\in\mathcal A=\{0,1,\dots,M-1\}$09 is known, while IPCW-DR is unbiased if $A\in\mathcal A=\{0,1,\dots,M-1\}$10 is correct and either $A\in\mathcal A=\{0,1,\dots,M-1\}$11 or $A\in\mathcal A=\{0,1,\dots,M-1\}$12 is correct; it also states a variance reduction result in which IPCW-DR variance is no larger than IPCW-IPS variance by a nonnegative gap (Kubota et al., 24 Mar 2026).

These constrained and outcome-specific formulations preserve the first-best principle—maximize policy value over admissible policies—while altering either the feasible set or the statistical object being estimated.

5. Large action spaces, new actions, and optimization pathologies

In large-action contextual bandits, first-best OPL is often defined as learning a policy $A\in\mathcal A=\{0,1,\dots,M-1\}$13 that nearly maximizes

$A\in\mathcal A=\{0,1,\dots,M-1\}$14

relative to the oracle policy that concentrates on the action with maximal true reward at each context,

$A\in\mathcal A=\{0,1,\dots,M-1\}$15

A central recent finding is that optimization difficulties can dominate estimator quality. For OPE-based objectives such as IPS, cIPS, DR, MIPS, OffCEM, and POTEC, the paper proves that for any OPE estimator linear in $A\in\mathcal A=\{0,1,\dots,M-1\}$16 and a standard softmax parameterization $A\in\mathcal A=\{0,1,\dots,M-1\}$17, there exist problem instances where gradient descent is trapped in a suboptimal region for $A\in\mathcal A=\{0,1,\dots,M-1\}$18 iterations, and that the nonconcave landscape can have exponentially many local maxima in $A\in\mathcal A=\{0,1,\dots,M-1\}$19. By contrast, for weighted log-likelihood objectives

$A\in\mathcal A=\{0,1,\dots,M-1\}$20

with an $A\in\mathcal A=\{0,1,\dots,M-1\}$21-regularized linear softmax policy, $A\in\mathcal A=\{0,1,\dots,M-1\}$22 is strongly concave in $A\in\mathcal A=\{0,1,\dots,M-1\}$23, so the loss admits a unique global minimum, no spurious stationary points, and can be optimized in $A\in\mathcal A=\{0,1,\dots,M-1\}$24 gradient steps to $A\in\mathcal A=\{0,1,\dots,M-1\}$25-optimality. Empirically, OPE-based methods are reported as highly fragile to batch size and learning-rate schedule, whereas PWLL methods remain robust across all configurations and often achieve higher validation reward and lower regret (Aouali et al., 3 Sep 2025).

A different response to large action spaces is structural decomposition. POTEC defines a clustering map $A\in\mathcal A=\{0,1,\dots,M-1\}$26 with $A\in\mathcal A=\{0,1,\dots,M-1\}$27 and factorizes the policy as

$A\in\mathcal A=\{0,1,\dots,M-1\}$28

Stage 1 learns a cluster-selection policy by a low-variance policy-gradient estimator that only reweights cluster probabilities, while Stage 2 uses a regression-based within-cluster selector. Under Full-Cluster-Support and Local Correctness, the estimator is unbiased and the second-stage policy

$A\in\mathcal A=\{0,1,\dots,M-1\}$29

is optimal within each cluster. The framework strictly generalizes pure policy-based and pure regression-based OPL: if $A\in\mathcal A=\{0,1,\dots,M-1\}$30, it degenerates to a pure regression-based policy; if $A\in\mathcal A=\{0,1,\dots,M-1\}$31, it degenerates to IPS policy gradient (Saito et al., 2024).

The evolving-action setting exposes a further limitation of standard first-best OPL. Let

$A\in\mathcal A=\{0,1,\dots,M-1\}$32

Standard IPS and DR require $A\in\mathcal A=\{0,1,\dots,M-1\}$33 to produce unbiased estimates, and DM requires an action to appear in the logged data to train $A\in\mathcal A=\{0,1,\dots,M-1\}$34. Hence no standard estimator can assign positive probability to any $A\in\mathcal A=\{0,1,\dots,M-1\}$35. To address this, LCPI uses discrete action features and a pseudo-inverse correction:

$A\in\mathcal A=\{0,1,\dots,M-1\}$36

where

$A\in\mathcal A=\{0,1,\dots,M-1\}$37

Under Local Combination Support and Local Linearity, LCPI is unbiased for the policy gradient and can extrapolate to new actions that share supported features. PONA then blends LCPI and DR:

$A\in\mathcal A=\{0,1,\dots,M-1\}$38

thereby trading off aggressiveness on new actions against conservatism on existing actions (Kishimoto et al., 18 May 2026).

Taken together, these results identify a recurring misconception: better off-policy value estimation does not automatically imply better first-best policy learning. In large action spaces, trainability, decomposition, and feature-based extrapolation can become the binding considerations.

6. Offline reinforcement learning, uniform guarantees, and partially identified environments

In episodic offline reinforcement learning, first-best OPL is tied to uniform policy evaluation. For a finite-horizon MDP with horizon $A\in\mathcal A=\{0,1,\dots,M-1\}$39, state space $A\in\mathcal A=\{0,1,\dots,M-1\}$40, action space $A\in\mathcal A=\{0,1,\dots,M-1\}$41, and offline data generated by a logging policy $A\in\mathcal A=\{0,1,\dots,M-1\}$42, the model-based plug-in estimator forms empirical reward and transition estimates, computes estimated occupancies $A\in\mathcal A=\{0,1,\dots,M-1\}$43, and evaluates

$A\in\mathcal A=\{0,1,\dots,M-1\}$44

Uniform convergence means

$A\in\mathcal A=\{0,1,\dots,M-1\}$45

with high probability. Once this holds, the empirical maximizer

$A\in\mathcal A=\{0,1,\dots,M-1\}$46

satisfies

$A\in\mathcal A=\{0,1,\dots,M-1\}$47

For the local class

$A\in\mathcal A=\{0,1,\dots,M-1\}$48

with $A\in\mathcal A=\{0,1,\dots,M-1\}$49, the sample complexity is

$A\in\mathcal A=\{0,1,\dots,M-1\}$50

and the paper states that this local-class result is rate-optimal up to logarithmic factors (Yin et al., 2020).

Online policy optimization in episodic tabular MDPs yields a related first-best notion. In that setting, the optimal policy $A\in\mathcal A=\{0,1,\dots,M-1\}$51 simultaneously minimizes $A\in\mathcal A=\{0,1,\dots,M-1\}$52 for all $A\in\mathcal A=\{0,1,\dots,M-1\}$53, and learning quality is measured by regret

$A\in\mathcal A=\{0,1,\dots,M-1\}$54

RPO-SAT combines optimistic $A\in\mathcal A=\{0,1,\dots,M-1\}$55-estimates, an $A\in\mathcal A=\{0,1,\dots,M-1\}$56-OMD policy-improvement step with decaying step size, and a reference-value mechanism satisfying the “Stable at Any Time” property. With high probability, it achieves

$A\in\mathcal A=\{0,1,\dots,M-1\}$57

When $A\in\mathcal A=\{0,1,\dots,M-1\}$58, this matches the information-theoretic lower bound up to logarithmic factors, making it nearly minimax optimal in that regime (Wu et al., 2021).

First-best OPL becomes still broader in incomplete or partially identified models. Russell’s framework defines a policy transform

$A\in\mathcal A=\{0,1,\dots,M-1\}$59

where $A\in\mathcal A=\{0,1,\dots,M-1\}$60 and $A\in\mathcal A=\{0,1,\dots,M-1\}$61 is a known bounded function. If the data are compatible with a family $A\in\mathcal A=\{0,1,\dots,M-1\}$62 of admissible states rather than a single distribution, the policymaker chooses

$A\in\mathcal A=\{0,1,\dots,M-1\}$63

Writing

$A\in\mathcal A=\{0,1,\dots,M-1\}$64

the first-best problem is

$A\in\mathcal A=\{0,1,\dots,M-1\}$65

The paper characterizes ex-ante learnability through PAMPAC-learnability and gives ex-post guarantees for the maximin empirical rule

$A\in\mathcal A=\{0,1,\dots,M-1\}$66

together with $A\in\mathcal A=\{0,1,\dots,M-1\}$67-level set guarantees based on local Rademacher complexity (Russell, 2020).

This broader literature shows that first-best OPL is not confined to fully identified, unconstrained treatment assignment. Depending on the environment, it may denote an empirical welfare maximizer in a treatment study, a knapsack-optimal allocation under budget and coverage constraints, a censoring-aware optimizer of restricted mean survival time, a near-minimax policy in reinforcement learning, or a maximin rule under partial identification.