Monte Carlo Optimistic Policy Iteration

• Monte Carlo Optimistic Policy Iteration is a simulation-based policy iteration method that uses incomplete Monte Carlo evaluations and immediate policy improvements.
• It operates on-policy by combining exploring starts, asynchronous updates, and partial state evaluations to address convergence in stochastic shortest path and discounted MDP setups.
• Recent studies emphasize that proper stepsize scheduling, lookahead-based improvement, and uniform action updates are crucial to ensure convergence and avoid nonoptimal cyclic behaviors.

Monte Carlo optimistic policy iteration (MC-O-PI) denotes a class of simulation-based policy-iteration schemes in which a current policy is evaluated only approximately, using Monte Carlo returns or closely related rollout targets, and the policy is then improved immediately—typically greedily or via lookahead—before evaluation has converged. In finite Markov decision processes, the term encompasses Monte Carlo Exploring Starts, stochastic shortest path recursions, initial-visit and first-visit action-value updates, and asynchronous variants. The central theoretical issue has been to determine when this partial, noisy form of policy iteration converges to the optimal value function and when seemingly minor changes in update structure destroy convergence [1808.08763] [2301.09709] [2606.10580] [2606.15978].

1. Canonical formulation and algorithmic structure

In its canonical state-value form, MC-O-PI alternates between a greedy policy-improvement step and a Monte Carlo policy-evaluation step. For a finite stochastic shortest path problem with state set (S={1,\dots,n}), action sets (U(i)), one-step cost (g(i,u)), and a cost-free absorbing termination state (0), the policy at iteration (t) is chosen greedily with respect to the current value estimate (J_t):
[
\mu_t(i)\in \arg\min_{u\in U(i)}\Big{g(i,u)+\sum_j P_{i,j}(u)J_t(j)\Big}.
]
Monte Carlo evaluation then simulates, for each state (i), a single trajectory starting at (i), following (\mu_t) until termination, and forms an unbiased sample return (J_{\mu_t}(i)+w_t(i)). The value update is
[
J_{t+1}(i)=(1-\gamma_t)J_t(i)+\gamma_t\big(J_{\mu_t}(i)+w_t(i)\big),
]
with deterministic stepsizes satisfying
[
\sum_{t=0}^\infty \gamma_t=\infty,\qquad \sum_{t=0}^\infty \gamma_t^2<\infty.
]
The method is called “optimistic” because policy improvement is performed before (J_t) has converged to (J_{\mu_t}); after one incomplete Monte Carlo evaluation, the algorithm recomputes a greedy policy and repeats [1808.08763].

A closely related action-value formulation is used in recent analyses of initial-visit MC-O-PI. There, the algorithm maintains (q_k(s,a)), chooses a deterministic greedy policy
[
\pi_k \in P(q_k)
:= \Big{ \pi \in \mathcal P :
q_k(s,\pi(s)) = \max_{a\in\mathcal A(s)} q_k(s,a)\;\forall s\Big},
]
samples an initial state-action pair ((s_0,a_0)) from a start distribution, simulates one episode under (\pi_k), and updates only the initial or first-visited state-action pairs through
[
q_{k+1}(s,a)
= q_k(s,a) + \alpha_k u_k(s,a)\bigl( q_{\pi_k}(s,a) + v_k(s,a) - q_k(s,a)\bigr).
]
Here (u_k(s,a)\in{0,1}) is the visit indicator and (v_k(s,a)) is a zero-mean noise term with bounded conditional second moment. The same papers distinguish initial-visit and first-visit Monte Carlo, and explicitly exclude every-visit updates because they induce bias in general MDPs [2606.10580].

Two structural features recur across the literature. First, MC-O-PI is usually on-policy: the trajectories used for evaluation are generated under the same policy that is about to be improved. Second, the algorithm is a stochastic approximation recursion rather than an exact dynamic-programming method. This places convergence on the interplay among Bellman monotonicity, exploration or exploring starts, stepsize schedules, and the geometry induced by asynchronous updates.

2. Stochastic shortest path theory and early convergence results

A major line of work studies MC-O-PI for the undiscounted stochastic shortest path (SSP) problem, where (\alpha=1) and termination replaces discounting as the mechanism ensuring finiteness. In one influential formulation, every stationary policy is assumed to be proper: from any starting state, the termination state is reached with positive probability within at most (n) steps, hence almost surely, and all cost-to-go functions (J_\mu) are finite. Under this assumption, the optimal Bellman operator
[
(TJ)(i)=\min_{u\in U(i)} \Big{ g(i,u) + \sum_{j=1}^n P_{i,j}(u)\,J(j)\Big}
]
has a unique fixed point (J^*), value iteration converges to (J^*), and for every proper policy (\mu), the policy-evaluation operator (T_\mu) has unique fixed point (J_\mu). The Monte Carlo OPI update above is then shown to generate a bounded sequence ({J_t}) whose Bellman residual
[
c_t = \max_{i\in S}\big((TJ_t)(i)-J_t(i)\big)
]
satisfies (\limsup_{t\to\infty} c_t \le 0) almost surely, leading to the theorem that (J_t\to J^*) almost surely. The same paper analyzes (TD(\lambda))-based OPI and proves almost sure convergence for every (\lambda\in[0,1]), with (\lambda=1) reducing to Monte Carlo and (\lambda=0) reducing to a value-iteration-like update [1808.08763].

A parallel SSP treatment appears under the name Monte Carlo Exploring Starts (MCES), explicitly identifying MCES with optimistic policy iteration. The algorithm updates
[
J_{t+1} = (1-\gamma_t) J_t + \gamma_t \big(J^{\mu_t} + w_t\big),
\qquad
T_{\mu_t}J_t = TJ_t,
]
and studies convergence in the undiscounted SSP setting with a terminal state (0). Under the strong assumption that all policies are proper and Robbins–Monro stepsizes, the Bellman error’s positive part vanishes asymptotically, the iterates are trapped between upper and lower stochastic-approximation envelopes converging to (J^*), and the method converges almost surely to the optimal cost vector. The same analysis also treats asynchronous exploring-starts variants and nonuniform state sampling, provided state-dependent stepsizes are used so that each state is updated infinitely often with diminishing effective stepsize [2007.10916].

These results established a core principle: in SSPs, Monte Carlo policy evaluation does not need to be exact at each policy-iteration step. Properness, bounded one-step costs, and standard stochastic-approximation conditions are sufficient to make partial evaluation plus immediate greedy improvement asymptotically safe. No explicit rates of convergence are given in these analyses; the guarantees are qualitative and almost sure.

3. Exploring starts, asynchronous updates, and partial-state evaluation

The practical appeal of MC-O-PI lies in not having to evaluate a policy on the full state space at every iteration. Several later papers therefore examine asynchronous versions in which only a subset of states or state-action pairs is updated from each sampled trajectory.

One discounted-MDP result considers the case where every stationary policy induces the same graph structure, recurrent classes are policy-independent, and the graph obtained after collapsing recurrent classes is acyclic. The associated stochastic OPI updates all states visited during each simulated trajectory, rather than updating only the starting state. At iteration (t), a greedy policy (\mu_t) is computed from (J_t); a trajectory is generated under (\mu_t) from a random initial state; for each visited state (i), the first-visit tail return
[
\tilde J^{\mu_t}(i)
= \sum_{k=N_t(i)}^\infty \alpha^{k-N_t(i)}
c\big(X_k^{\mu_t},\mu_t(X_k^{\mu_t})\big)
]
is formed; and every visited state is updated by stochastic approximation. Under the acyclic transient-state assumption, convergence is proved by first showing convergence on recurrent classes and then propagating it upward through a reverse topological order on transient states. The outcome is almost sure convergence (J_t\to J^*) for both deterministic state-independent stepsizes and state-dependent visit-counter stepsizes [2102.00030].

A more general recent result revisits the classical “uniform state-action updates” requirement. In its strict form, that condition means the update distribution (\mu_k) is uniform over all state-action pairs, which for initial-visit MC-O-PI effectively requires uniform initialization over the entire state-action space. This is precisely the condition used in Tsitsiklis-style analyses, but it is impractical in large or unknown state spaces. A 2026 paper shows that this requirement can be relaxed substantially: for initial-visit MC-O-PI, it is enough that updates be uniform only over the actions within each state,
[
\mu_k(s,a)=\mu_k(s,a') \quad \forall s,\ \forall a,a'\in\mathcal A(s),
]
while the frequencies with which different states are initialized may vary arbitrarily. The proof abandons the classical commutativity argument and instead shows that the mean-field dynamics generate monotonically improving policies; noise is then controlled by a lock-in argument based on seed, growth, and lock-in events. Under strict exploring starts, tie-breaking that assigns every greedy action probability at least (\beta>0), and Robbins–Monro stepsizes satisfying an additional comparability condition, the iterates converge almost surely to (q_*) [2606.10580].

These asynchronous results clarify that “Monte Carlo” in MC-O-PI need not mean a full sweep over the state space. Partial-state evaluation can be enough, but only when the update geometry remains compatible with policy improvement. This suggests that the essential question is not simply how many states are updated, but how the induced update frequencies interact with Bellman monotonicity and stepsize normalization.

4. Lookahead-based improvement and the single-trajectory convergence theorem

A longstanding open problem concerns the most literal form of Monte Carlo policy iteration: estimate the value of a policy from data collected from a single sample path obtained by implementing that policy, and then improve greedily with respect to the estimate. A 2023 result resolves this problem for a first-visit variant, but only when the policy-improvement step uses lookahead rather than simple greedy improvement.

In the tabular discounted setting, let (V_k) be the current value estimate. The improved policy (\mu_{k+1}) is not chosen by the one-step greedy relation (T_{\mu_{k+1}}V_k = TV_k). Instead, it is an (H)-step lookahead policy satisfying
[
T_{\mu_{k+1}} T^{H-1} V_k = T^H V_k.
]
A single trajectory is then generated under (\mu_{k+1}) from an exploring-start distribution with full support. For each visited state (i), first-visit Monte Carlo evaluation provides an (m)-step rollout target
[
T_{\mu_{k+1}}^m T^{H-1}V_k(i) + w_k(i),
]
and the value update is performed only on the visited states using diminishing stepsizes. The key technical step is a bound on the discrepancy between the lookahead policy’s exact value and the lookahead base value:
[
|J^{\mu_{k+1}} - T^{H-1}V_k|_\infty
\le
\frac{\alpha^{H-1}}{1-\alpha}|TV_k - V_k|\infty.
]
This leads to a contraction estimate for the composite operator (V \mapsto T{\mu(V)}^m T^{H-1}V). If
[
\alpha^{H-1} + 2(1+\alpha^m)\frac{\alpha^{H-1}}{1-\alpha} < 1,
]
then the iterates converge almost surely to (J^*) [2301.09709].

The same paper extends the argument to linear function approximation. With (V_k = \Phi\theta_k), the policy is still obtained from (H)-step lookahead, but policy evaluation may be carried out by first-visit Monte Carlo least squares, finite-step gradient descent, or another routine satisfying a uniform error bound. If the evaluation method obeys
[
|\mathbb{E}[\Phi\theta^{\mu_{k+1}} \mid \mathcal F_k] - T^{H-1}\Phi\theta_k|_\infty
\le
\kappa |T^{H-1}\Phi\theta_k - J^{\mu_{k+1}}|_\infty + \delta_{\text{app}},
]
then the paper proves asymptotic bounds on both the value-function error and the induced policy-performance gap; the resulting policies perform close to optimal within a function-approximation error term [2301.09709].

A common misconception is that the simple first-visit Monte Carlo ES scheme with one-step greedy improvement is fully settled by this result. The theorem is more specific: convergence is proved for a first-visit, single-sample-path policy-iteration scheme provided that policy improvement uses lookahead rather than a simple greedy step [2301.09709]. This leaves the role of lookahead as a structural regularizer rather than a mere computational enhancement.

5. Beyond uniformity: positive results and a certified counterexample

Recent work gives both a broadened positive theory and a sharp impossibility result. Taken together, these papers delineate the boundary between benign and destructive nonuniform update structure.

A 2026 positive result proves that initial-visit MC-O-PI converges even when different states are updated at arbitrary frequencies, as long as updates are uniform over actions within each state. A second 2026 paper shows that a natural alternative—scalar-stepsize, unnormalized asynchronous state-value recursion with fixed nonuniform state-selection probabilities—can fail dramatically. In that counterexample, a three-state, two-action discounted MDP with update frequencies (q=(0.02,0.18,0.80)) induces a diagonally scaled greedy-policy mean field with a certified nonconstant attracting hybrid periodic orbit. With a bounded unbiased geometric-horizon estimator and Robbins–Monro stepsizes, the stochastic recursion remains trapped near that cycle with positive probability and therefore fails to converge. The geometric explanation is that uniform sampling yields radial residual contraction, whereas scalar nonuniform sampling anisotropically distorts the residual dynamics and can generate switched attracting cycles [2606.15978].

The current landscape can be summarized as follows:

Setting	Update structure	Asymptotic result
SSP Monte Carlo OPI	Synchronous state-value Monte Carlo evaluation	(J_t \to J^*) almost surely
SSP MCES	Exploring starts; asynchronous variants with component-wise stepsizes	(J_t \to J^*) almost surely
Discounted MDP with acyclic transient structure	All states visited in each trajectory are updated	(J_t \to J^*) almost surely
Initial-visit MC-O-PI	Uniform only over actions within each state	(q_k \to q_*) almost surely
Scalar-stepsize nonuniform asynchronous MC-O-PI	Fixed nonuniform state-selection probabilities, unnormalized	Failure to converge with positive probability

This comparison suggests that nonuniformity itself is not the decisive obstruction. A plausible implication is that the critical issue is whether the nonuniform update frequencies can be absorbed into a policy-improving mean field. When action symmetry within a state is preserved, policy monotonicity can still be recovered; when a single scalar stepsize is combined with strongly nonuniform state sampling, the diagonal distortion can overwhelm the residual geometry and create attracting cycles [2606.10580] [2606.15978].

6. Broader interpretations and related variants

The core MC-O-PI template—partial simulation-based evaluation followed by immediate policy improvement—also appears outside the tabular convergence literature. A classic on-line policy-improvement method evaluates, at each decision state (x), the long-term expected reward of every candidate action by Monte Carlo simulation while following a fixed base policy thereafter. The chosen action is
[
P'(x)=\arg\max_{a\in\mathcal A(x)} \hat V_P(x,a),
]
where (\hat V_P(x,a)) is the Monte Carlo estimate of the action value under the base policy. Applied to backgammon, this approach yields a substantial reduction, by as much as a factor of 5 or more, in the error rate of the base players, across initial policies ranging from a random policy to TD-Gammon [2501.05407]. Although this setting is presented as on-line Monte-Carlo search rather than as MC-O-PI, its structure is the same: partial Monte Carlo evaluation, then immediate greedy improvement at the current state.

The simulation-based OPI perspective has also been extended to multi-agent MDPs with Kullback–Leibler control cost. There, the greedy policy-improvement step has an explicit Boltzmann form
[

\pi_{P_0}(s' \mid s)

\frac{P_0(s' \mid s)\exp(-\gamma V(s'))}
{\sum_{s''} P_0(s'' \mid s)\exp(-\gamma V(s''))},
]
and policy evaluation uses finite-horizon (m)-step TD rollouts. Both synchronous and asynchronous versions converge asymptotically to the optimal value function and an optimal joint policy [2410.15156]. This suggests that Monte Carlo optimistic policy iteration is best regarded as a design pattern—greedy or softened improvement combined with simulation-based partial evaluation—rather than as a single fixed recursion.

At the same time, the convergence theory remains sharply conditional. Positive results require properness, acyclic transient structure, uniformity over actions within each state, sufficiently deep lookahead, or analogous structural devices. Negative results show that scalar nonuniform sampling can create stable nonoptimal cycles. The mature view of MC-O-PI is therefore neither that it is generically unstable nor that it converges under arbitrary asynchronous Monte Carlo updates. Rather, its behavior is now understood to depend on a precise interaction among rollout sampling, update normalization, policy-improvement depth, and the Bellman geometry of the underlying control problem.