Scalar-Stepsize Nonuniform Monte Carlo Optimistic Policy Iteration: A Certified Counterexample

Abstract: Tsitsiklis proved convergence of Monte Carlo optimistic policy iteration under a uniform update structure and identified nonuniform update frequencies as a delicate obstruction. We give a certified negative answer for the natural scalar-stepsize, unnormalized asynchronous state-value recursion with fixed nonuniform state-selection probabilities. In a three-state, two-action discounted MDP, the nonuniform update frequencies induce a diagonally scaled greedy-policy mean field with a certified nonconstant attracting hybrid periodic orbit. With a bounded unbiased geometric-horizon estimator and Robbins--Monro stepsizes, the original stochastic recursion remains trapped near the cycle with positive probability and therefore fails to converge. The example pinpoints a geometric obstruction: uniform sampling gives radial residual contraction, whereas scalar nonuniform sampling anisotropically distorts the residual dynamics and can generate switched attracting cycles.

• The paper constructs and certifies a counterexample showing that scalar stepsize MC-OPI with nonuniform updates fails to converge.
• It rigorously employs analytic and computer-assisted techniques, including 100-digit interval arithmetic, to demonstrate a periodic orbit.
• The findings highlight the need for component-sensitive normalization to ensure convergence in tabular reinforcement learning methods.

Scalar-Stepsize Nonuniform Monte Carlo OPI: Certified Nonconvergence

Introduction and Problem Setting

This paper rigorously establishes a negative result for scalar-stepsize, unnormalized asynchronous Monte Carlo optimistic policy iteration (MC-OPI) under nonuniform state-selection probabilities. The classical convergence result by Tsitsiklis demonstrates global convergence for Monte Carlo OPI when state updates are chosen uniformly. The crux of that result lies in the scalar symmetries introduced by uniform sampling, which yield a residual contraction that suffices for convergence guarantees. However, the current work analyzes the setting where a single scalar stepsize is used across nonuniform state update frequencies—a natural protocol in tabular reinforcement learning—and exposes a geometric obstruction that causes nonconvergence.

The main contribution is the construction and computational certification of a concrete discounted Markov decision process (MDP) with three states and two actions. The paper demonstrates, using both analytic and computer-assisted techniques, that the MC-OPI recurrence in this setting, implemented with fixed, nonuniform state selection and the canonical Robbins–Monro stepsizes $\gamma_t = 1/(t+N)$, is trapped with positive probability in a nonconstant periodic orbit rather than converging to the optimal value.

Technical Framework

The primary object of study is the asynchronous tabular state-value MC-OPI update:

$J_{t+1} = J_t + \gamma_t e_{I_t} (R_t - J_t(I_t))$

where $I_t$ is the sampled state at iteration $t$ with law $q_i = P(I_t = i)$, $\gamma_t$ is a global scalar stepsize, and $R_t$ is a bounded unbiased geometric-horizon Monte Carlo estimator for the cost-to-go. Crucially, the update is not normalized by the relative sampling frequencies.

In the absence of uniform sampling, the limiting mean-field ODE becomes anisotropic:

$\dot{J} = D(J^{\mu(J)} - J)$

where $D = \mathrm{diag}(q_1, \dots, q_n)$, and $\mu(J)$ denotes the (possibly set-valued) greedy policy with respect to $J_{t+1} = J_t + \gamma_t e_{I_t} (R_t - J_t(I_t))$0. The analysis must accommodate the discontinuities in $J_{t+1} = J_t + \gamma_t e_{I_t} (R_t - J_t(I_t))$1 at policy-tie surfaces and the resulting hybrid switching dynamics.

Certified Counterexample Construction

The core result is a certified construction of a three-state, two-action discounted MDP (specified with rational data, $J_{t+1} = J_t + \gamma_t e_{I_t} (R_t - J_t(I_t))$2) and a nonuniform update law $J_{t+1} = J_t + \gamma_t e_{I_t} (R_t - J_t(I_t))$3. Using computer-assisted rigorous numerics, the authors show that this system possesses a strict, locally attracting nonconstant periodic orbit for the piecewise-smooth mean-field dynamics. Moreover, for initial values within the basin of attraction and small enough additive noise, stochastic paths under the MC-OPI recursion remain trapped in a tube surrounding the cycle with strictly positive probability, precluding convergence.

Strong numerical certificates are provided:

• Existence and uniqueness of the periodic orbit are proven with a Krawczyk inclusion argument.
• Transversality and no-sliding conditions are established for Filippov non-smooth analysis at switching surfaces.
• A contraction mapping for the Poincaré return map is rigorously certified in a neighborhood of the orbit, yielding a lower bounded invariant region.
• Detailed code and certified logs, utilizing 100-digit interval arithmetic and exact rationals, are made available for reproducibility.

The stochastic lifting argument follows Faure-Roth and Benaim-Hofbauer-Sorin, showing via martingale maximal inequalities and shadowing that with nonzero probability, the stochastic process remains close to the deterministic attractor forever.

Mechanism and Theoretical Implications

The divergence from the classical OPI convergence picture is attributed to the loss of scalar commutativity in the nonuniform regime. With uniform sampling, the mean-field operator is a contraction aligned with the residual dynamics, precluding oscillatory behaviors. However, in the nonuniform case, the diagonal rescaling $J_{t+1} = J_t + \gamma_t e_{I_t} (R_t - J_t(I_t))$4 distorts these dynamics, introducing anisotropy and enabling the formation of attracting cycles that are separated from the optimal Bellman fixed point by a large margin—numerically confirmed to be $J_{t+1} = J_t + \gamma_t e_{I_t} (R_t - J_t(I_t))$5 in supremum norm in the provided instance.

The certified cycle does not correspond to a pathological effect arising from function approximation, projection, or sample-based model errors; instead, it is an inherent instability in the exact tabular setting induced solely by the combination of scalar stepsize and nonuniform update frequencies.

A key technical boundary is identified: provided the update distribution's condition number $J_{t+1} = J_t + \gamma_t e_{I_t} (R_t - J_t(I_t))$6 is sufficiently small compared to the discount factor, a Tsitsiklis-style argument can still yield convergence. However, the constructed counterexample demonstrates failure far outside this extremely restrictive regime (e.g., $J_{t+1} = J_t + \gamma_t e_{I_t} (R_t - J_t(I_t))$7, $J_{t+1} = J_t + \gamma_t e_{I_t} (R_t - J_t(I_t))$8).

Connections to Literature and Broader Context

This work sharpens the line between tractable and pathological regimes for asynchronous MC-OPI. The negative result complements earlier positive results under uniformity, within-state action balancing, or with component-wise normalization. Specifically, recent extensions (Oliviers et al., 9 Jun 2026) that relax uniformity with appropriate within-state normalizations still avoid the instability presented here, underscoring the necessity of frequency-matched updates for stability.

The phenomenon identified persists even in the exact tabular setting, distinguishing it from previously observed periodic orbits in approximate or projected policy iteration settings.

Open Problems and Directions

Several major open questions are identified:

• The characterization of necessary and sufficient conditions guaranteeing convergence of MC-OPI under general nonuniform asynchronous update schemes remains unresolved.
• Whether normalization or adaptive stepsizing suffices in all discounted tabular MDPs is an open problem.
• Extension and automation of certified counterexample discovery could illuminate the topology of convergence boundaries as state sampling distribution skewness varies.
• Investigation of analogous diagonal distortion mechanisms in Q-value-based MC-OPI, function approximation, or policy-gradient algorithms is warranted.

Conclusion

This paper rigorously demonstrates the existence of a geometric-mechanical obstruction to convergence for MC-OPI with unnormalized scalar stepsizes and nonuniform state selection. The construction of certified cycles, supported by formal verification at the level of exact rational arithmetic, establishes the necessity of component-sensitive normalization schemes for convergence guarantees in asynchronous RL algorithms. These results compel careful design of state and action sampling strategies: naive, unnormalized implementations may exhibit robust nonconvergent dynamics, even in small tabular MDPs.