One-Step Policy Optimization (OSPO)

• One-Step Policy Optimization (OSPO) is a family of methods that replace long-horizon credit assignments with local, single-step updates, streamlining policy improvement.
• It covers diverse approaches such as stateless RL, surrogate policy gradients, and native one-step generative policies used in control, diffusion models, and multi-agent systems.
• OSPO applications demonstrate benefits like faster inference speeds, improved sample efficiency, and enhanced stabilization through variance-reduction techniques.

Searching arXiv for papers on One-Step Policy Optimization and closely related formulations. One-Step Policy Optimization (OSPO) denotes a family of policy-learning formulations in which the optimization target is attached to a single decision, a single generative pass, or a single surrogate improvement step rather than to long-horizon temporal credit assignment. In recent work, this label covers stateless reinforcement learning and black-box optimization, where an episode degenerates to one state-action-reward interaction; zeroth-order optimization recast as single-step policy optimization; native one-step generative policies for continuous control and robotics; and step-wise alignment methods for diffusion models, GUI agents, and multi-agent dispatch (Viquerat et al., 2021, Qiu et al., 17 Jun 2025, Gao et al., 4 Apr 2026, He et al., 15 May 2026, Zhao et al., 21 Jul 2025). The acronym is also overloaded: in generative search for LLMs, “OSPO” can specifically denote “Owen-Shapley Policy Optimization,” which is a segment-level credit-assignment method rather than a single-horizon control formulation (Nath et al., 13 Jan 2026).

1. Conceptual scope and terminology

Across the literature, “one-step” is used in at least three technically distinct senses. The first is a single-step environment: the trajectory reduces to one decision with immediate reward, as in stateless deep reinforcement learning and black-box optimization. The second is a single policy-improvement operator: a fixed batch collected under a behavior policy is used to compute one surrogate objective whose maximizer becomes the next policy, an interpretation used to relate TRPO, PPO, and approximate importance-sampling objectives (Qiu et al., 17 Jun 2025, Tomczak et al., 2019). The third is a native one-step generative policy: a latent variable is mapped to an action or action chunk in one forward pass, without iterative denoising or ODE integration at deployment (Zou et al., 28 Jan 2026, Gao et al., 4 Apr 2026).

A further terminological complication is that “OSPO” is not always a generic label. In recommendation-oriented LLM training, "Owen-Shapley Policy Optimization" (Nath et al., 13 Jan 2026) is a named algorithm whose central operation is redistribution of sequence-level advantage across semantically coherent segments; the shared acronym does not imply the same mathematical object as single-step control or black-box search.

Interpretation of OSPO	Core mechanism	Representative papers
Single-step / stateless RL	One state, one action, immediate reward	(Viquerat et al., 2021, Qiu et al., 17 Jun 2025)
One-step policy-improvement operator	One surrogate optimization step from a fixed batch	(Tomczak et al., 2019)
Native one-step generative policy	One latent-to-action or latent-to-chunk mapping at inference	(Zou et al., 28 Jan 2026, Gao et al., 4 Apr 2026, Chen et al., 31 Jul 2025)
Acronym-specific named method	Shapley–Owen credit redistribution for LLMs	(Nath et al., 13 Jan 2026)

2. Single-step RL reduction and surrogate policy gradients

In the stateless formulation, the environment has a fixed state $s_0$, the agent samples one action $a$, and the episode ends immediately. Depending on indexing convention, papers write this as $T=0$ or $T=1$, but the substance is the same: there is no temporal credit assignment, no state transition chain, and the return is the immediate reward. Under this reduction, vanilla policy gradient collapses to the REINFORCE score-function form

$$\nabla_\theta J(\theta) = \mathbb{E}_{a\sim \pi_\theta} \left[ \nabla_\theta \log \pi_\theta(a)\, \hat R(a) \right],$$

with $\hat R(a)$ taken either as the raw reward or as a normalized/clipped variant used for variance reduction (Viquerat et al., 2021, Qiu et al., 17 Jun 2025).

The paper "Policy-based optimization: single-step policy gradient method seen as an evolution strategy" (Viquerat et al., 2021) gives a canonical construction. Policy-Based Optimization (PBO) treats black-box minimization as a one-step RL problem with a state-independent Gaussian policy $\pi_\theta(a)=\mathcal N(\mathbf m,\mathbf C)$, where $\mathbf m$ and the full covariance $\mathbf C$ are produced by three neural networks parameterizing mean, standard deviations, and correlations. The loss is

$$L(\theta)= \mathbb{E}_{a\sim \pi_\theta} \left[ \log \pi_\theta(a)\,\hat R(a) \right], \qquad \hat R(a)=\max\!\left(\frac{r(a)-\mu_r}{\sigma_r},0\right),$$

and the update is stochastic gradient ascent with Adam. In this setting, one “episode” is exactly one function evaluation.

A distinct but related notion appears in "Policy Optimization Through Approximate Importance Sampling" (Tomczak et al., 2019). There, “one-step policy optimization” is not a one-step environment but a one-step improvement map: trajectories from a behavior policy $a$0 define a surrogate objective $a$1, and the next policy is obtained by approximately maximizing that surrogate once per batch. This view places PPO- and TRPO-style methods inside a broader OSPO family of single surrogate updates, with explicit bias–variance control through smoothed importance weights.

3. Search distributions, evolution strategies, and zeroth-order optimization

One of the most developed OSPO interpretations is the search-distribution view. In PBO, the policy is simultaneously the RL policy and the search distribution over candidate solutions, which makes the method conceptually adjacent to evolution strategies. Like CMA-ES, it samples a population from a Gaussian distribution, evaluates objective values, and updates the distribution toward better regions; unlike CMA-ES, its mean and full covariance are updated by stochastic policy gradient through neural parameterization rather than by closed-form covariance-update heuristics (Viquerat et al., 2021).

The paper makes this relationship explicit. PBO uses a full-covariance Gaussian, discards negative normalized rewards by clamping at $a$2, and uses several recent generations with exponential decay, which is explicitly likened to CMA-ES evolution-path-style temporal smoothing. This suggests that, in one important lineage, OSPO is best understood as policy-gradient optimization of a search distribution rather than as conventional sequential RL (Viquerat et al., 2021).

"Zeroth-Order Optimization is Secretly Single-Step Policy Optimization" (Qiu et al., 17 Jun 2025) sharpens that connection into an equivalence theorem. For additive perturbation policies of the form

$a$3

the single-step policy objective

$a$4

is exactly the smoothed zeroth-order objective $a$5. The standard two-point finite-difference estimator

$a$6

is shown to be mathematically identical to a single-step REINFORCE estimator with baseline $a$7 in the Gaussian case, and equivalent up to a constant scaling for sphere and coordinate perturbations. Under this view, smoothing is the expectation under a stochastic policy, perturbation directions are policy actions, and baseline subtraction in finite differences is the REINFORCE baseline.

The same paper proposes ZoAR, which imports RL variance-reduction devices into ZOO: an averaged baseline from recent evaluations and query reuse analogous to replay. The result is a particularly clear OSPO template: a one-step stochastic policy over perturbations, a black-box reward, and a variance-controlled score-function update (Qiu et al., 17 Jun 2025).

4. Native one-step generative policies for control and robotics

A second major OSPO lineage replaces one-step horizon reduction with one-step generation. Here the policy is a stochastic or latent-conditioned generator that maps noise to action in one forward pass, avoiding the iterative denoising of diffusion and flow-matching policies. The core claim across this literature is that multimodal policies need not require multi-step inference if refinement is absorbed into the training objective.

MeanFlow-based policies are central in this line. "One Step Is Enough: Dispersive MeanFlow Policy Optimization" (Zou et al., 28 Jan 2026) defines one-step sampling as

$a$8

and combines it with dispersive regularization and PPO-style RL fine-tuning. The paper reports true one-step generation, $a$9 Hz control, and $T=0$0-$T=0$1 inference speedup. In offline RL, "Latent Policy Steering through One-Step Flow Policies" (Im et al., 5 Mar 2026) uses a frozen one-step MeanFlow policy $T=0$2 as a behavior-constrained generative prior, while a latent actor $T=0$3 is optimized by backpropagating original-action-space Q-gradients through the differentiable one-step decoder: $T=0$4 This removes latent-critic distillation and structurally constrains the optimized policy to the dataset manifold.

Score-based and mirror-descent variants develop the same one-step thesis in online control. "Score-Based One-step MeanFlow Policy Optimization" (Kim et al., 22 May 2026) builds a one-step MeanFlow actor whose target velocity field is derived directly from the Q-function via score estimation and a probability-flow ODE, yielding a single generation step for continuous control. "One-Step Flow Policy Mirror Descent" (Chen et al., 31 Jul 2025) embeds one-step flow and MeanFlow policies into policy mirror descent, with the PMD target

$T=0$5

and proves that the one-step discretization error of straight interpolation flow matching is bounded by the target action-distribution variance. In that analysis, one-step sampling is most accurate precisely when the learned policy has become low-variance and near-deterministic.

Drift-based formulations push the same principle further. "Drifting Field Policy: A One-Step Generative Policy via Wasserstein Gradient Flow" (Koo et al., 8 May 2026) frames policy improvement as reverse-KL Wasserstein-2 gradient flow toward a soft target policy and derives a tractable top-$T=0$6 surrogate resembling behavior cloning on critic-selected actions. "Drift-Based Policy Optimization: Native One-Step Policy Learning for Online Robot Control" (Gao et al., 4 Apr 2026) formalizes a fixed-point drifting objective in which a one-step generator is regressed toward drifted targets constructed from attraction to expert or high-value samples and repulsion from its own current outputs. DBPO then adds a Gaussian stochastic interface and an anchor regularizer so that PPO can optimize the pretrained one-step backbone without sacrificing 1‑NFE deployment; the reported real-world control frequency is $T=0$7 Hz.

5. Step-wise alignment, multi-agent dispatch, and other domain-specific uses

The OSPO idea also appears in settings where the environment is sequential but the update is intentionally concentrated on one local unit of optimization. In "Flash-GRPO: Efficient Alignment for Video Diffusion via One-Step Policy Optimization" (He et al., 15 May 2026), one diffusion timestep per prompt is sampled for policy optimization, while other timesteps are run deterministically to produce the final video. Two mechanisms make this viable: iso-temporal grouping, which removes timestep-confounded variance by keeping all trajectories within a prompt-group at the same sampled timestep, and temporal gradient rectification, which divides by an analytically derived $T=0$8 factor so that gradients from different timesteps have comparable scale. The result is approximately $T=0$9 training acceleration relative to full-trajectory GRPO under the reported budget.

For multi-turn LLM agents, OSPO-like reasoning often takes a step-level form. STEP, "Success-Rate-Aware Trajectory-Efficient Policy Optimization" (Chen et al., 17 Nov 2025), replaces trajectory-level GRPO with success-rate-aware task resampling, step decomposition, and step-level GRPO augmentation. It computes advantages weighted by $T=1$0, where $T=1$1 is a smoothed per-task success rate, and forms local step groups for low-success tasks. "Building Autonomous GUI Navigation via Agentic-Q Estimation and Step-Wise Policy Optimization" (Wang et al., 14 Feb 2026) likewise trains an agentic-Q model $T=1$2 as a binary success-probability estimator from self-generated trajectories and then runs step-wise GRPO, RLOO, or REINFORCE++ on per-step returns $T=1$3. Both works are one-step in the sense that policy optimization is performed over step-level samples rather than over full trajectories.

A more literal single-step reduction appears in multi-agent dispatch. "One Step is Enough: Multi-Agent Reinforcement Learning based on One-Step Policy Optimization for Order Dispatch on Ride-Sharing Platforms" (Zhao et al., 21 Jul 2025) shows that, under a homogeneous fleet assumption, the standard advantage can be approximated by a normalized one-step reward difference across agents: $T=1$4 with an additional utility-dispersion penalty yielding

$T=1$5

This removes value-function estimation entirely and makes PPO-style updates depend only on one-step group-relative rewards.

The acronym-specific use should be distinguished from all of the above. "Owen-Shapley Policy Optimization (OSPO): A Principled RL Algorithm for Generative Search LLMs" (Nath et al., 13 Jan 2026) keeps a trajectory-level sequence reward but redistributes the resulting advantage across segments and tokens using Shapley–Owen attributions. Its defining equation is

$T=1$6

where $T=1$7 is a normalized token attribution and $T=1$8 is the group-relative sequence advantage. Here “OSPO” names a cooperative-game-theoretic credit-assignment algorithm, not a single-step environment or one-step generative policy.

6. Assumptions, limitations, and recurrent trade-offs

The exactness of OSPO formulations is usually narrow. For the ZOO equivalence, the correspondence between zeroth-order finite differences and single-step policy optimization is exact only for a one-step horizon with additive perturbation policies, and the query-reuse variant targets an average of past smoothed gradients rather than the current one, introducing a standard bias–variance trade-off (Qiu et al., 17 Jun 2025). PBO exhibits related issues: off-policy importance weighting was reported unstable near optima, and on multimodal objectives such as Griewank it tends to get stuck similarly to CMA-ES (Viquerat et al., 2021).

One-step generative policies replace iterative inference with stronger assumptions about geometry or variance. FPMD’s main theoretical bound states that the one-step flow discretization error is controlled by the target action-distribution variance, so low-variance policies are precisely the regime in which one-step inference is most justified (Chen et al., 31 Jul 2025). LPS requires spherical latent geometry and a differentiable one-step MeanFlow decoder to keep latent steering within the prior’s typical set (Im et al., 5 Mar 2026). DBPO requires an explicit anchor regularizer to keep PPO updates near the pretrained one-step manifold (Gao et al., 4 Apr 2026). These recurrent design choices suggest that native one-step control is easiest when the learned action manifold is already well structured.

Several domain-specific limits are structural rather than numerical. The ride-sharing OSPO derivation depends on cooperative homogeneity and approximate equality of per-agent value functions; the paper explicitly states that the method is not appropriate when some agents must systematically sacrifice their own utility for global benefit or when agent roles are heterogeneous (Zhao et al., 21 Jul 2025). The GUI agentic-Q framework propagates the terminal success label $T=1$9 to every intermediate step, which gives a practical supervised signal but is still a coarse process label (Wang et al., 14 Feb 2026). STEP and Flash-GRPO show that naively compressing multi-step optimization to a single local unit can create misleading credit signals or unstable gradients unless additional grouping and normalization mechanisms are introduced (Chen et al., 17 Nov 2025, He et al., 15 May 2026).

Taken together, these works suggest that OSPO is less a single algorithm than a design principle. In one family it means that the environment itself is one-step; in another, that the policy generator is one-step; in another, that the update operator is one-step relative to a fixed batch or a single timestep. The unifying theme is the replacement of long-horizon or multi-stage refinement by an explicitly constructed local objective, with success depending on how accurately that local objective preserves the geometry, variance structure, and trust-region properties of the original problem.