SAO: Single-Rollout Asynchronous Optimization

• Single-Rollout Asynchronous Optimization (SAO) is an RL framework that uses one trajectory per prompt and direct importance sampling for immediate, asynchronous updates.
• It employs strict double-sided token-level clipping and a critic-based value model to manage high variance and policy lag in long-horizon, agentic tasks.
• Empirical studies show SAO outperforms group-based methods in stability and effectiveness, achieving better performance on coding, reasoning, and dynamic evaluation benchmarks.

Single-Rollout Asynchronous Optimization (SAO) is an asynchronous reinforcement learning framework for agentic LLMs that replaces group-wise sampling with single-rollout sampling, using one rollout per prompt, and combines this change with practical value-model training designs and a strict double-side token-level clipping strategy to address stability and off-policy challenges under policy lag (Hou et al., 8 Jul 2026). It is motivated by the inefficiency of synchronous, batch-interleaved RL pipelines on long-horizon agentic tasks and by the observation that group-wise sampling in the widely-used GRPO framework does not naturally fit asynchronous agentic training. SAO is presented as a method for stable asynchronous optimization in settings with sparse, delayed, or step-level rewards, interleaved environment-model actions, and high-variance, non-iid data streams (Hou et al., 8 Jul 2026).

1. Problem setting and motivation

SAO arises in the context of RL post-training for LLMs, where previous RL pipelines were mostly synchronous and batch-interleaved. In agentic coding, reasoning, and tool-use tasks, rollout lengths can vary substantially, producing a straggler effect and leaving resources idle. The SAO formulation targets precisely this regime: long-horizon trajectories, asynchronous data arrival, and optimization under policy lag rather than in a lockstep batch regime (Hou et al., 8 Jul 2026).

A central motivation is the mismatch between group-wise estimators and asynchronous execution. GRPO-style training requires synchronizing a group of responses per prompt for advantage estimation through group mean and variance. In the SAO account, this creates latency and staleness, amplifies off-policy effects, and is incompatible with online or agentic tasks that provide only a single feedback per trajectory. SAO therefore abandons group-wise synchronized sampling in favor of immediate single-rollout optimization (Hou et al., 8 Jul 2026).

The method is also framed against existing asynchronous LLM RL work that emphasized throughput more than optimization stability or task effectiveness. In this setting, the algorithmic issue is not only resource utilization but also whether asynchronous updates remain stable when trajectories are generated from stale policies and when agentic trajectories interleave model-generated actions with environment feedback (Hou et al., 8 Jul 2026).

2. Core optimization scheme

The defining operational choice in SAO is single-rollout sampling: one trajectory per prompt, sent for optimization as soon as the rollout is generated. This removes the need to batch or synchronize groups and reduces delay by improving trajectory freshness. The corresponding cost is increased variance and a stronger off-policy challenge, because each trajectory may have been generated by a different rollout policy version (Hou et al., 8 Jul 2026).

SAO addresses this with direct importance sampling based on rollout log-probabilities recorded at generation time. Instead of requiring an explicit historical policy checkpoint $\pi_{\theta_{\text{old}}}$, it uses the rollout policy probability and the current policy probability to form the correction term

$$r_t(\theta)=\exp\!\big(\log \pi_\theta(a_t\mid s_t)-\log \pi_{\text{rollout}}(a_t\mid s_t)\big).$$

This removes one staleness problem and reduces compute, since the method does not need $\pi_{\theta_{\text{old}}}$ for the importance-sampling correction (Hou et al., 8 Jul 2026).

Optimization stability is further controlled by a strict double-sided token-level clipping and masking rule. Two bounds are defined, $1-\epsilon_\ell$ and $1+\epsilon_h$, and the trust region is strictly restricted to $r_t(\theta)\in[1-\epsilon_\ell,1+\epsilon_h]$. Tokens outside the interval are masked out, meaning their gradient is zero rather than merely clipped. The paper states the objective as

$$L(\theta)=\mathbb{E}_t\!\left[f(r_t(\theta),\epsilon_\ell,\epsilon_h)\,\hat{A}_t\,\log \pi_\theta(a_t\mid s_t)\right],$$

with

$$f(x;\epsilon_\ell,\epsilon_h)= \begin{cases} x, & 1-\epsilon_\ell < x < 1+\epsilon_h \ 0, & \text{otherwise.} \end{cases}$$

A related formulation in the paper presents the reinforcement-learning objective with clipped importance weights and token-level advantages. The important point is that SAO treats token-level off-policy deviations as a primary stability concern rather than a secondary systems artifact (Hou et al., 8 Jul 2026).

3. Value-model design and agentic trajectory semantics

Because single-rollout training has higher variance, SAO makes the critic central rather than optional. Its value-model design includes faster value updates, frozen-attention training, token-level value modeling, and a skip-observation variant of token-level GAE. These components are introduced as practical designs that improve generalization and reduce instability in asynchronous single-rollout RL (Hou et al., 8 Jul 2026).

For every policy step, SAO performs $K>1$ value-model updates, with the summary explicitly giving $K=2$ as an example. This is intended to ensure rapid adaptation of the baseline to the current policy. The value model also freezes attention layers and optimizes only MoE head layers. The stated observation is that attention weights are robust from pretraining and MoE layers suffice for quick adaptation, yielding lower, less noisy gradients and stabilizing value updates (Hou et al., 8 Jul 2026).

The method uses token-level rather than step-level value modeling. The summary reports that token-level value modeling better captures the granularity of logic in agentic chains than step-level variants, and that step-level modeling underperforms in the appendix. This is particularly relevant when trajectories contain tool calls, environment interactions, or multi-turn reasoning traces (Hou et al., 8 Jul 2026).

A further specialization addresses interleaved action and observation tokens. In agentic environments, standard GAE would propagate estimates across both model-generated actions and environment feedback, but observation tokens are not generated by the model. SAO therefore skips observation tokens in advantage computation. For trajectories of the form $$r_t(\theta)=\exp\!\big(\log \pi_\theta(a_t\mid s_t)-\log \pi_{\text{rollout}}(a_t\mid s_t)\big).$$0, the update is computed from one action boundary to the next:

$$r_t(\theta)=\exp\!\big(\log \pi_\theta(a_t\mid s_t)-\log \pi_{\text{rollout}}(a_t\mid s_t)\big).$$1

$$r_t(\theta)=\exp\!\big(\log \pi_\theta(a_t\mid s_t)-\log \pi_{\text{rollout}}(a_t\mid s_t)\big).$$2

The stated rationale is to avoid propagating noise from stochastic environment feedback into the value baseline (Hou et al., 8 Jul 2026).

The paper also states that a large, diverse value-pretraining corpus is needed for effective critic cold start, and that this significantly improves early training and enables stable single-rollout operation. This makes SAO an explicitly critic-based single-rollout method rather than a critic-free one (Hou et al., 8 Jul 2026).

4. Empirical results, stability, and deployment

SAO is reported to train stably for one thousand steps and to consistently outperform GRPO and its variants on agentic coding and reasoning benchmarks including SWE-Bench Verified, BeyondAIME, and IMOAnswerBench (Hou et al., 8 Jul 2026). The summary further lists AIME2025 and HMMT among the math reasoning evaluations.

On SWE-Bench Verified with a Qwen3-30B-A3B backbone, SAO achieves 29.8% accuracy, whereas optimized GRPO with DIS gets approximately 27.0%. On math reasoning benchmarks, the reported values include 97.3% on AIME2025 and 74.8% on BeyondAIME. The summary characterizes these gains as consistent outperformance across all benchmarks considered (Hou et al., 8 Jul 2026).

Stability under asynchrony is emphasized as a distinguishing empirical property. The details state that GRPO typically collapses after approximately 160 steps in asynchronicity, while SAO remains stable throughout training for more than 1000 steps. Ablation studies are reported to show that frozen-attention critic training, faster critic updates, and skip-observation GAE each contribute materially to performance or stability, and that removing any of them significantly reduces one or both (Hou et al., 8 Jul 2026).

The method is also evaluated in a simulated online learning setting in which the model must adapt to changing evolving environments. The summary states that SAO tracks shifting distributions much faster and with lower lag than running-mean baselines because the value-model baseline aligns quickly to changed reward signals. The same source reports successful deployment of SAO in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B), positioning the method as viable at production scale (Hou et al., 8 Jul 2026).

5. Relation to asynchronous RL systems and co-design frameworks

SAO belongs to a broader wave of asynchronous RL work, but its stated emphasis is algorithmic stability and task effectiveness under policy lag rather than only end-to-end pipeline throughput. Related systems in the same period focus on disaggregation, staleness control, and version tracking across large-scale RL training infrastructure (Guan et al., 5 Feb 2026, Hu et al., 29 Apr 2026, Li et al., 19 Jan 2026, Jiang et al., 10 Feb 2026).

System	Setting	Distinct mechanism
RL-VLA$$r_t(\theta)=\exp\!\big(\log \pi_\theta(a_t\mid s_t)-\log \pi_{\text{rollout}}(a_t\mid s_t)\big).$$3 (Guan et al., 5 Feb 2026)	VLA models	fully-asynchronous policy training framework encompassing environment interaction, rollout generation, and actor policy updates
DORA (Hu et al., 29 Apr 2026)	LLM post-training	multi-version streaming rollout with intra-trajectory policy consistency, data integrity, and bounded staleness
StaleFlow (Li et al., 19 Jan 2026)	RL post-training	global consistency protocol plus trajectory and parameter servers for staleness-constrained rollout coordination
FlexMARL (Jiang et al., 10 Feb 2026)	LLM-based MARL	micro-batch driven asynchronous pipeline, experience store, and per-agent policy versioning

These systems illuminate the environment in which SAO was proposed. RL-VLA$$r_t(\theta)=\exp\!\big(\log \pi_\theta(a_t\mid s_t)-\log \pi_{\text{rollout}}(a_t\mid s_t)\big).$$4 decouples environment interaction, rollout generation, and actor updates and validates throughput improvements up to 59.25%, with gains up to 126.67% under optimized allocation strategies (Guan et al., 5 Feb 2026). DORA maintains multiple policy versions concurrently to achieve full bubble elimination without compromising intra-trajectory policy consistency, data integrity, and bounded staleness, reporting throughput up to 2--3 times higher than state-of-the-art systems on open-source benchmarks and 2--4 times acceleration over synchronous training in large-scale industrial applications (Hu et al., 29 Apr 2026). StaleFlow addresses the joint problem of data staleness and skewness by enforcing the explicit constraint

$$r_t(\theta)=\exp\!\big(\log \pi_\theta(a_t\mid s_t)-\log \pi_{\text{rollout}}(a_t\mid s_t)\big).$$5

while reporting up to $$r_t(\theta)=\exp\!\big(\log \pi_\theta(a_t\mid s_t)-\log \pi_{\text{rollout}}(a_t\mid s_t)\big).$$6--$$r_t(\theta)=\exp\!\big(\log \pi_\theta(a_t\mid s_t)-\log \pi_{\text{rollout}}(a_t\mid s_t)\big).$$7 higher throughput than state-of-the-art systems without compromising convergence (Li et al., 19 Jan 2026). FlexMARL, in the multi-agent setting, uses a micro-batch driven asynchronous pipeline with experience-store version tracking and reports up to $$r_t(\theta)=\exp\!\big(\log \pi_\theta(a_t\mid s_t)-\log \pi_{\text{rollout}}(a_t\mid s_t)\big).$$8 speedup and up to $$r_t(\theta)=\exp\!\big(\log \pi_\theta(a_t\mid s_t)-\log \pi_{\text{rollout}}(a_t\mid s_t)\big).$$9 better hardware utilization than existing frameworks (Jiang et al., 10 Feb 2026).

Within this landscape, SAO can be understood as addressing a different layer of the asynchronous RL stack. The systems papers concentrate on orchestration, staleness constraints, and resource utilization; SAO concentrates on what optimization rule remains stable and effective when the rollout unit is a single agentic trajectory arriving asynchronously. This suggests a complementarity between SAO-style algorithmic stabilization and DORA-, StaleFlow-, RL-VLA$\pi_{\theta_{\text{old}}}$0-, or FlexMARL-style systems design, although such integration is not itself claimed in the provided material.

6. Single-rollout RL beyond SAO

SAO is part of a larger movement away from multi-rollout, group-based RL toward single-rollout training, but the surrounding literature shows that “single-rollout” is not a single algorithmic recipe. BASIS is a critic-free post-training algorithm that also samples only one rollout per prompt at each online training step, yet leverages rich information across prompts in the entire batch to improve value-function estimation. Its reported effect is a 69% reduction in MSE relative to REINFORCE++, and it achieves lower MSE with one rollout than group mean estimators with 8 rollouts (Gong et al., 26 May 2026). This provides a contrasting design point to SAO: BASIS seeks variance reduction through batchwise information sharing without an auxiliary value network, whereas SAO makes the critic an essential stabilization component (Hou et al., 8 Jul 2026, Gong et al., 26 May 2026).

A second comparison comes from multimodal RLVR. MSSR is a group-free single-rollout framework for multimodal reasoning that introduces entropy-based advantage shaping to prevent entropy collapse. The paper states that naive single-rollout methods in multimodal settings often lead to training collapse, and that entropy-based shaping is essential for stability. MSSR reaches similar validation accuracy to a group-based baseline with half the training steps and surpasses the group-based baseline when trained for the same number of steps (Liu et al., 20 Dec 2025). This parallel development reinforces a broader pattern: single-rollout efficiency is attractive, but stability requires task-specific controls, whether through SAO’s direct importance sampling, strict double-sided token-level masking, and critic design, or through MSSR’s entropy-shaped advantage (Hou et al., 8 Jul 2026, Liu et al., 20 Dec 2025).

Several common misconceptions are clarified by this literature. First, single-rollout optimization does not imply critic-free training: SAO explicitly requires an improved critic, while BASIS is explicitly critic-free (Hou et al., 8 Jul 2026, Gong et al., 26 May 2026). Second, asynchronous RL is not reducible to throughput engineering: SAO is motivated by the observation that throughput-oriented asynchronous systems leave training stability and task effectiveness underexplored (Hou et al., 8 Jul 2026). Third, asynchrony does not automatically entail unconstrained policy drift: DORA and StaleFlow show that version tracking and bounded staleness can be made first-class design principles, and SAO addresses the remaining algorithmic drift through direct importance sampling and strict token-level trust-region enforcement (Hu et al., 29 Apr 2026, Li et al., 19 Jan 2026).

Taken together, these results position SAO as a specific answer to a narrower question than generic asynchronous RL: how to make asynchronous, single-trajectory updates work reliably for agentic LLM training when feedback is sparse, trajectories are long and interleaved with environment observations, and group-based baselines are structurally mismatched to the training regime (Hou et al., 8 Jul 2026).