Semi-online Reinforcement Learning

• Semi-online RL is a paradigm that bridges offline and online reinforcement learning by simulating multi-turn decision-making using offline data while incorporating expert recovery mechanisms.
• It employs dual-level advantage estimation and a Patch Module to maintain long-horizon credit assignment and robust error correction in complex tasks.
• The framework has been validated on GUI automation and similar tasks, demonstrating improved efficiency and policy stability over traditional offline RL methods.

Semi-online reinforcement learning (semi-online RL) is a paradigm that bridges the traditional divide between fully offline and fully online RL by simulating online multi-turn decision-making during training with offline or static data, while integrating mechanisms to recover from divergence between self-generated rollouts and expert demonstrations. This approach enables stable, data-efficient training for agents performing long-horizon tasks, such as GUI automation, where online exploration is expensive or impractical, and offline RL struggles to capture trajectory-level dependencies and robust self-conditioning. The semi-online RL framework introduced by UI-S1 (Lu et al., 15 Sep 2025) typifies the latest advances, providing algorithmic structures, recovery modules, compositional advantage estimation, and practical evaluation metrics tailored for dynamic, real-world sequential interaction.

1. Motivation and Challenges in GUI Automation

Automating graphical user interface (GUI) interaction with RL presents unique challenges due to distributional mismatch and credit assignment complexity. Offline RL methods operate on fixed expert trajectories, yielding stable updates but failing to generalize when agents must rely on their own sequential outputs—the offline-to-online gap stems from traditional training that conditions exclusively on ground-truth histories. In contrast, online RL exposes the agent to the full dynamics of its own outputs but incurs deployment cost, safety risks, and severe sample inefficiency, especially as GUI environments impose sparse, delayed rewards and high-dimensional, complex interaction spaces.

Semi-online RL directly addresses these challenges by aligning training with the real-world, rollout-based execution distribution, injecting the agent's own actions into the trajectory context, but leveraging offline data to minimize the need for online interactions. This enables efficient policy updates that both accommodate multi-turn planning and maintain the stability advantages of supervised or offline-like learning.

2. Core Semi-Online RL Methodology

Semi-online RL as formulated in UI-S1 operates by rolling out policies on offline trajectories, tracking and preserving the model’s own historical actions and reasoning at every dialogue step. The agent is conditioned on a self-generated trajectory context $$\mathcal{H}_t = \{(S_1, a_1, \mathcal{T}_1), \ldots, (S_{t-1}, a_{t-1}, \mathcal{T}_{t-1})\}$$ (where $\mathcal{T}_i$ denotes model reasoning/inferences), and produces candidate actions and thoughts at each turn: $$(a_t, \mathcal{T}_t) \sim \pi(\cdot\,|\,I, S_t, \mathcal{H}_t)$$.

For each episode, transitions are simulated using the expert trajectory whenever the model action agrees with the expert; if not, specialized recovery mechanisms are engaged. The reward structure aggregates multiple criteria, combining formatting, type correctness, and exact action match:

$$r_t = 0.1\, r_\text{format} + 0.4\, \mathbb{1}[r_\text{format} = 1]\, r_\text{type} + 0.5\, \mathbb{1}[r_\text{format} \cdot r_\text{type} = 1]\, r_\text{acc}$$

This composite reward configuration, coupled with discounted returns $R_t = \sum_{k = t}^T \gamma^{k-t} r_k$, enables the agent to internalize both local accuracy and trajectory-level success, which is critical for tasks with substantial temporal dependencies.

Dual-level advantage estimation is performed:

• Step-level: $A^S$ normalizes return differences across rollouts at the same time step.
• Episode-level: $A^E$ assesses full-trajectory return. The combined advantage is $A(a_t) = A^E(\tau) + \omega A^S(a_t)$, with $\omega$ weighting transient versus long-term contributions. Policy optimization utilizes a PPO-like clipped surrogate objective with importance sampling over the semi-online rollouts.

3. Recovery Mechanism: Patch Module

A central innovation is the Patch Module, which addresses the challenge of rollout/expert trajectory divergence—a core bottleneck of strict offline training. Rather than prematurely terminating rollouts upon mismatch, as is typical in pure offline paradigms, the Patch Module adaptively "repairs" the trajectory by injecting the expert action $a_t^*$ (retrieved from the demonstration) in place of the incorrect model action $a_t$, and optionally synthesizes a compatible reasoning trace $\mathcal{T}_t^{\text{patch}}$. This strategy achieves two objectives:

• Retains training continuity and long-horizon credit assignment even when the current policy differs from the expert.
• Exposes the agent to deep trajectory-level supervision, improving robustness to cascading errors.

The module supports multiple patching strategies:

• Thought-Free Patch: direct action substitution without synthetic reasoning.
• Off-Policy Thought Patch: reasoning synthesized by an auxiliary policy.
• On-Policy Thought Patch: reasoning generator is the main policy (maximizing coherence).

By allowing multiple patches per trajectory (regulated by a divergence threshold), the agent experiences diverse recovery scenarios analogous to real-world error recovery, facilitating practical deployment.

4. Performance and Evaluation Metrics

The Semi-Online Performance (SOP) metric was constructed as a reliable proxy for real-world task completion, circumventing the prohibitive cost of exhaustive online deployment. SOP integrates:

• Progress (PG): average step-wise success ratio.
• Task Success Rate (TSR): fraction of fully successful tasks. The aggregate score is $\text{SOP} = (\text{PG} + \text{TSR}) / 2$.

SOP exhibits strong linear correlation with true online success rates (e.g., AndroidWorld, $R^2 = 0.934$), outperforming conventional offline metrics and facilitating efficient development cycles in GUI automation research. This metric prioritizes agent performance when conditioned solely on its internal output history, replicating deployment conditions rather than metric inflation via expert-aligned contexts.

5. Experimental Results and Empirical Findings

Empirical benchmarks on tasks including AndroidWorld, AITW-Gen, AITW-Web, and MiniWob++ validate that UI-S1-7B, trained with semi-online RL, achieves substantial improvements over both baseline supervised finetuning (SFT) and conventional offline RL:

• +12.0% improvement on AndroidWorld, +23.8% on AITW-Gen vs. base model, reflecting effective handling of multi-turn, long-horizon tasks.
• Performance gains are realized not only in multi-turn settings but also in one-step tasks (e.g., GUI Odyssey), confirming no degradation in local action proficiency.
• Supervised fine-tuning benefits (moderately) from additional data, but semi-online RL consistently provides superior gains, especially under complex, non-myopic task evaluation.

The results further demonstrate shorter average completion steps (improved efficiency) compared to offline RL, which in several instances leads to policy degradation when naive rollout termination is used. The Patch Module’s contribution is pronounced in environments that penalize myopic error correction and reward holistic task completion.

6. Applications and Broader Implications

Semi-online RL is particularly well suited to high-stakes, large action-space, and multi-step decision problems where offline data is plentiful but live exploration is restricted. Immediate applications include:

• Mobile and desktop GUI automation, allowing agents to robustly execute complex workflows across diverse applications and dynamic states.
• Robotics for iterative planning, where simulated rollouts need to closely match the sequential, error-prone nature of deployment.
• Dialogue and multi-turn decision making, where the agent's history must reflect its own outputs rather than a curated oracle.

The patch-and-recover strategy, along with SOP-based evaluation, offers a blueprint for domains characterized by cascaded dependencies, sparse rewards, and limited exploratory horizons.

7. Outlook and Future Directions

Potential future research includes extending patching and recovery mechanisms to self-improving curricula, enhancing the sophistication of synthetic context generation, and integrating uncertainty quantification to optimally blend model and expert actions—mitigating compounding error in even deeper decision chains. Additionally, the semi-online RL paradigm could be fruitfully combined with semi-supervised reward learning or semi-pessimistic techniques to further reduce human annotation and improve generalization under distribution shift.

Semi-online RL thus defines a new axis in the RL methodology spectrum, yielding practical, scalable, and robust solutions for domains where pure offline and online methods fail to reconcile stability with long-horizon autonomy.