---
title: 'ClawGUI-RL: Open-Source GUI RL Framework'
url: https://www.emergentmind.com/topics/clawgui-rl
type: topic
---

# ClawGUI-RL: Open-Source GUI RL Framework

ClawGUI-RL is the reinforcement learning (RL) backbone of the ClawGUI open-source framework, designed to enable training, evaluation, and deployment of GUI agents capable of interacting with complex software applications through their user interfaces rather than through programmatic APIs. This system is the first validated, open-source infrastructure supporting both parallel virtual environments and real physical devices for GUI online RL, integrating advanced advantage estimation and dense reward supervision to address environment instability, non-standardized evaluation, and deployment bottlenecks in the field [2604.11784].

## 1. Architecture and Data Flow

ClawGUI-RL is structured around three primary subsystems that together form a scalable, unified RL pipeline:

- **Environment Manager:** Abstracts both emulator-based (Dockerized Android) and real-device backends. The interface unifies environment reset, step execution, and rendering operations via a Python API. Key features include health monitoring, automated crash recovery, and spare-server management.
- **Reward Module:** Computes both a sparse, binary episode-level outcome reward ($R_{outcome} \in \{0, 1\}$) and a dense per-step reward via the Process Reward Model (PRM). Success at the episode level is determined either by system checks (on rooted emulators) or LLM-based judgement (on real devices).
- **RL Trainer:** Supports a policy-gradient loop with Reinforce++, PPO, GRPO, or GiGPO for policy optimization. It aggregates trajectories, estimates hierarchical advantages (as required by GiGPO), and applies gradient-based updates to policy parameters.

Data flows with parallel workers sampling tasks from the Environment Manager, executing policy actions, receiving observations and dense rewards, and, upon episode completion, aggregating the outcome reward. These trajectories are used for hierarchical advantage computation and batched policy updates.

## 2. Formal MDP Specification for GUI Agents

ClawGUI-RL defines GUI interaction as a Markov Decision Process (MDP) $\mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{O}, T, O, R, \gamma)$ where:

- $\mathcal{S}$: Internal states (e.g., device memory, screen image)
- $\mathcal{A}$: Discrete action space (tap, swipe, text input, navigation)
- $\mathcal{O}$: Observation space ($o_t \in \mathbb{R}^{H\times W\times 3}$, RGB screenshot)
- $T:$ State-action transition function ($T: \mathcal{S}\times\mathcal{A}\rightarrow \mathcal{S}$)
- $O:$ State-to-observation mapping
- $R:$ Reward function ($R:\mathcal{S}\times \mathcal{A} \times \mathcal{S}\rightarrow \mathbb{R}$)
- $\gamma$: Discount factor $(\gamma\in[0,1))$

At each timestep $t$, the agent observes $o_t$, executes $a_t \sim \pi_\theta(a \mid o_{0:t})$, transitions according to $T$, and receives $r_t=R(s_t,a_t,s_{t+1})$. Episodes terminate after $T_{max}$ steps or at success/failure.

## 3. Support for Virtual and Real Devices

ClawGUI-RL's Environment Manager unifies two backend types:

- **Virtual Environments:** Each environment is a Docker-based emulator, exposing a REST API for state resets and action stepping. Full root access allows deterministic verification of task success via direct state or UI-tree queries, and spare-containers are automatically rotated on failure.
- **Real Devices:** Managed via ADB over USB/TCP, with task scenarios labeled for LLM-based judgement due to lack of root/system access. Each device responds to input commands, and visual state is captured via remote screencap. The outcome judge relies on LLM prompting against the goal state.

This design allows seamless interleaving of emulator and real-device workers within the same training job, covering both scalable simulation and deployment realism.

## 4. Credit Assignment and Reward Shaping

### GiGPO Advantage Estimation

Group-in-Group Policy Optimization (GiGPO) is centrally integrated, providing a hierarchical advantage estimator:

- **Global (Episode-Level) Advantage:** Across $N$ rollouts of a task, total return $G_i$ is normalized within the group to produce $A^G_i$.
- **Micro (Step-Level) Advantage:** Steps across rollouts are clustered by anchor state; within each micro-group, future returns are normalized to yield $A^M_{i,t}$.
- **Convex Combination:** Final advantage is $A_{i,t} = \lambda A^G_i + (1-\lambda)A^M_{i,t}$, with tunable $\lambda$.

This facilitates fine-grained, step-level credit assignment without value functions, supporting dense and efficient policy learning.

### Process Reward Model and Reward Function

Rewards combine:

- **Episode Outcome Reward:** $R_{outcome} = 1$ for success or 0 otherwise (based on perfect verification or LLM judge).
- **Step-Level PRM Reward:** At each step, a pretrained LLM judge assesses whether $a_t$ advances the task, returning $p_t$; $r^{step}_t = 1$ if $p_t \geq 0.5$, 0 otherwise.

Total return per episode:
$$
G = \sum_{t=0}^{T-1}\gamma^t r_t + \gamma^T R_{outcome}
$$

If training a new PRM, cross-entropy loss between LLM outputs $p_t$ and human/script labels $y_t$ is minimized.

## 5. Training Protocol, Hyperparameters, and Empirical Results

### End-to-End Training Loop

The standard pipeline alternates between parallel trajectory collection and policy updates. The pseudocode is as follows:

```python
Initialize policy parameters θ
Initialize EnvironmentManager with N_env emulator + M real devices
for epoch in 1…N_epochs:
    trajectories = []
    for worker in 1…(N_env+M):  # parallel workers
        obs = env.reset(task_id=random_task())
        worker_traj = []
        for t in 0…T_max−1:
            a_t ~ π_θ(a | obs)
            obs_next, r_step, done, info = env.step(a_t)
            worker_traj.append((obs, a_t, r_step))
            obs = obs_next
            if done: break
        R_outcome = judge_outcome(obs)  # system/LLM judge
        worker_traj[-1] = worker_traj[-1] + (R_outcome,)
        trajectories.append(worker_traj)
    if use_GiGPO:
        A = compute_gigpo_advantages(trajectories, γ, λ)
    else:
        A = compute_grpo_advantages(trajectories, γ)
    loss = 0
    for traj, adv in zip(trajectories, A):
        for (o_t, a_t, r_t), A_t in zip(traj, adv):
            logp = log π_θ(a_t | o_t)
            loss += −A_t * logp
    θ ← θ − η ∇_θ loss + β EntropyBonus(π_θ)
```

### Experimental Hyperparameters and Reward Ablation

The main ClawGUI-2B run employs:

- 64 Docker emulators (no real devices for this run)
- 8×A6000 GPUs (48 GB RAM each)
- GiGPO with group size 8, $\lambda=0.5$
- Sampling temperature $\tau=0.7$
- Learning rate $\eta=1\times10^{-6}$
- Batch size: 8 trajectories per update
- Discount factor: $\gamma=0.99$
- 3 training epochs

Ablation on reward types indicates that dense, step-level PRM rewards with GiGPO yield a +2.6 point absolute gain in MobileWorld GUI-Only Success Rate over binary (episode-level) rewards (17.1% vs. 14.5%).

### Benchmark Comparison

| Model                | MobileWorld SR (GUI-Only) |
|----------------------|--------------------------|
| MAI-UI-2B            | 11.1                     |
| Qwen3-VL-32B         | 11.9                     |
| UI-Venus-72B         | 16.4                     |
| ClawGUI-2B (ours)    | 17.1                     |

ClawGUI-2B exceeds the same-scale MAI-UI-2B baseline by 6.0 points and outperforms larger untrained models, demonstrating that infrastructure and credit shaping have greater impact than model scale alone in this regime.

## 6. Usage Example and Pipeline Launch

Launching a ClawGUI-RL training job requires minimal setup:

```python
from clawgui_rl import EnvironmentManager, RLTrainer, GiGPO

# 1. Launch environments
env_mgr = EnvironmentManager(
    num_emulators=64,
    num_real_devices=0,
    emulator_image="zju/andro-emulator:latest",
    spare_pool_size=10
)

# 2. Build the RL trainer
trainer = RLTrainer(
    env_manager=env_mgr,
    policy_model="MAI-UI-2B",
    judge_model="Qwen3.5-72B",
    algorithm=GiGPO,
    group_size=8,
    gamma=0.99,
    learning_rate=1e-6,
    batch_size=8,
    entropy_coef=0.01
)

# 3. Start training for 3 epochs
trainer.train(num_epochs=3, tasks="MobileWorld-GUIOnly")

# 4. Save checkpoint
trainer.save("clawgui2b_checkpoint.pt")
```
This script provisions all emulators, streams trajectories, computes dense rewards with LLMs, applies GiGPO, and updates the policy network over multiple epochs.

## 7. Limitations and Future Directions

ClawGUI-RL's open-source RL infrastructure is the first to address scalable training across both emulators and real devices. However, notable challenges include:

- On real devices, reward verification currently relies on LLM-based judgement in the absence of root/system signals.
- RL loop is Android-only; extending to iOS requires new device drivers.
- Agents are strictly reactive—there is no learned world model for multi-step look-ahead or planning.
- LLM-based PRM and outcome judges impose significant CPU/GPU overhead.

Proposed future enhancements include device-privacy-preserving RL, GUI-world model learning to enable planning, unified support for CLI and GUI interaction, and extending the environment abstraction to iOS, Linux, Windows, and HarmonyOS ecosystems. This suggests rapid advances are plausible as the community builds on the open ClawGUI-RL foundation [2604.11784].

Source: https://www.emergentmind.com/topics/clawgui-rl