--- title: Decoupled Atomic Training for GUI Agents url: https://www.emergentmind.com/topics/decoupled-atomic-training type: topic --- # Decoupled Atomic Training for GUI Agents Decoupled Atomic Training (DART) is a reinforcement learning (RL) framework for vision-language model (VLM) based graphical user interface (GUI) agents, characterized by a highly modular asynchronous architecture and adaptive data curation to address the challenges of long-horizon, sparse-reward GUI tasks. DART decomposes the RL pipeline into four independently operating modules, enabling non-blocking communication and significantly improved resource utilization. This strategy addresses inefficiencies endemic to conventional, tightly coupled RL pipelines and underpins state-of-the-art results on the OSWorld benchmark [2509.23866]. ## 1. Motivation and Core Principles Conventional RL pipelines for GUI agents involve rollout generation, data buffering, and model updates proceeding in strict sequence. In such a coupled system, GPU resources and environments frequently remain idle, either awaiting environment responses or model updates. This inefficiency is exacerbated by the intrinsically long-horizon and delayed-feedback nature of GUI tasks, which commonly require tens of interaction steps and yield sparse rewards. Decoupled Atomic Training breaks this paradigm by refactoring the RL system into four atomic, asynchronously communicating modules: - **Environment Cluster**: Hundreds of real or simulated desktops, each running a separate GUI task instance. - **Rollout Service**: A GPU-backed model-inference pool supporting asynchronous inference requests. - **Data Manager**: A centralized, lightweight MySQL-backed trajectory store, also responsible for adaptive data curation logic. - **Trainer**: An independently scheduled step-wise GRPO (Generalized Regularized Policy Optimization) trainer on its own GPU cluster. All inter-module communication proceeds via non-blocking message passing. This architecture ensures that no single component idles awaiting the completion of another's task, maximizing throughput and hardware utilization. ## 2. Asynchronous System Architecture The DART system design is best summarized by its atomic, asynchronous message-passing schedule: ```latex \[ \begin{tikzpicture}[->,>=latex, node distance=3cm] \node (Env) [rectangle,draw] {Env Cluster}; \node (Roll) [rectangle,draw, right of=Env] {Rollout Service}; \node (DM) [rectangle,draw, right of=Roll] {Data Manager}; \node (Train) [rectangle,draw, right of=DM] {Trainer}; % environment requests inference \draw (Env) -- node[above]{RolloutRequest(task, state)} (Roll); % rollout service does inference and returns action \draw (Roll) -- node[below]{Action(state)} (Env); % env streams trajectory chunks \draw (Env) -- node[above]{TrajectoryChunk(...)} (DM); % data manager notifies trainer \draw (DM) -- node[above]{ReadyForTrain(task\_id)} (Train); % trainer pulls & updates \draw (Train) -- node[below]{PullTrajectories} (DM); \draw (Train) -- node[below]{UpdatePolicy} +(0,-2) coordinate (update); % per-worker model sync \draw[dashed] (update) -- node[right]{NewWeights} (Roll); \end{tikzpicture} \] ``` An equivalent sketch in Python-style pseudocode: ```python # Environment Cluster (many parallel workers) while True: task = DataManager.request_task() state = env.reset(task) for step in range(max_steps_dynamic(task)): action = RolloutService.predict(state, task) next_state, reward, done = env.step(action) DataManager.push_trajectory_chunk(task, step, state, action, reward) state = next_state if done: break # Rollout Service (shared model pool) def predict(state, task): # non-blocking inference RPC return model_inference(state, task) # Data Manager def push_trajectory_chunk(...): store_in_db(...) def request_task(): # sample tasks according to dynamic rollout freq return next_task # Trainer (asynchronous) while True: tasks_to_train = DataManager.pop_ready_tasks() for task in tasks_to_train: traj = DataManager.fetch_trajectories(task) grads = compute_stepwise_GRPO_gradients(traj) optimizer.step(grads) new_weights = optimizer.get_weights() RolloutService.async_update_weights(new_weights) ``` Each module proceeds independently, with only per-worker model synchronization during weight updates, avoiding global "stop-the-world" stalls. ## 3. Mathematical Framework: Rollout Sampling and Adaptive Curation ### 3.1 Dynamic Rollout and Trajectory Length Rollout allocation for each task $\tau$ is dynamically determined based on recent success rates $p_\tau$: \[ N(\tau) = \begin{cases} N_{\max}, & p_\tau \le p_{\rm low} \\ N_{\min}, & p_\tau \ge p_{\rm high} \\ N_{\min} + \frac{(1 - p_\tau)(N_{\max}-N_{\min})}{1 - p_{\rm high}}, & p_{\rm low} 0} L_i \right) \] ### 3.2 Multi-level Adaptive Data Curation - **Experience Pool Supplementation**: If all $N(\tau)$ recent online rollouts for a task achieve zero reward, a successful trajectory is drawn from a pre-collected pool $\mathcal P(\tau)$, ensuring at least one positive example per batch. - **High-Entropy Step Selection**: For step $t$, \[ H_{t,i} = -\sum_{v=1}^V p_{t,i,v}\,\log p_{t,i,v}, \quad H_t = \frac{1}{|r_t|+|a_t|} \sum_{i=1}^{|r_t|+|a_t|} H_{t,i} \] Only the top 80% highest entropy steps participate in the GRPO update. - **Truncated Importance Sampling**: \[ w(h,s,a) = \min\left\{\, \frac{\pi^{\rm Train}_{\theta_{\rm old}}(a|h,s)}{\pi^{\rm Rollout}_{\theta_{\rm old}}(a|h,s)},\; C \right\},\quad C > 0 \] This accounts for mismatches between rollout and training policy. The step-wise GRPO objective is: \[ \begin{aligned} \mathcal J(\theta) &= \mathbb E_{(h,s,a,R)\in\mathcal D}\Big[ \mathbb I[H_t\ge\tau_{0.2}]\,w(h,s,a)\, \min\left(\rho_{\theta},\,\mathrm{clip}(\rho_{\theta},1-\epsilon,1+\epsilon)\right)A \Big] \\ &\quad - \beta\,\mathrm{KL}(\pi_\theta^{\rm Train}\,\|\,\pi_\theta^{\rm Ref}) \\ \rho_{\theta} &= \frac{\pi_\theta^{\rm Train}(a|h,s)}{\pi_{\rm old}^{\rm Train}(a|h,s)}, \quad A = \frac{R - \bar R}{\sigma_R} \end{aligned} \] where normalization over rewards is obtained as: \[ \bar R = \frac{1}{|\mathcal D|} \sum_{(h,s,a,R)} R, \quad \sigma^2_R = \frac{1}{|\mathcal D|} \sum_{(h,s,a,R)} (R - \bar R)^2 \] ## 4. System Efficiency Gains Decoupled Atomic Training demonstrates significant efficiency improvements over coupled baselines. The following metrics are reported on identical hardware [2509.23866]: | Metric | Baseline | DART (Ours) | Improvement | |-------------------------|-----------------------|-----------------------|-------------| | Training Throughput | 22.6 actions/min | 43.6 actions/min | 1.9× | | Env Utilization | 12.2% | 67.7% | 5.5× | | GPU Utilization | 29.6% | 46.7% | 1.6× | These gains arise primarily from rollout-wise environment scheduling, fine-grained trajectory assignment, and per-worker model synchronization that eliminates global stalls. ## 5. Empirical Results on the OSWorld Benchmark DART-GUI-7B, initialized from UI-TARS-1.5-7B and trained using the decoupled atomic approach, was evaluated on the 203-task OSWorld-Verified suite with the following results: - **Overall Task Success Rate**: 42.13% - +14.61% absolute over the UI-TARS-1.5-7B baseline (27.52%) - +7.34% over the prior open-source state-of-the-art (OpenCUA-32B, 34.79%) - **Per-task Examples**: - OS tasks: 31.25% → 62.50% (+31.25%) - LibreOffice Writer: 39.13% → 60.86% (+21.73%) - Thunderbird: 40.00% → 60.00% (+20.00%) An ablation study (Pass@1 on 45 tasks) revealed successive component benefits: | Configuration | Pass@1 (%) | |-------------------------|------------| | Baseline (decoupled) | 28.67 | | + Dynamic Rollout (DR) | 50.90 | | + Dynamic Length (DTL) | 66.11 | | + High-Entropy (HE) | 68.33 | | + Distribution Alignment (DA) | 70.55 | | **All components** | 72.28 | This aggregate demonstrates that both the decoupled atomic architecture and the adaptive data pipeline are essential for achieving robust performance. ## 6. Significance and Open-Source Availability Decoupled Atomic Training, through its atomic module design paired with multi-level adaptive data curation, eliminates cross-module stalls and prioritizes learning from challenging, informative data, while robustly correcting for off-policy distribution mismatch. These advances directly translate to improved sample efficiency, hardware utilization, and benchmark performance in multi-turn GUI RL agent training. The complete framework, data, and model checkpoints are made available at [computer-use-agents.github.io/dart-gui], providing reproducibility and extensibility for the open-source agentic RL community [2509.23866]. Source: https://www.emergentmind.com/topics/decoupled-atomic-training