CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agents

Abstract: Reinforcement learning with verifiable rewards (RLVR) has driven breakthroughs in domains such as math, tool-use, and software engineering, yet its extension to computer-use agents (CUAs) has been bottlenecked by the scarcity of scalable training data with deterministic rewards. Constructing such data for CUAs requires consistent task instruction, executable environment, and verifiable reward. However, hand-curated benchmarks achieve high reward fidelity but cover few applications and LLM-as-judge-based datasets scale broadly but lack reliable verification. We present CUA-Gym, a scalable pipeline that co-generates task instructions, environment states, and reward functions. Concretely, a Generator agent constructs the initial and golden environment states, and a separate Discriminator agent writes the reward function from the task specification. An orchestrator agent drives the two through iterative rounds upon execution. Generated tuples then pass a final filter combining LLM majority voting and agent rollouts, ensuring quality beyond the per-task adversarial loop. To address the scarcity of training environments, we further synthesize CUA-Gym-Hub, a broad suite of high-fidelity mock web applications grounded in real-world software-use distributions, expanding the scale of CUA RLVR data by magnitude. Using this pipeline, we construct CUA-Gym, a dataset of 32,112 verified RLVR training tuples grounded in 110 environments. Trained with GSPO on CUA-Gym, our CUA-Gym-A3B and CUA-Gym-A17B achieve 62.1% and 72.6% on OSWorld-Verified, outperforming prior open-source CUAs at comparable scales, with performance scaling smoothly in both data volume and environment diversity. The same checkpoints also improve on the held-out WebArena benchmark, indicating transfer beyond the training environments. We will open-source the full synthesis pipeline, dataset, CUA-Gym-Hub environments, and models.

• The paper introduces CUA-Gym, an automated adversarial pipeline that co-generates tasks, environments, and verifiable rewards for computer-use agents.
• The paper demonstrates that RLVR-trained policies on CUA-Gym data significantly outperform SFT models, achieving up to 72.6% success on benchmarks.
• The paper shows that diversifying both data and environment scales, through mock web and desktop setups, significantly enhances RL learning and generalization.

CUA-Gym: Scalable RLVR Training for Computer-Use Agents

Motivation and Problem Formulation

CUA-Gym addresses the bottleneck in scaling reinforcement learning with verifiable rewards (RLVR) for computer-use agents (CUAs). While RLVR has established itself as the core post-training paradigm for domains like mathematics and software engineering, its application to CUAs has lagged due to the scarcity of high-fidelity, large-scale training data with deterministic, programmatically checkable rewards.

Constructing effective RLVR data for CUAs requires not only well-specified task instructions, but also executable initial environment states and reward functions that can reliably verify task completion. Hand-curated RLVR datasets for CUAs are labor-intensive—authoring a single instance often requires hours of expert work—and cover only a narrow band of real applications. In contrast, LLM- or VLM-as-a-judge approaches offer data scale but lack reward determinism and reliability, destabilizing policy optimization. Previous code-native pipelines are scalable in web-only mocks but fail to generalize to the open-ended, cross-platform workflows typical of real-world computer use.

CUA-Gym proposes a solution by automating joint generation of tasks, environments, and rewards through an agentic, adversarial co-generation pipeline, and by synthesizing a broad suite of interactive, controllable mock environments designed for reproducibility and coverage.

Data Synthesis Pipeline and Reward Integrity

CUA-Gym's data pipeline is centered around two adversarial coding agents—a Generator and a Discriminator—coordinated by an Orchestrator operating under a strict information barrier. For each task, the Generator synthesizes both initial and golden environment states by executing setup scripts, while the Discriminator—isolated from the Generator’s code or artifacts—writes a reward function derived solely from the natural-language task specification and endpoint environment states.

Figure 1: Overview of the CUA-Gym data synthesis pipeline, with separate Generator and Discriminator agents co-generating initial/golden states and reward functions, mediated by an Orchestrator and multi-stage filtering.

The Orchestrator orchestrates iterative rounds between the two agents until five formal agreement conditions are simultaneously satisfied, verifying executability, appropriate reward sensitivity (golden state yields reward 1.0, initial state yields 0.0), and the absence of forbidden static analysis patterns that could signal reward function vulnerabilities (e.g., hardcoded success/failure, proxy-checks, or reliance on implementation traces).

After convergence, each candidate tuple passes through a two-stage filter: (1) a multi-LLM ensemble voting protocol, rejecting instructions or rewards with ambiguity, inconsistency, or known hack risks, and (2) a teacher-agent rollout phase that ensures true end-to-end solvability and reward correlation under realistic agent policies, further rejecting instances that are ambiguous or infeasible.

This fully-automated pipeline transforms RLVR data production for CUAs into a process that efficiently explores both the task and environment axes.

Environment Scaling: CUA-Gym-Hub and Mock Web Applications

CUA-Gym treats environment diversity as a primary scaling axis, paralleling trajectory volume in its downstream contribution to RL-based agent performance. Existing benchmarks provide limited application coverage, especially on desktop and cross-application workflows. To overcome this, CUA-Gym introduces CUA-Gym-Hub—a suite of self-contained, programmatically resettable mock web applications designed for high-fidelity task synthesis, extensibility, and empirical grounding in occupational software-usage distributions (O*NET and the Anthropic Economic Index).

Figure 2: Multi-agent pipeline for synthesizing mock environments with design, implementation, and end-to-end interactive verification.

CUA-Gym-Hub mocks are synthesized and verified through a multi-agent pipeline: a Plan Agent derives UI and data schema specifications; a Dev Agent constructs the mock SPA; a Web Agent exhaustively verifies interactive functionality against the original specification. All mocks implement a unified, session-scoped HTTP state API enabling deterministic injection and inspection, critical for clean RL experimentation. This design enables state injection and session isolation, making possible the creation of many distinct tasks per environment without code changes, and supporting highly parallel RL rollouts.

Figure 3: State-injected environments in CUA-Gym-Hub—showing the same mail mock instantiated with varied task-specific world states, enabling distinct task distributions and parallel RL rollouts.

Empirical Results and Scaling Laws

CUA-Gym’s empirical evaluation demonstrates significant advances in both post-training performance and scalability properties:

• On the OSWorld-Verified benchmark, RLVR-trained CUA-Gym-A3B achieves 62.1% and CUA-Gym-A17B achieves 72.6% success, outperforming prior open-source CUAs at comparable scale and closing the gap with proprietary models.
• Compared to their supervised fine-tuned (SFT) initializations, RLVR on CUA-Gym data improves multi-domain and multi-application workflows most substantially, with gains persisting at large scales without evidence of diminishing returns.
• On the held-out WebArena browser benchmark (with site clones disjoint from CUA-Gym-Hub), these same policies exhibit positive transfer, indicating that the synthesized web-mock skill generalizes beyond the training distribution.

  Figure 4: Data scaling curves for RLVR training on CUA-Gym—showing OSWorld-Verified score and training reward, with larger training sets yielding higher asymptotic performance and earlier departure from the SFT baseline.

Key findings regarding scaling:

• Increasing the number of unique, verified RLVR tuples (from 1.4K to 12K+) directly lifts both asymptotic and early-stage RL learning curves, with no signs of saturation in the explored range.
• Environment diversity, controlled via the CUA-Gym-Hub pool, is orthogonal to data volume; holding task count constant, increasing the number of environments yields meaningful performance gains that cannot be substituted by simply generating more tasks per environment.

Additionally, RL training on CUA-Gym data induces emergent behaviors not seen in SFT-only models. Notably, the RL-trained policy spontaneously compresses multi-step deterministic UI action chains into single batched tool-call turns, reducing average trajectory length by 33–45% at fixed task success, paralleling the efficiency improvements observed with RL in other agent domains.

Dataset and Environment Coverage

CUA-Gym encompasses 32,112 verified RLVR tuples across 110 environments, spanning 16 desktop applications and 94 mock web environments. Task distributions are balanced across application categories, difficulty levels (45% hard, 38% cross-application), and scenario types. Compared to previous datasets, CUA-Gym is the largest fully open-source RLVR dataset using programmatic reward verification with both desktop and web scope.

Figure 5: Environment coverage of CUA-Gym across occupational categories, with explicit mapping to O*NET SOC major groups and application types.

Limitations

• Reward functions in CUA-Gym evaluate terminal environment states, not action trajectories, so behaviors that recreate the goal state through non-intended or destructive means are not penalized if they meet all terminal checks.
• While the information barrier and forbidden-pattern scans significantly reduce reward hacking, the protocol does not offer formal guarantees—semantically loose or proxy-based checks may escape with low frequency.
• CUA-Gym-Hub environments, though high-fidelity, remain approximations: authentication flows, external integrations, latency, and rare edge cases are beyond current scope.
• RL experiments are computed under a single seed due to cost, so reported performance scaling trends should be interpreted as preliminary.

Implications and Future Directions

CUA-Gym validates that the RLVR data recipe underpinning recent progress in math and code agent training transfers to the more complex, GUI- and OS-based CUA domain, given sufficient engineering to automate both reward creation and environment synthesis. By separating environment scaling from task generation, CUA-Gym establishes a new research substrate for exploring generalization, transfer, and scaling laws in GUI agents.

The open-sourcing of all synthesis code, environments, RLVR data, and agent checkpoints positions CUA-Gym as an infrastructure catalyst to accelerate further study in:

• Transfer learning between synthetic and real-world environments,
• Investigation of reward function expressivity and hacking robustness,
• Environment generator innovations beyond SPA web mocks,
• Data curation strategies targeting rare but crucial real-world workflows,
• Efficient RL finetuning and inference architectures for large-scale CUAs.

Conclusion

CUA-Gym provides a scalable, modular, and rigorously verified pipeline for RLVR agent training in the computer-use domain, demonstrating strong and consistent scaling trends in capability with both data and environment diversity. The work offers a comprehensive open-source toolkit and evidence that environment and reward diversity are critical bottlenecks for progress in general-purpose GUI agents, while defining a foundation for future practical and theoretical advances in agent RL.