Gym-Anything: CUA Environment Framework

Updated 2 July 2026

Gym-Anything is a framework that converts arbitrary applications into interactive agent environments for large-scale computer-use agent evaluation.
It employs a multi-agent pipeline—creation, audit, and summarization—to automate environment generation with verifiable, contamination-controlled task setups.
Empirical results reveal that iterative test-time auditing and trajectory feedback significantly enhance performance in long-horizon, economically impactful digital workflows.

Gym-Anything is a framework designed to convert arbitrary software applications into interactive agent environments by automating the environment creation process, enabling large-scale development and evaluation of computer-use agents (CUAs) across a broad landscape of economically valuable digital workflows. The framework’s flagship instantiation, CUA-World, encompasses over 10,000 long-horizon tasks spanning 200 software suites relevant to diverse occupational domains, with rigorous train/test splits and verifiable configurations. Gym-Anything reframes the traditional environment engineering bottleneck as a multi-agent, evidence-audited coding task, driving scalability in benchmark construction and facilitating breakthroughs in both agent evaluation and training, particularly for high-value, long-horizon, and real-world computer use scenarios (Aggarwal et al., 7 Apr 2026).

1. Motivation and Historical Context

Prior to Gym-Anything, CUA evaluation was constrained by:

A narrow scope of short-horizon, consumer-facing tasks (e.g., basic e-commerce, OS configuration) that failed to measure performance on genuine, economically impactful software.
Manual, labor-intensive environment engineering pipelines, with typical setup and configuration efforts scaling superlinearly in the number of applications or domains considered.
Design limitations in task and environment diversity, precluding robust, large-scale training and assessment of agents’ generalization capabilities across software stacks relevant to real-world occupations (D'Oro et al., 7 May 2026).

This context motivated an approach that could automate (and hence scale) the encoding of real software, realistic data, and verifiable task objectives into sandboxes suitable for RL-style agent training and evaluation.

2. Multi-Agent Environment Creation Pipeline

Gym-Anything formalizes environment creation as a collaborative multi-agent process involving three agents:

Creation Agent ( $\mathrm{Agent}_C$ ): Given a software-agnostic spec, generates scripts to automate installation (install.sh), configuration (configure.sh), and task-specific setup (task_setup.sh), plus a declarative configuration file. It launches the application in a container or VM and seeds it with real-world (not synthetic placeholder) data.
Audit Agent ( $\mathrm{Agent}_{\mathrm{audit}}$ ): Operates independently, using an adversarial system prompt to inspect all evidence produced by the creation agent—screenshots, execution logs, file-system artifacts—against a quality checklist addressing correct software launch state, genuine data presence, and absence of configuration errors.
Summarization Agent: Periodically consolidates accumulated state and reports in shared memory $M$ as the environment library expands, ensuring sublinear scaling of creation time as the software corpus grows.

This creation–audit loop continues iteratively until the environment passes all specified verification criteria, reducing human intervention in environment engineering to minimal oversight and review (Aggarwal et al., 7 Apr 2026).

3. GDP-Grounded Selection and Benchmark Construction

To maximize economic relevance, Gym-Anything selects software and domains using a GDP-weighted taxonomy. The process:

Begins with $\sim$ 900 occupations from O*NET, with weighting derived from national annual wage bills.
Infers, via web mining and LLM-guided search, each occupation’s $p_{\mathrm{computer}}$ (share of tasks performed via computers), and distributes occupational economic impact to software categories ( $s_{\mathrm{category}}$ ) and specific titles ( $s_{\mathrm{software}}$ ).
The overall attribution is $\mathrm{GDP}_{\mathrm{software}} = \sum_{\mathrm{occ}} \mathrm{GDP}_{\mathrm{occ}}\times p_{\mathrm{computer}}\times s_{\mathrm{category}}\times s_{\mathrm{software}}\,.$
Filters (e.g. proprietary licenses, non-sandboxable/hardware-coupled apps) ensure all included software is feasible for automated sandboxing.
The final curated set guarantees both coverage of top GDP-implicated workflows and strategic breadth (e.g., Healthcare, Education, STEM), crossing all 22 SOC major groups.

The resulting CUA-World suite encompasses 200 applications and over 10,000 tasks configured with realistic, domain-appropriate datasets (e.g., Hubble FITS files, CT scan archives) and implementation-verifiable objectives (Aggarwal et al., 7 Apr 2026).

4. CUA-World and CUA-World-Long: Task Design and Splitting

The CUA-World benchmark includes 9,720 train and 2,500 test tasks, each representing a concrete workflow with environment, data, and configuration assembled by the multi-agent pipeline.

Contamination Control: To preclude train–test leakage, every instruction pair is compared by an LLM-judge with an 8-point similarity scale; any pair above threshold forms an edge in a contamination graph, with connected components assigned wholly to train or test, yielding provable zero cross-split contamination.
Long-Horizon Benchmarking: The CUA-World-Long subset draws one ~500+ step task per software, targeting maximal workflow complexity. Tasks are engineered via a failure-driven process in which failure modes by prior strong “teacher” agents are detected (e.g., retry loops, premature halt), and new tasks synthesized to stress those weak points while adhering to constraints of objectivity, realism, difficulty, and implementability.

Tasks are fully annotated with “privileged information” $\mathcal{I}$ (e.g., expected record counts not visible to the agent), enabling checklist-based evaluation and robust integrity checks (Aggarwal et al., 7 Apr 2026).

5. Automated Evaluation and Auditing Methodology

Evaluation in Gym-Anything leverages vision-LLM (VLM)–based, checklist-driven scoring with privileged access to environment state and ground-truth outputs:

For each task $\mathcal{T} = (\mathcal{E}_{s_0}, p, V)$ , subtasks $\mathrm{Agent}_{\mathrm{audit}}$ 0 are extracted and assigned weights.
The VLM assigns credit as $\mathrm{Agent}_{\mathrm{audit}}$ 1 with binary judgments per subtask.
An additional integrity checklist $\mathrm{Agent}_{\mathrm{audit}}$ 2 verifies the correct environment, absence of fabrication, and required UI path; any violation zeroes the score.
Audits reveal $\mathrm{Agent}_{\mathrm{audit}}$ 3 agreement between this protocol and human markup, substantially exceeding purely end-state or programmatic accuracy.

For CUA-World-Long and other multi-stage workflows, behavioral metrics such as retry-loop rates, verification checks issued, and premature terminations are quantified, informing targeted improvement schemes (e.g., test-time auditing) (Aggarwal et al., 7 Apr 2026).

6. Empirical Results and Data-Scaling Properties

CUA-World serves as both a benchmark and a large-scale training corpus. In a trajectory distillation experiment:

A 2B-parameter VLM (Qwen3-VL-2B) fine-tuned on $\mathrm{Agent}_{\mathrm{audit}}$ 42,000 successful “teacher” (Kimi-K 2.5) trajectories from CUA-World-Train improved from 1.6% pre-distillation pass rate to 4.4%, exceeding the no-distillation Qwen3-VL-4B (3.9%) despite being half the size.
The average checklist score rose from 12.7 to 22.5, surpassing the 4B model’s 19.3.
Scaling analyses show near-log-linear performance gains: doubling the number of environments or tasks yields $\mathrm{Agent}_{\mathrm{audit}}$ 53.5-point increases in mean scores per agent.

On CUA-World-Long (200 tasks, median $\mathrm{Agent}_{\mathrm{audit}}$ 6 steps), top models (Gemini-3-Flash, GPT-5.4) under a 500-step limit achieved 7.5% and 3.0% pass rates, respectively; relaxing constraints to 2,000 steps and higher costs raises Gemini-3-Flash to 11.5% and GPT-5.4 to 27.5%.

Test-time auditing—where an independent VLM reviews full trajectories and prompts agents to address missing subtasks—further increases pass rates (e.g., Gemini-3-Flash climbs from 11.5% to 14.0% on CUA-World-Long), illustrating that iterative audit-and-correct cycles can mitigate early stopping and oversight (Aggarwal et al., 7 Apr 2026).

7. Impact, Limitations, and Future Directions

Gym-Anything’s contributions are multifold:

Establishes a modular, coding-centric pipeline for large-scale, adversarially audited agent environment creation across all principal digital work domains.
Provides empirically rich, contamination-controlled, long-horizon training and evaluation data, supporting rigorous agent development and generalization assessment.
Demonstrates that test-time auditing and trajectory-level feedback drive measurable gains even for ultra-long tasks.
Suggests that scaling environment and task diversity—rather than solely increasing model size—yields proportional agent performance gains.

Limitations include current restriction to software that is fully sandboxable and free or open-source, finite scope (200 applications, though extensible), and primarily English-language task annotations. The framework’s audit-centric approach, grounded in adversarial agent verification and privileged evidence, may serve as a methodological template for future work in high-confidence, broad-coverage CUA pipeline construction (Aggarwal et al., 7 Apr 2026, D'Oro et al., 7 May 2026).

A plausible implication is that extending Gym-Anything to thousands of programs and integrating richer data modalities, cross-lingual support, and domain-specific skill artifacts (e.g., those in VISUALSKILL (Jiang et al., 16 Jun 2026)) could further accelerate research on generalizable, robust computer-use agents equipped for complex real-world digital economies.