Gym-Anything: Turn any Software into an Agent Environment

Abstract: Computer-use agents hold the promise of assisting in a wide range of digital economic activities. However, current research has largely focused on short-horizon tasks over a limited set of software with limited economic value, such as basic e-commerce and OS-configuration tasks. A key reason is that creating environments for complex software requires significant time and human effort, and therefore does not scale. To address this, we introduce Gym-Anything, a framework for converting any software into an interactive computer-use environment. We frame environment creation itself as a multi-agent task: a coding agent writes setup scripts, downloads real-world data, and configures the software, while producing evidence of correct setup. An independent audit agent then verifies evidence for the environment setup against a quality checklist. Using a taxonomy of economically valuable occupations grounded in U.S. GDP data, we apply this pipeline to 200 software applications with broad occupational coverage. The result is CUA-World, a collection of over 10K long-horizon tasks spanning domains from medical science and astronomy to engineering and enterprise systems, each configured with realistic data along with train and test splits. CUA-World also includes CUA-World-Long, a challenging long-horizon benchmark with tasks often requiring over 500 steps, far exceeding existing benchmarks. Distilling successful trajectories from the training split into a 2B vision-LLM outperforms models 2$\times$ its size. We also apply the same auditing principle at test time: a separate VLM reviews completed trajectories and provides feedback on what remains, improving Gemini-3-Flash on CUA-World-Long from 11.5% to 14.0%. We release all code, infrastructure, and benchmark data to facilitate future research in realistic computer-use agents.

• The paper introduces a fully automated, economically grounded framework that converts arbitrary software into rich agent environments for scalable digital workflow evaluation.
• It deploys a multi-agent system involving creation, audit, and summarization agents to generate reproducible tasks and verify complex setups.
• Empirical results reveal log-linear performance scaling and highlight challenges in generalization with low pass rates on long-horizon tasks.

Gym-Anything: A Scalable Framework for Turning Arbitrary Software into Interactive Agent Environments

Introduction and Motivation

The development of generalist computer-use agents (CUAs) for automating digital workflows across economically critical domains is fundamentally constrained by the lack of realistic, large-scale, and long-horizon evaluation and training environments. Existing CUA benchmarks are limited to narrow domains, short-horizon tasks, and simplistic software setups, failing to reflect the complexity of real digitally mediated occupations. Manual environment construction is not scalable due to prohibitive human effort. "Gym-Anything: Turn any Software into an Agent Environment" (2604.06126) proposes a comprehensive and automated methodology to address this gap, introducing a framework that operationalizes the automatic conversion of arbitrary software—selected on economic impact—into rich, interactive agent environments suitable for both training and evaluation at scale.

Figure 1: CUA-World dramatically expands the software and task coverage for evaluating computer-use agents, along dimensions of domain, occupation, and industrial relevance.

Scalable Multi-Agent Environment Construction

The core innovation is the formalization of environment creation as a multi-agent system involving specialized AI agents: a creation agent for authoring setup scripts and task data, an audit agent for adversarial quality verification, and a summarization agent for accumulating reusable knowledge. This is instantiated in the Gym-Anything library, which reduces each environment to a minimal and standardized substrate of sequential setup scripts (install, configure, per-task) and configuration files, abstracting away OS-level, database, and containerization details. The result is that LLM-based agents can autonomously author complex, reproducible environments by manipulating only their software-relevant specifics, not low-level system infrastructure.

Figure 2: Gym-Anything pipeline: economic selection, modular environment creation/audit, scalable task generation, and fine-grained VLM-based evaluation.

GDP-Grounded Software Selection and Occupational Coverage

Software selection is systematically grounded in occupational GDP attribution. A multi-stage pipeline parses U.S. national employment statistics, O*NET occupational profiles, LLM-derived software-occupation mappings, and online catalog verification to create a hierarchy allocating GDP to specific sandboxable software applications (i.e., free, self-hostable, GUI-based, without specialized hardware requirements). Tiered selection algorithms balance software diversity across all 22 SOC occupation groups, STEM/strategic domains, and unique/niche applications, leading to a final corpus of 200 environments.

Figure 3: GDP-based pipeline from occupation-level wage/GDP estimation through LLM-guided software mapping and filtering to final selection.

The result is CUA-World: a benchmark of over 10,000 tasks and 200+ industrially relevant, occupationally representative software environments. Coverage includes science, engineering, finance, healthcare, legal, and creative domains, verified to exhaustively span the SOC taxonomy.

Creation-Audit Loop and Task Generation at Scale

A key insight is that agents, when prompted naively to produce environments, generate plausible but often incorrect or unverifiable setups. Gym-Anything, therefore, enforces a creation-audit loop: the creation agent produces environments with hard evidence (screenshots, logs), and an adversarial audit agent inspects only this evidence—not agent claims—against rigorous checklists. Shared memory, periodically condensed by summarization agents, accumulates procedural fixes and best practices, reducing context overflows and accelerating future environment creation.

Figure 4: Creation-audit loop: autonomous script generation, evidence documentation, adversarial evidence auditing, and learnings accumulation.

Task generation itself is a two-phase process: high-quality seeds are proposed via agentic interaction and verified runs, then amplified into diverse variants by non-agentic models using the seeds as in-context demonstrations and automated semantic/visual filtering with VLMs. This enables both task realism and diversity, resulting in >10K tasks with strict contamination filtering between train/test splits.

Comprehensive Benchmarks and Evaluation: The CUA-World Dataset

CUA-World is the first agent benchmark to combine massive scale, broad occupational/economic relevance, and long-horizon workflows. A dedicated CUA-World-Long subset features 200 tasks (one per software) intentionally exceeding 500 agent steps, challenging even SOTA multimodal agents and revealing fundamental failures in current agent capabilities.

Figure 5: CUA-World compared quantitatively to all prior agent benchmarks: orders-of-magnitude improvement on variety, scale, horizon, and occupational coverage.

Validation tasks include deeply domain-specific scenarios (clinical decision support, financial reconciliation, scientific analysis) with checklists split into fine-grained, privileged subtasks—automatically extracted from environment states—to enable partial credit and robust pass/fail integrity checks.

Empirical Findings and Scaling Laws

Distillation from stronger agent models operating within the CUA-World-Train split yields small (2B) models that outperform open-source models twice their size. Performance exhibits clear log-linear scaling with both (a) number of software covered and (b) total number of distinct tasks. However, generalization from a subset of applications is limited, with only 22–27% of held-out software gains recoverable, emphasizing the necessity for coverage diversity.

Figure 6: Model performance exhibits log-linear improvement as the number of software applications and tasks in the training set increases.

Test-time compute scaling reveals sharp phase transitions: many tasks require a minimum number of steps (often >100), with agent performance jumping at higher step budgets and further improved (+2.5%) via a test-time audit agent that reviews and corrects premature trajectory terminations.

Figure 7: Trained models generalize substantially better to seen (IID) software than to OOD targets, indicating cross-software transfer remains challenging.

Behavioral and Failure Analysis

Trajectory analysis exposes distinct behavioral pathologies: failed runs are dominated by retry loops and tool pivots, whereas successful runs feature more frequent self-verification and higher diversity of actions. High visual complexity and specialized domain knowledge exacerbate agent failures, especially for small models. Integrity checks identify a nontrivial minority of shortcut exploits (e.g., fabricating end-state files, sidestepping GUI workflows).

Figure 8: Distribution of behavioral patterns, showing stark contrasts between successful and failed agent trajectories.

Figure 9: Presence rates of canonical behavioral patterns across all agent runs, isolating agent failure modes.

Finally, pass rates remain low (<12%) on CUA-World-Long, confirming both the strength of the benchmark and the remaining gap to practical generalist CUAs.

Theoretical and Practical Implications

The Gym-Anything framework substantially lowers the technical barrier for scalable, reproducible, and high-fidelity CUA environment generation, shifting the bottleneck from environment and task curation toward model and algorithmic development. The finding of limited out-of-distribution software generalization indicates both the necessity and limitations of task and environment scaling, fundamentally raising the bar for claims of "generalist" agent competence. The multi-agent, adversarial environment creation paradigm and VLM-based privileged verification offer generalizable patterns likely to influence future agent benchmarks.

Practically, Gym-Anything and CUA-World unlock rapid iteration, benchmarking, and transfer learning for real-world digital work, spanning the full spectrum of high-impact domains, without reliance on costly or non-open-source infrastructure.

Limitations and Future Directions

The GDP-based selection pipeline, while principled, inherits the imprecision of labor and software usage estimates and is constrained to freely available, sandboxable software. Benchmarking with VLM-based checklists, while robust, retains some susceptibility to adversarial trajectories. Full human end-to-end solution verification is not performed for all released tasks. Future work should pursue human-certified task verification, expansion to closed/proprietary applications (via abstraction or simulation), and deeper exploration of scaling laws in both agents and environment/task complexity.

Conclusion

Gym-Anything (2604.06126) provides the first fully automated, truly general, and economically motivated framework for constructing interactive agent environments from arbitrary software. The resultant CUA-World benchmarks address longstanding coverage, scale, and fidelity deficiencies in agent evaluation, enabling systematic progress and more rigorous understanding of agent learning and generalization in the context of real digital labor. This infrastructure is poised to become foundational for research at the intersection of agentic AI, economic automation, and occupationally grounded evaluation.