Environment scalability and stability for large-scale interactive RL

Engineer reproducible, fault-tolerant, and high-throughput RL environments that can reliably support millions of interactive episodes across browsers, virtual machines, and simulators, ensuring stable large-scale training of GUI-centered agents.

Background

The report highlights environment fragility, resource intensity, and crash-proneness as practical bottlenecks in deploying large-scale RL environments for GUI agents. They present a unified sandbox platform and orchestration across VMs and browser sandboxes, but identify the fundamental challenge of achieving scalability and stability as an open problem.

This problem underpins their infrastructure work (session tracking, crash recovery, checkpointing, GPU acceleration) aimed at reproducible, fault-tolerant rollouts at high concurrency and throughput.

References

While recent advances in native agent models have shown promise by unifying perception, reasoning, action, and memory through end-to-end learning, open problems remain in data scalability, multi-turn reinforcement learning (RL), the limitations of GUI-only operation, and environment stability.

— UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning  (2509.02544 - Wang et al., 2 Sep 2025) in Abstract (Page 1)

Our experiments show that joint RL across mobile and desktop is beneficial, improving over the single-domain specialists on average. Extending joint RL to all available environments, covering mobile, desktop, web, and visual tool use together, is a promising direction for further gains. However, doing so is resource-intensive, as it requires sustaining many heterogeneous environments concurrently throughout training, and we leave it for future work.

— MintAct: A Unified Visual Agent for Digital Environments  (2609.22083 - Gao et al., 18 Sep 2026) in Section Limitation, paragraph “Scaling joint RL to all environments”