- The paper introduces ChainWorld to compose extended desktop workloads from atomic OSWorld tasks, enabling robust evaluation of multi-step planning and error recovery.
- It leverages algorithmic composition to sequence isolated tasks into realistic workflows, with agents’ success rates dropping from over 90% to below 60% in chained scenarios.
- It underscores critical limitations in current agent designs, providing actionable insights for advancing hierarchical planning, persistent memory, and digital workflow automation.
ChainWorld: Composing Long-Horizon Desktop Workloads from Atomic OSWorld Tasks
Motivation and Problem Statement
The proliferation of LLM-based agent frameworks for automating computer use has led to rapid progress in single-turn and short-horizon GUI interaction tasks. However, existing benchmarks such as OSWorld (Xie et al., 2024), WorkArena++ (Boisvert et al., 2024), and OdysseyBench (Wang et al., 12 Aug 2025) primarily evaluate agent competence on atomic or narrowly-scoped activities, failing to capture the compositional planning and error recovery needed for executing realistic, long-horizon desktop workflows. ChainWorld is introduced to address this deficit: it systematically composes extended desktop workloads by chaining atomic OSWorld tasks, thereby facilitating robust evaluation of an agent's ability to orchestrate multi-step, interdependent operations across diverse software environments.
Methodology
ChainWorld’s task synthesis leverages the atomic actions and tasks formally cataloged in OSWorld (Xie et al., 2024). Through algorithmic composition, atomic tasks are sequenced or nested to form long-horizon workloads reflecting realistic desktop interactions, including document processing, data analysis, and project management scenarios. Each composed workload is annotated with explicit dependencies and error-handling requirements, ensuring that successful completion demands not only sequential execution but also contextual memory, cross-app coordination, and recovery from unexpected failures. The synthetic workloads integrate multi-modal inputs and outputs—files, links, textual content, and various GUI controls—mirroring the heterogeneity of real-world digital workflows.
ChainWorld also includes rigorous task evaluation using both automatic and agent-as-judge methods (Zhuge et al., 2024), with reward structures calibrated for compositional success and process integrity rather than atomic completion alone. This enables direct measurement of an agent's planning capacity, adaptation to intermediate failure, and persistence over extended activity chains.
Experimental Results
The benchmark is instantiated across a broad set of contemporary desktop agent architectures, including multimodal LLM-based systems (e.g., Kimi K2.5 (Team et al., 2 Feb 2026), Gemini 3.1 Pro [Google DeepMind 2026], Claude Opus 4.7 [Anthropic 2026]), RL-based computer use agents (Lai et al., 19 Aug 2025), and trajectory synthesis baselines (Xu et al., 2024, Sun et al., 2024). ChainWorld reports strong numerical results that highlight the performance gap between atomic task completion and compositional workload achievement. For example, agents that previously exhibited >90% success on isolated OSWorld tasks show a marked drop to <60% success on chained workloads, with failure modes deriving from context loss, insufficient error recovery, and planning inconsistencies.
Additionally, ChainWorld reveals contradictory claims relative to prior benchmarks: agents deemed "generalist" or "autonomous" by prior atomic-task metrics (Xie et al., 17 Jun 2025, Deng et al., 2023) frequently underperform in compositional scenarios. This finding underscores the limitations of current evaluation protocols and exposes critical gaps in the architectural design of desktop automation agents.
Implications and Future Directions
ChainWorld establishes a new standard for evaluating generalist agents in desktop use cases, emphasizing long-horizon planning, robust error handling, and compositional reasoning. Practically, this reveals key failure modes for LLM/GPT-based agents, such as context fragmentation and insufficient trajectory resilience, thereby informing system design for production-grade digital assistants. Theoretically, ChainWorld motivates research in hierarchical planning, persistent memory architectures, advanced workflow synthesis, and evaluative frameworks that go beyond atomic reward signals.
Future developments may include adaptive workload generation based on real-world user logs, integration with decentralized evaluation (e.g., crowd-sourced process verification), and extension to multi-agent collaboration across shared digital environments. ChainWorld's composable workload paradigm is also applicable to enterprise automation, scientific workflow orchestration, and longitudinal software testing.
Conclusion
ChainWorld advances agent benchmarking for desktop automation by composing long-horizon workloads from atomic OSWorld tasks, exposing the planning, coordination, and recovery challenges inherent to realistic computer use. The benchmark’s strong empirical findings and rigorous evaluation protocol highlight significant limitations in current agent systems and provide actionable directions for future research in AI-powered digital workflow automation.