CocoaBench: Evaluating Unified Digital Agents in the Wild

Abstract: LLM agents now perform strongly in software engineering, deep research, GUI automation, and various other applications, while recent agent scaffolds and models are increasingly integrating these capabilities into unified systems. Yet, most evaluations still test these capabilities in isolation, which leaves a gap for more diverse use cases that require agents to combine different capabilities. We introduce CocoaBench, a benchmark for unified digital agents built from human-designed, long-horizon tasks that require flexible composition of vision, search, and coding. Tasks are specified only by an instruction and an automatic evaluation function over the final output, enabling reliable and scalable evaluation across diverse agent infrastructures. We also present CocoaAgent, a lightweight shared scaffold for controlled comparison across model backbones. Experiments show that current agents remain far from reliable on CocoaBench, with the best evaluated system achieving only 45.1% success rate. Our analysis further points to substantial room for improvement in reasoning and planning, tool use and execution, and visual grounding.

• The paper introduces CocoaBench, a benchmark that evaluates unified digital agents across diverse, multimodal tasks using freeform instructions and automatic scripts.
• The study shows that coding-oriented action policies are vital, with top proprietary models achieving up to 45.1% task success amid compositional challenges.
• The analysis highlights significant reasoning, planning, and visual grounding failures, guiding future improvements in unified agent architectures.

CocoaBench: Evaluation of Unified Digital Agents in Open-World Digital Environments

Motivation and Benchmark Design

The development of general-purpose digital agents capable of handling real-world digital tasks increasingly necessitates unified, compositional abilities—particularly in vision, search, and coding. Existing evaluation suites predominantly assess such competencies in domain isolation (e.g., code-only, browser-only, or GUI-centric benchmarks), failing to capture the true operational context of emerging agentic architectures that integrate multimodal and multi-tool reasoning.

CocoaBench is introduced to systematically assess these unified agents. Every task in CocoaBench is minimally specified by a freeform instruction and an automatic evaluation function; tasks are intentionally decoupled from fixed runtimes or predefined tool APIs. This enables assessment under diverse infrastructures while enforcing that agents autonomously select, integrate, and utilize tools in open-world-like scenarios rather than within narrow app domains.

Figure 1: CocoaBench evaluates agents on complex compositional tasks (e.g., digital shopping) demanding adaptive combination of vision, search, and coding actions.

CocoaBench consists of 153 human-authored tasks, curated to require adaptive composition of the three core capabilities. The design prioritizes: (1) coverage across challenging, realistic domains; (2) resource stability for reproducibility; (3) robust, automatic evaluation via task-specific scripts rather than human or LLM judges.

Task Diversity and Core Capabilities

Tasks span nine high-level domains including Business, Science, Technology, Travel, Education, and Logic/Puzzles. Resource heterogeneity is significant: tasks reference publicly-hosted web pages, documents (PDF, CSV, BibTeX), videos, images, as well as synthetic and logged GUI artifacts.

Figure 2: (a) Task domain coverage, (b) task resource type distribution (webpages, docs, images, etc.), and (c) human-annotated core capability requirements (Vision, Search, Coding); the majority of tasks require multi-capability composition.

Human annotation specifies each task’s minimum capability requirements. Notably, 98% of tasks demand more than one core capability. Coding is explicitly labeled as important in over 56% of tasks; analysis shows stronger agents actually over-index on code/tool usage for multi-step reasoning.

Experimental Protocol

CocoaBench supports two main evaluation paradigms:

• Product-level evaluation: End-to-end agentic systems are benchmarked (e.g., ChatGPT Agent Mode, OpenClaw instantiated with top models, Codex, Claude Code, Deep Research).
• Controlled model comparison: A shared, minimalist scaffold ("Cocoa-Agent"), constructed atop AIO-Sandbox, runs selected LLM backbones under uniform tool APIs and agentic logic. This laddered evaluation allows for both model (backbone) and scaffold (system design) disentanglement.

The agent scaffold provides granular tools for browser navigation (DOM and GUI levels), vision input, code and shell execution, and file manipulation—39 tools in total, grouped by core capability class.

Main Results: Agent Performance and Efficiency

Figure 3: Overall accuracy for agentic systems and model backbones under shared scaffolds on the full CocoaBench task suite.

Performance analysis demonstrates that the best system (GPT-5.4 under Codex or OpenClaw scaffolds) achieves 45.1% task success. Top-tier proprietary models (Claude Sonnet 4.6, Gemini 3.1 Pro) achieve 15–34% depending on scaffold and runtime integration. In contrast, large open-source multimodal models (Kimi-k2.5, Qwen3.5-397B) are substantively less competitive, with sub-12% accuracy. This quantifies a substantial reliability/robustness gap between proprietary and open-source model backbones on unified compositional digital tasks.

Scaffolding strongly impacts agent efficacy and cost/latency tradeoffs. Coding-focused scaffolds (Codex, Claude Code) are shown to generalize well as universal problem-solving shells, despite originally being designed for domain-restricted settings. The light-weight Cocoa-Agent scaffold lags in accuracy but enables standardized comparison and reproducible scaling.

Figure 4: Pareto analysis of agent accuracy versus cost (left) and completion time (right), showing that high performance does not always correspond to greater resource use.

Cost and latency are not direct correlates of task success; Codex achieves superior accuracy at median cost and time per task, while model/scaffold combinations with higher resource footprints do not always yield proportionately better results.

Tool Use Analysis

Figure 5: Per-model tool call distribution grouped by Vision, Search, and Coding categories under the Cocoa-Agent scaffold.

A detailed call-site analysis shows that high-performing models—e.g., GPT-5.4, Gemini 3.1 Pro—allocate over 60% of actions to coding tools (code and shell execution), relegating browser/DOM interaction mainly to information retrieval. Weaker models, conversely, overuse vision/search primitives and under-leverage programmatic processing, resulting in lower compositional task completion.

Structured Failure Taxonomy

Figure 6: Left—aggregate error distribution across all six models; Right—side-by-side breakdown, GPT-5.4 vs. Kimi-k2.5.

A three-layer failure taxonomy is developed and used to annotate 712 failed trajectories: (1) Reasoning/Planning breakdowns, (2) Tool Execution errors, and (3) Visual Grounding failures. Reasoning/planning failures dominate (54+% of cases), with incorrect subgoal formation, incomplete strategies, and misapplied logic being the most prevalent subcategories. Tool execution failures (e.g., infinite loops, anti-bot barrier misinterpretation, or context/trace truncation issues) account for 11–20% depending on model. Visual grounding failures (e.g., detail misperception, parametric knowledge mapping, DOM vs. rendered output confusion) constitute the rest.

Comparative error analysis between GPT-5.4 and Kimi-k2.5 reveals pronounced gaps: Kimi-k2.5 demonstrates more frequent procedural errors, context sensitivity issues (format, memory, infinite loops), and lower reliability in fine-grained vision tasks.

Implications and Future Directions

CocoaBench’s compositional, open-ended design constitutes a reliable, scalable methodology for holistic evaluation of unified digital agents. The results demonstrate a marked disconnect between isolated-task agent performance and reliability on real-world open digital tasks requiring multi-capability synthesis.

Numerical findings clarify that:

• No evaluated system surpasses 46% reliability across benchmarked tasks.
• Coding-oriented action policies are critical for general-purpose agentic performance.
• Significant error mass is accrued in strategic long-horizon planning and robust visual grounding, particularly for large open-source models.

These outcomes provide actionable direction for future agentic research: reinforcement learning for improved reasoning/planning, compositional architecture design, more robust tool-use primitives, and integrated perceptual modules for visual context adaptation. The benchmark, task set, and code are open-sourced to stimulate development of agents that robustly generalize without scaffold or domain overfit.

Conclusion

CocoaBench fills a critical methodological gap for benchmarking unified digital agents in compositional, open-world tasks. The clear gap between current proprietary and open-source backbones, the essentiality of code-oriented action spaces, and the persistence of reasoning and perception failures collectively mark compositional digital agency as an unsolved challenge requiring new algorithmic innovation and system integration (2604.11201).