- The paper introduces iOSWorld, a native iOS benchmark featuring a persistent user persona across 26 interconnected apps to evaluate personalized mobile agents.
- It utilizes a POMDP framework with vision-only and vision+XML modalities, showing notable performance gains (up to 93% pass rate on single-app tasks) with XML access.
- The study identifies challenges in action grounding, long-horizon planning, and memory over distributed data, urging advancements in hierarchical agent architectures.
iOSWorld: A Benchmark for Personally Intelligent Phone Agents
Motivation and Problem Setting
Contemporary mobile agent benchmarks overwhelmingly neglect the requirements for high-fidelity personalized assistance on mobile platforms, primarily focusing on impersonal, stateless environments and lacking the requisite cross-app, persistent user identity necessary for real-world deployment. iOSWorld directly addresses these deficiencies by introducing a native iOS benchmark that implements a persistent user persona—Jordan Avery—across 26 interconnected, purpose-built iOS apps seeded with synthetic yet semantically coherent personal data. The resulting ecosystem allows for the evaluation of agents' proficiency in reasoning over distributed, longitudinal user data, mimicking authentic modality-switching and privacy-centric interactions quintessential to phone-based agents.
Benchmark Environment and Task Suite
iOSWorld is operationalized as a partially observable Markov decision process (POMDP) environment, leveraging a deterministic iOS simulator built around Xcode/XCUITest and exposing agents to both vision-only (screenshot input) and privileged vision+XML (accessibility tree) modalities. The input space thus covers both standard deployment conditions and an upper bound granted by accessibility access, facilitating rigorous analysis of action grounding and interface-specific bottlenecks. The action space is comprehensive, accommodating pixel-based taps and text entry (vision), and expanding to ID-based targeting and app launching (XML).
The suite consists of 133 tasks partitioned into:
- Single-app tasks (27): Evaluate base competence in individual apps.
- Multi-app tasks (60): Require agents to transfer information and coordinate actions across multiple apps (2–8 per task).
- Memory and personalization tasks (46): Mandate pattern recognition and synthesis from distributed personal data, testing agents’ ability to infer habitual routes, recurring relationships, financial behaviors, and temporal patterns without explicit instruction.
Task creation and rubric design were systematically validated by human annotators, ensuring that all objectives are feasible, unambiguous, and grounded in the seeded digital life of the user persona.
Experimental Protocol and Model Evaluation
The benchmark evaluates frontier and open-source VLM/LLM-based computer-use agents, specifically Claude Opus 4.6, Claude Sonnet 4.6, GPT-5.4, GPT-5.4 Mini, Gemini 3 Flash, and the open-source Qwen3.5 35B-A3B. Each model operates under both vision-only and vision+XML modalities, with a budget of 50 interaction steps per episode. Trajectory-level task outcome is judged by an LLM-evaluator validated against human annotators (Îş=0.77, 89% accuracy).
Key empirical findings:
- The best-performing configuration (Claude Opus 4.6, vision+XML) achieves:
- 93% pass rate on single-app tasks.
- 54% on memory/personalization.
- 37% on multi-app tasks.
- 52% overall pass rate.
- Privileged XML access yields substantial gains—Opus 4.6 rises from 26% to 52% overall (+25.6pp); Sonnet 4.6 from 29% to 47% (+18.0pp); GPT-5.4 from 20% to 40% (+19.5pp).
- Smaller models such as Qwen3.5 (35B-A3B) and GPT-5.4 Mini do not benefit from XML input and in fact exhibit degraded performance, indicating context-size limitations (e.g., Qwen3.5 drops from 13% to 11% overall with XML, frequently failing due to action loops and context overflows).
Analysis of Bottlenecks and Failure Modes
While XML access addresses interaction failures introduced by coordinate estimation and inefficient navigation in vision-only mode—particularly in multi-app switch and label-targeted subtasks—it introduces new complexities for small-capacity models that cannot effectively consume or leverage added context (~3,100 tokens/step). The majority of unsolved tasks, even for the most capable agents, result from "budget exhaustion" (step-limit reached before success), representing incomplete planning and insufficient loop recovery.
Failure taxonomy is distributed as:
- 51% budget exhausted: Most common in multi-app/memory tasks.
- 26% early give-up.
- 23% premature stop: The agent ends before full criterion satisfaction.
The granular analysis indicates action grounding (coordinate estimation, robust back/forward navigation), long-horizon planning, and memory (especially over distributed in-app data) as central open challenges. Quantitative action trace analysis reveals high miss rates for coordinate-based tapping (10-12% for frontier models in vision-only mode), which XML targeting largely eliminates.
Implications for AI and Future Developments
iOSWorld exposes substantive gaps in the current generation of computer-use agents regarding personalized, longitudinal reasoning, robust multi-app workflows, and memory-intensive planning—characteristics critical to deployment as personal phone assistants. Contrary to progress on desktop and web agents, the mobile paradigm imposes unique interface frictions, context and modality challenges, and privacy constraints. The benchmark underscores both the technical necessity and complexity of supporting user-centric reasoning and memory, as well as the challenge of scaling context and tool-use without overwhelming model capacity.
Moving forward, research directions should prioritize:
- Hierarchical and planning-augmented agent architectures to mitigate budget exhaustion on long-horizon tasks.
- Dynamic context management and task decomposition to ensure tractable XML-centric input even for middle-tier models.
- Advanced grounding and multi-modal interface optimization tailored to mobile UI-specific challenges (dense touch targets, absence of universal hardware navigation).
- Richer agent personalization schemes and multi-persona task construction to enhance generalizability across user profiles.
Conclusion
iOSWorld sets a new standard for evaluating personal intelligence in mobile agents, combining true cross-app continuity, rich personal data, and real iOS interaction. The benchmark reveals that while much progress has been made, even the most advanced models demonstrate substantial limitations when operating in realistic, personalized mobile settings. Its open-source nature and extensible framework offer a robust foundation for future research in building truly assistive, context- and history-aware mobile agents; advancement on this benchmark will constitute measurable progress towards practical, trustworthy, user-facing AI on consumer devices.
Citation: "iOSWorld: A Benchmark for Personally Intelligent Phone Agents" (2606.09764)
