PhoneWorld: Scalable Mobile-Agent Environments

• PhoneWorld is a scalable pipeline that converts real mobile GUI trajectories and screenshots into executable mock-app environments for agent training.
• It recovers app structure and connectivity using vision-language models to generate stateful, verifiable mock apps with automated task synthesis.
• PhoneWorld enhances benchmark performance by integrating additional supervision and expanding app coverage within fixed training budgets.

PhoneWorld is a system for scaling the construction of phone-use agent environments from real mobile GUI behavior. It is defined as a reusable pipeline that converts real GUI trajectories and screenshots into controllable phone-use environments, executable tasks, automatic verifiers, and training rollouts, with the explicit aim of addressing the scarcity of controllable, reproducible, and resettable mobile environments for agent training and evaluation (Tang et al., 28 May 2026). In the PhoneBuddy training recipe, PhoneWorld serves as the mock-app environment complementary to a real-app environment on physical devices; there it is described as reconstructing runnable mock apps from real GUI usage structure rather than producing static UI mockups (Tang et al., 22 Jun 2026).

1. Historical and conceptual setting

Earlier mobile-computing literature had already framed mobility as both a trend and an established reality, with mobile devices functioning as the visible endpoint of a broader technological stack spanning hardware, software, computer networks, websites adapted for mobile use, payment services, location systems, and carrier networks (Alcantara et al., 2011). In parallel, work on mobile web adaptation treated the phone as a primary web endpoint rather than a secondary display, motivating user-centric and content-selective rendering for small screens (Kuppusamy et al., 2012). These lines of work characterize the phone as a stateful convergence device for communication, commerce, navigation, and real-time service access rather than merely a voice terminal.

Within that longer trajectory, PhoneWorld emerges at the point where phones are treated as an execution surface for general-purpose agents. PhoneBuddy states this motivation directly: phones are becoming an important execution surface for general-purpose agents, but the relevant real-device environment is slow, stateful, side-effectful, and hard to reset or verify (Tang et al., 22 Jun 2026). PhoneWorld addresses this problem by turning real usage structure into runnable mock Android apps with mutable state and automatic verification, thereby making large-scale agent interaction feasible without abandoning the GUI structure of real mobile behavior (Tang et al., 28 May 2026).

2. Pipeline for constructing runnable mock apps

PhoneWorld begins from two principal inputs for each target app: representative screenshots and real usage episodes or GUI trajectories. These trajectories include a natural-language user instruction, a sequence of screenshots, and a sequence of actions recorded on the real device, and they are manually collected by human operators performing exploratory use rather than only fixed benchmark scripts (Tang et al., 28 May 2026).

The first stage is app-structure recovery. Representative screenshots are used to induce a page taxonomy, and the paper states that Claude Code is prompted to browse screenshots and identify recurring page types such as home page, detail page, and profile page. For each app this produces a taxonomy of about 25–30 categories and a classification prompt. A lightweight vision-LLM then classifies the screenshot corpus into that taxonomy. Once screenshots are mapped to page types, PhoneWorld computes a page-frequency distribution across episodes and assigns priority labels: P0 for must-build pages, P1 for recommended pages, and P2 for long-tail pages only if needed for downstream tasks (Tang et al., 28 May 2026).

PhoneWorld also recovers connectivity structure from trajectories. For every episode, each consecutive pair of visited pages induces a directed edge, and aggregation across episodes yields a weighted directed page-transition graph. The graph specifies which routes dominate real usage and therefore which inter-page connections the mock app must preserve. This transition recovery is paired with per-page PRD generation: for each page type, a vision-LLM examines 2–3 representative screenshots and produces a structured product requirements document covering page layout, interactive elements, transition relations, and visual attributes (Tang et al., 28 May 2026).

App construction then proceeds with reusable modules and explicit state separation. The current system includes 18 reusable modules, and generated apps are implemented in Kotlin with Jetpack Compose. Each app is backed by read-only app content and mutable SQLite state. Read-only content stores entities and visible records; mutable state stores agent-relevant changes such as favoriting an item, modifying a cart, posting a comment, or sending a message. Because the apps run offline, PhoneWorld also builds a local BM25 search engine over the read-only content. Code generation is iterative rather than one-shot: a coding agent reads the PRDs, generates code, compiles the APK, runs a self-review checklist, and fixes issues over several iterations. After automatic smoke tests for launch, tab switching, search, detail-page navigation, and representative write operations, human reviewers compare mock and real apps side by side, and the review cycle usually converges in 1–2 rounds (Tang et al., 28 May 2026).

PhoneBuddy presents the same construction logic at a higher level. It states that PhoneWorld starts from real GUI trajectories and screenshots, recovers which screens matter, how screens connect, which actions need to be supported, and which state changes need to be saved, and then builds runnable mock Android apps with read-only content and writable state (Tang et al., 22 Jun 2026).

3. Environment model, task synthesis, and verification

PhoneWorld’s runtime abstraction is the separation between read-only content and mutable state. Read-only content provides the realistic static substrate for browsing and search; mutable SQLite-backed state persists actions taken during an episode and can be reset between runs (Tang et al., 28 May 2026). This makes the environment both agent-interactive and verifier-friendly.

Tasks are derived directly from environment artifacts rather than authored independently. The generator cross-references three sources: read-only app content, database schema, and per-page PRDs. The paper distinguishes two main task classes. Information-seeking tasks require retrieval of values from read-only content, and verification checks whether the final answer contains the expected key values. State-changing tasks require a DB mutation such as favoriting, commenting, messaging, or cart modification, and verification queries the SQLite database after execution (Tang et al., 28 May 2026).

PhoneWorld can also synthesize cross-app tasks by exploiting shared entities across apps. The paper’s worked example instructs an agent to find the project review meeting time in QQ and then send a message to Zhang San telling him to attend on time; verification is performed by querying the messages table. Another concrete task asks the agent to search for “Project Group” and tap Favorite in QQ. The corresponding task representation includes the app, difficulty, max steps, goal, and a verification block over a database table. For the favorite example, the verifier checks user_collections with rules requiring target_type = "group" and target_id = 4 (Tang et al., 28 May 2026).

Verification is intentionally deterministic. PhoneWorld uses answer-based verifiers for information-seeking tasks and SQLite-based verifiers for state-changing tasks. The paper gives the following SQL-style verifier for the messaging example:

SELECT COUNT(*) FROM messages WHERE user_id = 1 AND content LIKE '%3%'

This use of explicit database predicates eliminates the need for model judging during task completion checks and makes the same environments suitable for both evaluation and data generation (Tang et al., 28 May 2026). In PhoneBuddy, this property is summarized as built-in rule-based verifiers over reconstructed app state (Tang et al., 22 Jun 2026).

In its current instantiation, PhoneWorld covers 34 apps across 16 domains and includes 120 audited benchmark tasks, of which 102 are single-app and 18 are cross-app (Tang et al., 28 May 2026).

4. Training substrate and empirical scaling behavior

A central feature of PhoneWorld is that the same environments used for evaluation can also generate supervision. The system first synthesizes a large task pool, runs agents on those tasks, retains only verifier-confirmed successful rollouts, and converts those episodes into multimodal supervised fine-tuning data. The paper reports 7,936 generated tasks, 3,354 successful episodes, and 36,193 interaction steps in the resulting PhoneWorld rollout corpus (Tang et al., 28 May 2026).

The main training experiments use a Qwen3.5-9B backbone with supervised fine-tuning for 2 epochs in LlamaFactory. Input is standardized as system prompt, current screenshot, user instruction, and textual summary of previous actions, with the model predicting the next thought-and-action output. The matched-budget setup keeps total supervision fixed at 72,386 steps: 36,193 AndroidWorld base steps plus 36,193 auxiliary steps, with partial replacement experiments substituting some auxiliary AndroidWorld steps with PhoneWorld rollouts (Tang et al., 28 May 2026).

The headline result is a 10K-step partial replacement. Under a fixed budget, replacing 10,000 auxiliary AndroidWorld steps with PhoneWorld supervision improves all four evaluation benchmarks at once (Tang et al., 28 May 2026).

Benchmark	Baseline	10K PhoneWorld replacement
HYMobileBench	15.5	33.2
AndroidControl	53.7	59.7
AndroidWorld	56.9	71.6
PhoneWorld	12.5	65.0

These correspond to gains of +17.7 on HYMobileBench, +6.0 on AndroidControl, +14.7 on AndroidWorld, and +52.5 on PhoneWorld (Tang et al., 28 May 2026). The full-replacement control is more nuanced: replacing the entire auxiliary AndroidWorld corpus with PhoneWorld raises HYMobileBench from 15.5 to 33.2, AndroidControl from 53.7 to 59.3, and PhoneWorld from 12.5 to 73.3, but lowers AndroidWorld from 56.9 to 46.6. The paper interprets this as evidence that PhoneWorld and AndroidWorld supervision are complementary rather than interchangeable (Tang et al., 28 May 2026).

The scaling studies sharpen that claim. Adding more PhoneWorld supervision monotonically improves PhoneWorld task success from 14.2 at 0 added steps to 64.2 at 10K, 70.0 at 20K, and 73.3 at 36,193 added steps. Under a fixed 10K PhoneWorld budget, expanding app coverage from 5 apps to 34 apps raises PhoneWorld from 46.7 to 65.0, HYMobileBench from 14.9 to 33.2, and AndroidWorld from 61.2 to 71.6, while AndroidControl remains roughly stable overall (Tang et al., 28 May 2026). The paper’s explicit conclusion is that, under a fixed PhoneWorld budget, expanding app coverage yields even larger gains (Tang et al., 28 May 2026).

PhoneBuddy independently validates PhoneWorld as a training environment inside a mixed real-plus-mock recipe. There, all models begin from a shared SFT dataset containing 950,758 action steps collected from both real-app and mock-app environments. Mixed reinforcement learning then combines a 50/50 real/mock rollout mixture for 50 online RL steps. Across a 150-task human evaluation on real phones, task success rises from 36.67% after SFT to 40.67% after real-app RL and 45.33% after mixed RL. On AndroidWorld, the same progression is 60.3%, 77.2%, and 83.2%. The gains are strongest on single-app and WeChat mini-app tasks, while cross-app performance does not improve meaningfully (Tang et al., 22 Jun 2026).

5. Relation to adjacent “world” formulations in mobile-agent research

PhoneWorld occupies a specific position within a broader family of mobile-agent and mobile-world abstractions. It is neither the only system concerned with “world modeling” in the mobile domain nor interchangeable with nearby benchmarks and models.

MobileWorld is a reproducible benchmark for autonomous mobile agents rather than a pipeline for constructing environments. It contains 201 tasks across 20 applications, emphasizes long-horizon and multi-application workflows, and introduces agent-user interaction tasks and MCP-augmented tasks. The paper states explicitly that it contains no mention of PhoneWorld and does not identify itself as the same system, a predecessor, or a successor (Kong et al., 22 Dec 2025). Its contribution is benchmark difficulty and deterministic evaluation, not environment synthesis.

MobileWorldBench and the MobileWorld dataset approach the problem from semantic world modeling. They model future GUI state transitions in natural language rather than executable mock apps or pixel predictions. The benchmark evaluates next-state generation and next-state QA, while the dataset contains 1.4M samples built from mobile GUI transitions. In downstream AndroidWorld experiments, augmenting M3A with a semantic world model improves success from 46.9 to 50.8 in zero-shot mode and to 54.3 with the finetuned model (Li et al., 16 Dec 2025). This line of work addresses phone-world dynamics in language space; PhoneWorld instead produces runnable environments with mutable state and deterministic verification.

MobiWorld addresses yet another layer of the mobile stack: the wireless network rather than the GUI. It is a generative world model for mobile wireless networks, designed to synthesize network element-level observations such as traffic load and user distribution and system-level indicators such as throughput and energy consumption under spatio-temporal, behavioral, and policy conditions (Chai et al., 13 Jul 2025). A plausible implication is that “world modeling” in the mobile domain now spans at least three strata: GUI execution environments, semantic GUI transition models, and network digital twins. PhoneWorld belongs to the first stratum.

6. Limitations, design tradeoffs, and significance

PhoneWorld is built around selective abstraction rather than full replication. The paper states that the mock apps preserve important screens, navigation paths, visible content, and state-changing operations, but they are not complete replicas of the original apps (Tang et al., 28 May 2026). This abstraction is deliberate: it makes reset, verification, and large-scale rollout generation tractable, but it also means that long-tail app functionality, hidden server-side logic, and the full behavioral complexity of commercial apps are not represented.

Several limitations are explicit. The audited PhoneWorld benchmark is compact, with 120 tasks. Training and evaluation use the same set of apps with disjoint task instances, so the experiments primarily demonstrate within-suite task generalization rather than transfer to entirely unseen mock apps. Human auditing remains necessary for realism checks, and HYMobileBench is internal rather than public (Tang et al., 28 May 2026). PhoneBuddy adds a closely related limitation from the mixed-RL perspective: PhoneWorld’s current task pool is primarily single-app, and long-horizontal cross-app workflows remain an open challenge (Tang et al., 22 Jun 2026).

The most important design tradeoff is therefore between scalability and fidelity. PhoneWorld simplifies interfaces relative to real apps, offers partial coverage rather than full behavioral reproduction, and abstracts state to the subset required for task execution and verification (Tang et al., 22 Jun 2026). Yet that same abstraction gives it the properties that real apps lack at scale: resetability, repeatability, safety, and automatic checking. PhoneBuddy makes the complementarity claim explicit: mock-app training is not a replacement for real-app RL, but a complementary source of scalable, resettable, and automatically checked interaction (Tang et al., 22 Jun 2026).

PhoneWorld’s broader significance lies in shifting the focus from constructing one mobile benchmark at a time to scaling the supply of environments themselves. The paper’s empirical results support that shift: a relatively small matched-budget substitution improves four benchmarks simultaneously, additional PhoneWorld supervision continues to improve in-domain performance, and broader app coverage under a fixed budget yields larger gains than narrower coverage (Tang et al., 28 May 2026). This suggests that, for phone-use agents, environment diversity is not merely an evaluation convenience but a primary training resource.