PhoneBuddy: Training Open Models for Agentic Phone Use

Abstract: Phones are becoming an important execution surface for general-purpose agents, but training open models for reliable phone use remains difficult because the environment that matters at deployment, real devices running real apps, is slow, stateful, side-effectful, and hard to reset or verify, while scalable mock environments only approximate real behavior. We present PhoneBuddy, a training recipe and open-model line for agentic phone use that combines a real-app environment with a mock-app environment, PhoneWorld, which reconstructs runnable mock apps from real GUI usage structure. PhoneBuddy first builds a shared supervised fine-tuning stage from trajectories collected in both environments, then compares real-app RL against mixed RL across both environments. Across a 150-task human evaluation on real phones spanning apps, mini-apps, and cross-app workflows, task success rate improves from 36.67\% after supervised fine-tuning to 40.67\% after real-app RL and 45.33\% after mixed RL. On AndroidWorld, the same progression rises from 60.3\% to 77.2\% to 83.2\%. These results show that mock-app training is not a replacement for real-app RL, but a complementary source of scalable, resettable, and automatically checked interaction. The gains are strongest on app and mini-app tasks, while long-horizontal cross-app workflows remain an important open challenge.

• The paper demonstrates that mixing real-app and mock-app reinforcement learning significantly improves agentic phone task completion.
• It employs a staged pipeline with supervised fine-tuning followed by dual RL regimes to harness both real-device feedback and scalable simulated signals.
• Empirical results show substantial gains in single and mini-app tasks while exposing challenges in long-horizon cross-app workflows.

Agentic Phone Use: Training Open Models with PhoneBuddy

Introduction

The "PhoneBuddy: Training Open Models for Agentic Phone Use" paper (2606.23049) addresses the central challenge of developing open models capable of robust, agentic interactions on real-world smartphones. The work introduces a training paradigm tailored for the realities of phone-based software environments, which are inherently stateful, side-effectful, and difficult to simulate or reset at scale. PhoneBuddy combines reinforcement learning (RL) on both real apps and carefully reconstructed mock apps, leveraging practical strengths and compensation for weaknesses in each training environment.

Problem Formulation

Traditional approaches to GUI-based agent training struggle to scale reliably from benchmark simulation to deployment-grade robustness due to a mismatch between (1) the limited realism and coverage of synthetic environments, and (2) the high operational cost and risk associated with running learning agents on real devices. Phone tasks often involve complex, cross-app workflows, device state dependencies, privacy constraints, and non-deterministic server interactions. The practical question is how to construct a training regime that demonstrably improves task completion on real devices, not just in idealized or restricted testbeds.

Methodology

PhoneBuddy employs a staged training pipeline, beginning with supervised fine-tuning (SFT) on a dual-environment dataset. Data is aggregated from both a real-app environment—where authentic apps run on physical devices—and a scalable mock-app environment, Phone World, which reconstructs interactive Android apps based on real GUI usage traces and supports automatic verification. This SFT phase establishes a common initialization for all subsequent training experiments.

The pipeline then branches into two RL regimes:

• Real-App RL: Fine-tuning the policy using rollouts and feedback exclusively from real devices and apps, optimizing performance under deployment conditions but constrained by rollout cost and verifiability.
• Mixed RL (Real+Mock): Jointly optimizing in both real and mock environments. This augments the signal from expensive real-world executions with broader, more easily verifiable task signals from mock apps.

All evaluation protocols, backbone architectures (Qwen3.5-4B), APIs, and action spaces are kept fixed to isolate the impact of RL environment configuration.

Empirical Results

The paper presents a thorough empirical comparison across four evaluation suites: Single-App Tasks, Cross-App Tasks, WeChat Mini-App Tasks (all with real-device human evaluation), and the AndroidWorld benchmark.

• Single-App and Mini-App Tasks: PhoneBuddy achieves strong, monotonic gains in task completion with mixed RL (62% for Single-App, 56% for Mini-App), outperforming prior open and closed-source models.
• AndroidWorld: Performance improves from 60.3% (SFT baseline) to 83.2% (Real+Mock), exceeding comparable models and demonstrating strong generalization.
• Cross-App Tasks: Marginal improvement (18–22%), highlighting persistent open challenges in state-tracking and information flow across app boundaries.

Critically, the combination of real-app and mock-app RL consistently delivers stronger results than SFT or real-app RL alone, especially for domains where automatic verification and task structure are amenable to mock-environment modeling. The improvements are robust, with cross-environment validation outside the main training pools.

Technical and Practical Implications

This work strongly supports the thesis that mock-app training environments, when reconstructed faithfully from real-world GUI structure, act as a valuable complement—not a replacement—for real-app RL. The mock environment provides advantages in resetability, coverage, and verification, helping optimize policies for stable, multi-step tasks. However, real-app RL remains indispensable for attuning models to the full complexity and irreducibility of real application logic and device state.

The most significant empirical limitation is in long-horizon, cross-app workflow tasks. Achieving robust cross-app capabilities will require explicit modeling and support for persistent state, information transfer, and intermediate verification—directions not yet addressed by current mock-app environment design.

On the practical side, the authors deliberately separate their focus on training from issues in privacy, safety, and deployment-layer robustness. The PhoneBuddy architecture, however, is framed as a modular component within a full-stack agentic phone framework, including runtime harnesses and safety boundaries.

Future Directions

Addressing the observed limitations will likely involve:

• Expanding mock environment coverage to simulate realistic, multi-app workflows with explicit mechanisms for state and artifact handoff.
• Enhancing long-horizon credit assignment, intermediate state tracking, and reasoning capabilities at both model and harness levels.
• Integrating domain-specific safety and privacy constraints directly into both training and runtime orchestration.
• Exploring more fine-grained credit assignment for generalization on tasks with weak or delayed rewards.

Conclusion

PhoneBuddy establishes a rigorous and practical foundation for training open models for agentic phone use, demonstrating that mixed-environment RL yields superior real-device task completion compared to prior regimes. The complementarity of real and mock training environments is empirically validated, with mock-apps providing essential coverage and automatic supervision. Cross-app workflows remain an open technical frontier, requiring methodological advances at both the environment and RL objective level. Future progress in agentic phone use will depend on tighter integration of scalable environment design, memory-oriented agent architectures, and safety-aware deployment strategies.