PhoneBuddy: Agentic Phone Training

• PhoneBuddy is a training approach that integrates real phone interactions with a simulated PhoneWorld environment to achieve agentic control.
• It uses shared supervised fine-tuning on nearly 1M action steps followed by distinct reinforcement learning branches to enhance task performance.
• Results show improved success rates on single-app and mini-app tasks, though challenges remain in handling long-horizon cross-app workflows.

Searching arXiv for the specified PhoneBuddy paper and closely related work. arxiv_search.query({"search_query":"all:PhoneBuddy OR ti:PhoneBuddy", "start":0, "max_results":10, "sortBy":"relevance", "sortOrder":"descending"}) PhoneBuddy is a training recipe and open-model line for agentic phone use that combines a real-app environment with a mock-app environment, PhoneWorld, which reconstructs runnable mock apps from real GUI usage structure. It is motivated by the observation that real devices running real apps are the environment that matters at deployment, but that environment is slow, stateful, side-effectful, and hard to reset or verify, while scalable mock environments only approximate real behavior. PhoneBuddy therefore uses a shared supervised fine-tuning stage from trajectories collected in both environments, followed by reinforcement-learning branches that either use real apps alone or a 50/50 mix of real and mock apps. Across a 150-task human evaluation on real phones spanning apps, mini-apps, and cross-app workflows, task success rate improves from 36.67\% after supervised fine-tuning to 40.67\% after real-app RL and 45.33\% after mixed RL; on AndroidWorld, the same progression rises from 60.3\% to 77.2\% to 83.2\% (Tang et al., 22 Jun 2026).

1. Problem formulation and environment trade-offs

PhoneBuddy is situated in a setting where large language–based agents are being asked not only to understand graphical user interfaces but to act reliably through them on real phones, including tasks with real side effects, complex state, and multi-step, cross-app workflows. The central design problem is a tension between fidelity and scalability: the real-app environment provides exact fidelity to production apps, realistic timing, authentic side effects, and exposure to real failure modes such as account-dependent logic, service outages, and permission flows, but it also imposes slow rollouts, expensive resets, fragile automatic verification, and risky irreversible side effects. The mock-app environment, PhoneWorld, offers fast rollouts, easy reset, rule-based verifiers for success, and scalable data collection, but it has approximate or simplified app logic, partial coverage of real side-effects and server behavior, and imperfect transfer (Tang et al., 22 Jun 2026).

The resulting claim is not that mock apps supersede real-device interaction. The reported ablations support a narrower thesis: combining both environments yields the strongest open-model performance on real-phone tasks. A common misunderstanding is therefore to treat scalable mock-app training as a replacement for real-app RL; the reported results instead characterize it as a complementary source of scalable, resettable, and automatically checked interaction.

2. PhoneWorld and mock-app reconstruction

PhoneWorld is a pipeline that ingests real GUI usage traces, specifically screenshots and action logs, and produces runnable Android mock apps with built-in state and rule-based verifiers. Its reconstruction pipeline has three phases: screen graph recovery, transition and state extraction, and mock app synthesis (Tang et al., 22 Jun 2026).

In screen graph recovery, real screenshots are clustered into discrete screens, and widget hierarchies and actionable elements are identified for each screen. In transition and state extraction, edges in the screen graph are inferred from the action trace, and on-screen inputs such as form fields are mapped to internal state variables. In mock app synthesis, Android layout XML is generated for each screen, transitions and a minimal Java or Kotlin stub are implemented to track state variables, and a rule-based success predicate is derived, such as whether the state variable order_confirmed = true.

Because PhoneWorld uses the same GUI structure as real apps, it preserves enough realism for transfer while offering reset, repeatability, and automatic outcome checks. This suggests that PhoneWorld is intended less as a full behavioral simulator of production services than as a structured approximation that retains interface topology and task-relevant state.

3. Shared initialization and reinforcement-learning branches

All PhoneBuddy variants start from Qwen3.5-4B and share the same multimodal phone-control API, a shared supervised fine-tuning initialization, and identical evaluation harness and prompts. The only difference is the final reinforcement-learning branch (Tang et al., 22 Jun 2026).

The supervised fine-tuning stage uses real-app trajectories of approximately 480 K action steps, collected on physical devices executing diverse tasks, and mock-app trajectories of approximately 470 K action steps, covering many single-app and mini-app workflows, for a total of 950,758 steps. The objective is next-action cross-entropy:

$$L_{\mathrm{SFT}}(\theta) = - \mathbb{E}_{(s,a)\sim D}\left[\log \pi_\theta(a \mid s)\right]$$

where $\pi_\theta$ is the model’s action distribution, $s$ is a multimodal observation consisting of screenshot, history, and instruction, and $a$ is the ground-truth action. Optimization uses full-parameter fine-tuning for 1,115 Adam steps, batch size 512 with packed 8,192-token sequences, and a learning rate linearly decayed from $1\mathrm{e}{-5}$ to $1\mathrm{e}{-6}$.

All RL stages optimize the expected return

$$J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)]$$

where $\tau=(s_0,a_0,\ldots,s_T,a_T)$ is a rollout and $R(\tau)\in\{0,1\}$ indicates task success. The gradient is approximated via a policy-gradient method, for example PPO with clipping:

$$\nabla_\theta J(\theta) \approx \mathbb{E}_{\tau}\left[\sum_{t=0}^{T}\nabla_\theta \log \pi_\theta(a_t\mid s_t)\bigl(R(\tau)-b\bigr)\right]$$

with a learned baseline $\pi_\theta$0 to reduce variance. Each RL branch runs 50 online RL steps under a step budget of 30 actions per episode.

Checkpoint	RL environment	Reward source
PhoneBuddy-4B-SFT	none (SFT only)	cross-entropy $\pi_\theta$1
PhoneBuddy-4B-Real	real only	rubric-based model judging
PhoneBuddy-4B-Real+Mock	real + mock	mixed (rubric + rule-verifier)

In the real-app RL branch, the reward is binary: 1 if a rubric-based model judge, specified as Gemini-3.1-Pro rubrics plus Qwen3.5-122B scoring, verifies full task completion from the interaction trace, else 0. In the mixed real+mock RL branch, the environment is 50\% real-app rollouts with rubric-judged reward and 50\% mock-app rollouts with built-in rule verifiers. The intended effect is to retain realism from real-app RL while gaining scale, resetability, and automatic supervision from PhoneWorld.

4. Evaluation protocol and aggregate performance

Evaluation is reported on two settings: a 150-task real-phone human evaluation suite and AndroidWorld, described as a dynamic simulator. The real-phone suite contains 50 Single-App tasks, 50 WeChat Mini-App tasks, and 50 Cross-App tasks (Tang et al., 22 Jun 2026).

Stage	Real phones (150)	AndroidWorld
SFT only	36.67	60.3
Real-RL	40.67	77.2
Real+Mock-RL	45.33	83.2

The aggregate pattern is monotonic in both environments: supervised fine-tuning establishes a baseline, real-app RL produces a further gain, and mixed real+mock RL produces the strongest reported performance. The paper does not report formal statistical-significance tests. The interpretation advanced by the results is therefore comparative and ablation-based rather than inferential in the statistical sense.

AndroidWorld exhibits a larger absolute improvement than the real-phone suite, rising from 60.3\% to 83.2\%. A plausible implication is that the mock-app component transfers cleanly to benchmarked interaction structure while still requiring real-app RL to anchor the policy to authentic device behavior and side effects.

5. Category-wise behavior and the cross-app gap

The per-category real-phone breakdown shows sharply different behavior across task classes. Single-App performance rises from 36.7 to 54.0 to 62.0 across SFT, Real-RL, and Real+Mock-RL. WeChat Mini-App performance moves from 54.0 to 48.0 to 56.0. Cross-App performance moves from 22.0 to 20.0 to 18.0 (Tang et al., 22 Jun 2026).

The strongest gains therefore occur on app and mini-app tasks, and the paper explicitly identifies long-horizontal cross-app workflows as an important open challenge. For single-app tasks, the reported interpretation is that PhoneWorld’s scalable coverage of single-app workflows transfers effectively once realism is maintained by real-app RL. For mini-app tasks, the recovery under Real+Mock after a dip under Real-RL suggests that PhoneWorld helps stabilize multi-step but structurally regular tasks.

Cross-app workflows remain a major weakness, at approximately 20\%, with no benefit from mock-app training in the current setup. The likely reasons given are that PhoneWorld does not yet reconstruct multi-app handoff patterns or persistent cross-app state, and that cross-app workflows demand long-horizon memory and intermediate verification beyond simple end-state checks. This suggests that current transfer is strongest when GUI structure and task completion can be localized within one app or a mini-app substrate, and weakest when success depends on inter-application artifact transfer and persistent state across a long horizon.

6. Limitations, deployment requirements, and scope

The main limitation reported for PhoneBuddy is not low-level phone control but workflow coverage. Real-app RL anchors the agent in authentic device behavior and side effects, while PhoneWorld supplies scale, reset, and automatic supervision; their combination outperforms either alone, but it does not close the gap on long-horizon cross-app coordination (Tang et al., 22 Jun 2026).

Future directions are stated in operational terms rather than as a single algorithmic prescription. Extending mock apps to model information handoff, artifact transfer, and persistent multi-app state is presented as an urgent next step. The paper also states that mock-app gains do not solve long-horizon state-tracking or runtime coordination, and proposes future training with intermediate verification signals and richer memory mechanisms.

Beyond training, deployable phone agents also require a robust runtime harness for privacy and safety, specifically identified as PhoneHarness and PhonePrivacy/PhoneSafety, together with guardrails against unintended side effects and ongoing monitoring. In that sense, PhoneBuddy delineates a boundary of current capabilities: shared SFT plus mixed real-app and mock-app RL significantly improves open-model performance on real-phone tasks, but does not by itself constitute a complete deployment stack.

The term “PhoneBuddy” should also be distinguished from unrelated phone-number security work. A separate line of research studies how phone numbers and cross-application features can be abused for targeted attacks, including social phishing, spear phishing, and vishing (Gupta et al., 2015). That work concerns phone numbers as cross-application keys rather than training open models for agentic phone use.