PerPilot: Personalized Mobile Agents

• PerPilot is a vision-language framework that defines personalized instructions as ambiguous, user-specific commands and resolves them using a two-stage personalization layer.
• It employs the PerInstruct benchmark—comprising 75 real and LLM-augmented mobile app instructions—to evaluate task difficulty and personalization accuracy.
• Empirical results show significant performance improvements over baselines, with memory accumulation reducing the need for exploration and human intervention.

PerPilot is a plug-and-play framework for making vision-language-model-based mobile agents handle personalized instructions: instructions containing ambiguous, user-specific elements whose meaning is not explicitly stated but depends on stable personal context such as “my home,” “my friend,” “my favorite song,” or “usual wake-up time.” The framework was introduced together with PerInstruct, a human-annotated benchmark for personalized mobile-agent behavior, and is designed to augment existing mobile agents rather than replace their UI-control backbones. Its central mechanism is a two-stage personalization layer that first detects personalized elements and then resolves them through memory-based retrieval and reasoning-based exploration, with human clarification as a fallback (Wang et al., 25 Aug 2025).

1. Concept and problem formulation

PerPilot is situated in the setting of VLM-based mobile agents that observe phone screenshots, interpret natural-language instructions, reason over UI state, and execute GUI actions step by step across apps. The paper argues that such agents are effective on explicit commands such as “Set an alarm for 7:30 am” but struggle with everyday instructions such as “Play my favorite song,” “Set an alarm for my usual wake-up time,” “Navigate to my home,” and “Reply hello to my friend,” because these instructions contain user-specific references that are not directly executable (Wang et al., 25 Aug 2025).

The paper defines personalized instructions as instructions containing ambiguous, user-specific elements, called personalized elements, that reflect individual preferences and routines and are stable and consistent over an extended period. Examples listed in the paper include “mom,” “my home,” “my school,” “favorite song,” “usual wake-up time,” “friend,” “professional group,” “frequent takeout,” “often used expression,” “dormitory WiFi,” “father’s birthday,” and “own computer” (Wang et al., 25 Aug 2025).

A key distinction in the formulation is that not all unresolved values are treated as personalization in the same way. The perception prompt explicitly states that words needing clarification or having different meanings for different people or devices are personalized, whereas app names and specific names are not. Thus “my school” and “my friend” are personalized elements, while “Mr. Beast” and “Bob” are not (Wang et al., 25 Aug 2025).

The framework is presented as a front-end layer for existing systems such as AppAgent, MobileAgent-v2, and UI-TARS. Its purpose is not to redesign low-level mobile control, but to transform underspecified personalized instructions into clarified executable instructions that those agents can carry out (Wang et al., 25 Aug 2025).

2. PerInstruct benchmark and task structure

PerPilot was introduced together with PerInstruct, described as the first human-annotated dataset specifically for evaluating the personalization ability of VLM-based mobile agents. PerInstruct contains 75 instructions spanning 27 widely used mobile apps and involving 28 personalized elements across diverse mobile scenarios. The apps cover five categories: Social and Communication, System Tools, Life Services, Entertainment, and Shopping / E-commerce (Wang et al., 25 Aug 2025).

The dataset was built by combining authentic user-generated requests with instructions augmented through LLM-based generation, and each entry contains an instruction ID, natural-language instruction, difficulty label, minimal step count, involved app(s), completed form of the instruction, personalized element(s), and personalized info. This structure makes the benchmark not merely a task list, but a decomposition from ambiguous phrasing to executable slots (Wang et al., 25 Aug 2025).

PerInstruct organizes tasks into three difficulty levels. “Simple” tasks contain one personalized element and straightforward single-app interaction; “Normal” tasks contain multiple personalized elements and several steps, still within one app; and “Hard” tasks contain multiple personalized elements, complex multi-step workflows, and often cross-app interaction. The examples given include “Open Alipay and pay my rent” as simple, “Open NetEase Cloud Music to play favorite song and forward it to friend” as normal, and “Open Baidu map, navigate to my home and then open QQ to send a message to friend that I am almost home” as hard (Wang et al., 25 Aug 2025).

The paper reports a human evaluation by seven human experts over five dimensions: personalization, difficulty, accuracy, coverage, and diversity. The reported average scores are 4.44 ± 0.32 for personalization, 4.40 ± 0.32 for difficulty, 4.69 ± 0.19 for accuracy, 4.00 ± 0.00 for coverage, and 4.33 ± 0.51 for diversity, with an overall average of 4.37 ± 0.26. It also reports a Difficulty Label Consistency of 0.86 and Distribution Entropy values of 0.98 for difficulty and 0.89 for diversity (Wang et al., 25 Aug 2025).

PerInstruct is used as the evaluation benchmark rather than the main supervised training corpus. The paper explicitly states that PerQwen, the fine-tuned open model used in the framework, is trained on separately generated data rather than on a standard train/dev/test split of PerInstruct (Wang et al., 25 Aug 2025).

3. Architecture and formal workflow

PerPilot is organized as a four-stage pipeline: Personalization Perception, Personalization Completion, Instruction Clarification, and Execution by Base Agent. In the first stage, the system determines whether an instruction is personalized and extracts all personalized elements. In the second, it resolves those elements by first trying memory-based retrieval and then, if necessary, reasoning-based exploration. In the third, it replaces the ambiguous personalized elements with explicit concrete values. In the fourth, it passes the clarified instruction to a base mobile agent such as UI-TARS, MobileAgent-v2, or AppAgent (Wang et al., 25 Aug 2025).

The perception module is formalized as

$$Perception(\mathcal{I},\mathcal{P}) \rightarrow (y_{personal}\in\{Y,N\},\{k^i\}_{i=1}^{n}),$$

where $\mathcal{I}$ is the user instruction, $\mathcal{P}$ is the set of principles for judging personalization, $y_{personal}$ indicates whether the instruction is personalized, and $\{k^i\}_{i=1}^{n}$ are the extracted personalized elements. The paper emphasizes that all personalized elements are extracted in a single pass, which is important for instructions containing multiple user-specific slots (Wang et al., 25 Aug 2025).

Memory completion is formalized as

$$RetrievalCompletion(\mathcal{I}, \mathcal{T}, \{k^i\}_{i=1}^{n}) \rightarrow \begin{cases} (Y,\mathcal{I}_{completed}), & \text{if } \forall k^i, \ k^i \in \mathcal{T} \ (Part, \mathcal{I}_{part}, \{k^i\}_{i=1}^{m}), & \text{if } \exists k^i \notin \mathcal{T} \text{ and } \exists k^j \in \mathcal{T} \ (N, \{k^i\}_{i=1}^{n}), & \text{if } \forall k^i, \ k^i \notin \mathcal{T} \end{cases}$$

where $\mathcal{T}$ is the memory store of personalized element–information pairs. Exploration completion is formalized as

$$ExploreCompletion(\mathcal{I}, \{k^i\}_{i=1}^{m}, \mathcal{L}_{app}) \rightarrow \mathcal{I}_{app},$$

where $\mathcal{L}_{app}$ denotes the available apps on the device and $\mathcal{I}_{app}$ is an exploration instruction telling the base agent where and how to retrieve the missing information (Wang et al., 25 Aug 2025).

The paper does not define an end-to-end optimization objective, retrieval similarity score, or learned routing loss. Instead, PerPilot is described as an inference-time LLM orchestration framework, with a procedural decision logic: detect personalized elements, query memory, explore only unresolved elements, and ask the user only when exploration fails (Wang et al., 25 Aug 2025).

4. Memory-based retrieval and reasoning-based exploration

The first completion mechanism is memory-based retrieval. PerPilot stores personalized element–information pairs such as “mom” $\mathcal{I}$0 “Susan,” “my school” $\mathcal{I}$1 “test school,” “my friend” $\mathcal{I}$2 “jack,” and “my home” $\mathcal{I}$3 a specific address. The paper characterizes this memory as a conceptually simple database rather than a vector-memory system; it does not describe embedding retrieval, nearest-neighbor search, or a learned retrieval score (Wang et al., 25 Aug 2025).

This memory design rests on the assumption that many personalized facts are stable enough to be reused: a mother’s name, a home address, a school, or a dormitory WiFi often remain constant over time. Once exploration or user clarification resolves an element, PerPilot stores it back into $\mathcal{I}$4, allowing later tasks to be completed with less exploration and less user interruption. This suggests a cumulative personalization process in which the framework becomes more effective through continued use (Wang et al., 25 Aug 2025).

If memory is insufficient, PerPilot invokes reasoning-based exploration. The key idea is that many personalized facts are already stored somewhere on the phone: contacts in messaging apps, addresses in map or shopping apps, favorite songs in music apps, frequent takeout in food-delivery apps, WiFi names in settings, and so on. Rather than asking the user immediately, the framework uses an LLM to infer which app is most likely to contain the missing information and to generate an exploration instruction for the base agent (Wang et al., 25 Aug 2025).

The paper’s examples are concrete. For “Navigate to my home,” the generated exploration instruction can be “From the app Taobao, retrieve my home address.” For “Open TikTok, search for a video about my school, and share it with my friend,” the unresolved element my friend can yield the exploration instruction “From the app wechat, obtain my friend name information.” The execution guidance also instructs the base agent not to search literally for the ambiguous placeholder, to recover full text if UI elements are truncated with ellipses, and to return concise core information (Wang et al., 25 Aug 2025).

Human intervention is reserved for the final fallback case. The framework therefore composes three modes of personalization resolution: stored personal memory, autonomous phone exploration, and user clarification. The paper treats the complementarity between memory and exploration as the central design principle of the system (Wang et al., 25 Aug 2025).

5. Models, training, and empirical results

PerPilot is evaluated with three base mobile-agent systems—UI-TARS, MobileAgent-v2, and AppAgent—and with two controller variants: o4-mini-2025-04-16 and PerQwen, a LoRA-fine-tuned version of Qwen3-8B. The fine-tuning data consist of 7,000 GPT-4o-generated samples for personalized instruction perception and personalized element extraction, and 3,000 GPT-4o-generated samples for reasoning-based exploration. The reported training settings are learning rate $\mathcal{I}$5, cosine scheduler, effective batch size 16, LoRA rank 8, LoRA alpha 32, dropout 0.1 on projection layers, epochs 5 or 8 depending on task, and warm-up ratio 0.05 (Wang et al., 25 Aug 2025).

The experiments use a Huawei nova13 running HarmonyOS and connected via ADB. LLM invocation settings are temperature 0.0, max tokens 4096, and random seed 1234. The evaluation metrics are success rate, element perception accuracy, exploration accuracy, and human intervention count (Wang et al., 25 Aug 2025).

The main benchmark results show large gains over baseline mobile agents:

Backbone	Baseline	PerPilot + PerQwen	PerPilot + o4-mini	Best with human intervention
UI-TARS	12.0%	50.7%	56.0%	68.0%
MobileAgent-v2	9.3%	20.0%	24.0%	49.3%
AppAgent	10.7%	16.0%	20.0%	46.7%

For UI-TARS, the paper reports a baseline overall success rate of 12.0% (9/75), rising to 50.7% with PerQwen, 56.0% with o4-mini, 62.7% with PerQwen plus human intervention, and 68.0% with o4-mini plus human intervention. The best gain over baseline is reported as +56.0 points. MobileAgent-v2 rises from 9.3% to 24.0% with o4-mini and to 49.3% with o4-mini plus human intervention. AppAgent rises from 10.7% to 20.0% with o4-mini and to 46.7% with o4-mini plus human intervention (Wang et al., 25 Aug 2025).

The difficulty-level results are especially pronounced. The UI-TARS baseline scores 28.1% on simple tasks and 0% on both normal and hard tasks. With PerPilot + o4-mini + human intervention, the corresponding scores become 81.3%, 81.0%, and 36.4%. This indicates that the framework is particularly important for instructions containing multiple personalized elements and multi-step structure, although hard cross-app tasks remain substantially less solved than simple and normal tasks (Wang et al., 25 Aug 2025).

The paper also isolates the personalization controller itself. For element perception accuracy, Qwen3-8B scores 26.7%, PerQwen 74.7%, and o4-mini 86.7%. For exploration accuracy without human intervention, the scores are 22.7%, 60.0%, and 73.3%, respectively. With human intervention, the exploration accuracies become 26.7%, 74.7%, and 86.7%. The corresponding success rates are 20.0% for Qwen3-8B, 62.7% for PerQwen, and 68.0% for o4-mini (Wang et al., 25 Aug 2025).

An additional empirical finding concerns memory accumulation. The paper reports that as more instructions are executed, the system relies less on exploration and less on human intervention, while relying more on memory retrieval. This suggests that PerPilot’s performance profile improves with continued usage because personalized element–value pairs are reused rather than rediscovered (Wang et al., 25 Aug 2025).

6. Limitations, interpretation, and significance

PerPilot is presented as a practical solution to a weakness of current mobile agents, but the paper also makes its limitations apparent. The framework depends on the competence of the base mobile agent: resolving a personalized slot does not ensure successful execution if the underlying UI agent cannot navigate reliably. Performance on hard multi-app tasks remains limited even in the best reported setting, with the strongest UI-TARS hard-task result reaching only 36.4% (Wang et al., 25 Aug 2025).

The memory design is intentionally simple, which is advantageous for interpretability but leaves unresolved questions about stale or shifting personal facts. The paper emphasizes that many personalized facts are stable, yet it also notes that some are not permanent: favorite songs change, favorite takeout changes, and routines evolve. The described framework does not include sophisticated temporal memory management or confidence-aware updating (Wang et al., 25 Aug 2025).

Reasoning-based exploration is also brittle in the ways typical of UI agents: the LLM may choose the wrong app, the information may be visually truncated or hidden, or the base agent may fail to reach the relevant screen. Cross-app workflows add further fragility. Privacy is another stated concern, and one motivation for PerQwen is to reduce reliance on closed models for tasks involving private user data (Wang et al., 25 Aug 2025).

Within those limits, the system’s significance is clear. PerPilot reframes personalization as a front-end reasoning problem rather than as a change to the low-level mobile control substrate. Its main contribution is to show that personalized instruction execution can be improved by combining three ingredients: a detector for personalized elements, a memory of previously resolved user-specific facts, and an exploration mechanism that searches likely apps for missing information. A plausible implication is that personalization-aware reasoning may become a standard architectural layer for future mobile agents, in much the same way that perception and action planning already are (Wang et al., 25 Aug 2025).