PerInstruct: Personalized Mobile-Agent Benchmark

• PerInstruct is a benchmark defined to address personalized instruction following by resolving ambiguous user-specific elements within multi-app scenarios.
• It combines authentic user requests with LLM-augmented synthetic data, using a structured annotation protocol that maps elements like 'my friend' to specific entities.
• Empirical evaluations with the PerPilot framework show significant performance improvements, with gains up to +56% over standard mobile agents on ambiguous tasks.

PerInstruct is a human-annotated benchmark for personalized mobile-agent instruction following, introduced together with the PerPilot framework in "PerPilot: Personalizing VLM-based Mobile Agents via Memory and Exploration" (Wang et al., 25 Aug 2025). In this setting, a personalized instruction is a user command containing ambiguous, user-specific elements whose interpretation depends on individual preferences, routines, or context. PerInstruct operationalizes this problem for mobile agents by collecting and annotating multi-app instructions in which the semantic gap is not primarily linguistic ambiguity in the abstract, but the need to ground expressions such as “my friend,” “my home,” or “my favorite song” in user-specific information.

1. Personalized instruction following as a benchmarked problem

PerInstruct is built around a precise definition of personalized instructions: user commands containing ambiguous, user-specific elements (“personalized elements”) reflecting individual preferences, routines, or context (Wang et al., 25 Aug 2025). These elements are described as typically stable over time, while their actual meaning varies per user and often must be resolved from user data or habits. The canonical example is “Play my favorite song,” where “favorite song” cannot be executed correctly without grounding the phrase in historical preference data.

This formulation distinguishes personalized instruction following from standard mobile-agent benchmarks centered on explicit, non-personalized commands. In PerInstruct, the challenge is not only UI navigation or app-state reasoning, but also the completion of under-specified requests before execution. A command such as “Open QQ to view the questions sent by my friend and copy it” is incomplete for an agent unless “my friend” is resolved to a concrete identity.

The paper formalizes the first stage of this process as a personalization perception problem:

$$\text{Perception}(\mathcal{I}, \mathcal{P}) \rightarrow (y_\text{personal} \in \{Y, N\}, \{k^i\}_{i=1}^{n})$$

where the instruction $\mathcal{I}$ is analyzed to determine whether personalization is present and, if so, which personalized elements $\{k^i\}$ must be grounded. This framing places PerInstruct at the intersection of instruction following, personalization, and agentic information retrieval.

2. Dataset construction and annotation protocol

PerInstruct combines authentic user-generated requests with LLM-augmented synthetic instructions, explicitly to ensure coverage and diversity (Wang et al., 25 Aug 2025). The resulting dataset contains 75 instructions spanning 27 widely-used mobile apps across five categories: Social Communication, System Tools, Life Services, Entertainment, and Shopping.

The annotation protocol is structured around executable personalization. For each instruction, annotators identify the personalized elements, formulate a completed instruction in which those elements are explicitly specified, assign a difficulty label, record the minimum step count, and list the relevant apps. Difficulty is determined by the number of personalized elements, task complexity, and cross-app involvement. The paper states that extensive human annotation and expert review were used to ensure accuracy and coverage.

A representative example illustrates the annotation scheme. The instruction “Open QQ to view the questions sent by my friend and copy it” is labeled as requiring the QQ app, and the personalized element “my friend” is converted into the completed form “Open QQ to view the questions sent by {Name} and copy it.” This design makes explicit that PerInstruct is not merely a corpus of ambiguous requests; it is a benchmark with gold completions that expose the hidden variables an agent must recover.

The dataset’s scope across social communications, navigation, shopping, media, and scheduling indicates that personalization is treated as a cross-application systems problem rather than an app-specific phenomenon. This suggests a benchmark intended to stress both semantic grounding and operational integration.

3. Instruction schema and difficulty taxonomy

PerInstruct organizes instructions by difficulty into three categories, each tied to concrete execution properties rather than purely linguistic criteria (Wang et al., 25 Aug 2025).

Difficulty	Definition
Simple	One personalized element, single-step, single app
Normal	Multiple personalized elements, multi-step, single app
Hard	Multiple personalized elements, multi-app, multi-step workflows

This taxonomy is important because it couples personalization density to agentic burden. A “simple” instruction may require only one grounded variable and one UI action, whereas a “hard” instruction may require multiple grounded variables, stateful planning, and coordination across apps. The inclusion of minimum step count as an annotation field reinforces that the benchmark is designed for embodied or semi-embodied digital action rather than text-only completion.

The schema also encodes a distinction between latent ambiguity and executable clarity. Each raw instruction has a corresponding completed instruction in which personalized elements are made explicit. This creates a natural two-stage evaluation perspective: first, detect and resolve the personalized elements; second, pass the clarified instruction to a downstream mobile agent. In effect, PerInstruct separates personalization understanding from execution while preserving the dependency between them.

4. Dataset analysis and validation

PerInstruct includes both quantitative validation and expert-based human evaluation (Wang et al., 25 Aug 2025). The paper reports a difficulty-label consistency (DLC) Spearman correlation of 0.86 between assigned labels and actual step complexity. It also reports distribution entropy (DE) of approximately 0.98 for difficulty and approximately 0.89 for app diversity, which the authors interpret as indicating balanced data.

Human evaluation was conducted on a 5-point Likert scale by 7 experts. The reported scores are 4.44 for Personalization, 4.40 for Difficulty, 4.69 for Accuracy, 4.00 for Coverage, and 4.33 for Diversity, with an average of 4.37.

These numbers support two distinct claims. First, the benchmark’s difficulty labels appear to align closely with operational complexity rather than being arbitrary annotations. Second, expert assessment judged the dataset to be strong on personalization fidelity and annotation accuracy. The comparatively lower coverage score of 4.00 does not indicate a defect by itself, but it does suggest that the benchmark’s authors treated coverage as a measurable axis rather than assuming completeness.

The paper further states that PerInstruct is currently the only mobile agent dataset explicitly focused on personalization, while other benchmarks focus on explicit, non-personalized tasks. Within the benchmark landscape described in the paper, this gives PerInstruct a distinct evaluative role: it isolates a failure mode that can remain hidden when instructions are fully specified.

5. PerPilot: the system designed to use PerInstruct

PerInstruct was introduced together with PerPilot, a plug-and-play LLM-powered framework for personalized instruction understanding and completion in VLM-based mobile agents (Wang et al., 25 Aug 2025). PerPilot is designed to augment existing agents rather than replace them, and its workflow follows a perception-completion-execution pipeline.

The first component is the personalization perception module, which detects whether an instruction contains personalized elements and extracts them. The second is the personalization completion module, which applies two strategies hierarchically. In memory-based retrieval, the agent consults a memory store $\mathcal{T}$ of known element-to-information mappings; if the required elements are already known, the instruction can be completed immediately or partially completed. When memory is insufficient, reasoning-based exploration is triggered: the LLM reasons about which installed app or apps are likely to contain the missing information and issues targeted exploration commands.

The exploration stage is formalized as

$$\text{ExploreCompletion}(\mathcal{I}, \{k^i\}, \mathcal{L}_{app}) \rightarrow \mathcal{I}_{app}$$

where the system converts an ambiguous instruction into an app-grounded completion process. If neither memory nor exploration yields the needed information, the framework prompts the user minimally for clarification.

The paper emphasizes that PerPilot is model-agnostic. It can use closed-source models such as o4-mini or a finetuned open-source model, PerQwen, based on Qwen3-8B. PerQwen is finetuned using 7K LLM-generated data for extraction and 3K for reasoning. The framework is also designed so that, with continued use, memory-based retrieval becomes increasingly effective, reducing both exploration and user interruption.

6. Empirical performance on PerInstruct and implications

PerInstruct functions not only as a dataset but also as an evaluation bed for personalization-aware mobile agents (Wang et al., 25 Aug 2025). The paper reports that, without PerPilot, success rates on PerInstruct are extremely low: UI-TARS achieves 12.0%, MobileAgent-v2 9.3%, and AppAgent 10.7%. These results are presented as evidence that standard agents cannot handle ambiguous, personalized tasks.

When augmented with PerPilot using o4-mini, the same systems improve to 68.0% for UI-TARS, 49.3% for MobileAgent-v2, and 46.7% for AppAgent. The gains are reported as +56.0, +40.0, and +34.7 respectively. With the open PerQwen version, UI-TARS reaches 62.7%. The paper additionally reports that element perception accuracy rises from 26.7% for a Qwen3-8B baseline to 74.7% for PerQwen, approaching o4-mini’s 86.7%.

System	Without PerPilot	With PerPilot (o4-mini)
UI-TARS	12.0%	68.0%
MobileAgent-v2	9.3%	49.3%
AppAgent	10.7%	46.7%

The analysis in the paper attributes these gains to explicit handling of personalized elements through memory and exploration. It also reports that performance improves further with minimal human intervention and is expected to get better as agent memory grows. Figure-based analysis is described as showing a shift over time from exploration and user intervention toward memory-based retrieval. The case studies presented in the paper indicate that baseline agents fail on ambiguous elements such as “my favorite takeout,” whereas PerPilot forms the necessary mappings and executes the clarified task.

Taken together, these results position PerInstruct as a benchmark for a specific systems capability: not just following instructions, but converting user-relative references into executable state. A plausible implication is that personalization-aware reasoning should be treated as a first-class evaluation axis for mobile agents, analogous to perception, planning, and tool use. In the framework established by the paper, PerInstruct supplies the data substrate for that shift, while PerPilot provides one concrete mechanism for exploiting it.