PhoneAgentBench: Mobile Agent Benchmark

• PhoneAgentBench is a multimodal mobile benchmark suite designed to assess planning, tool invocation, memory, and screen context understanding in MPAs.
• It comprises six datasets with 1,235 real-world QA pairs, covering tasks from app recognition and reference resolution to mobile function calling.
• Empirical findings show significant performance gains through multitask fine-tuning, emphasizing the impact of data mixture in optimizing mobile agent capabilities.

PhoneAgentBench is a specialized benchmark suite for multimodal mobile phone agents (MPAs), introduced in the context of the DaMo framework for multitask supervised fine-tuning of multimodal LLMs (MLLMs). It is designed to evaluate whether an MLLM can function as a practical assistant on a smartphone by jointly testing planning, tool use, memory, and screen understanding, rather than only isolated GUI perception. In the DaMo paper, PhoneAgentBench is described as the first specialized benchmark to evaluate MLLMs on multimodal mobile phone tasks, and it comprises 1,235 QA pairs spanning real-world industrial mobile application scenarios (Shi et al., 22 Oct 2025).

1. Definition and benchmark rationale

PhoneAgentBench was created to address two benchmark gaps in mobile-agent evaluation. The first is the lack of real mobile-agent benchmarks: existing agent benchmarks such as PlanBench, REALM-Bench, ToolBench, BFCL, API-Bank, ReflectionBench, and LTM focus on planning, tool use, or memory in isolation, usually in text-only environments. The second is the prevalence of GUI-only mobile benchmarks: datasets such as MobileViews, SeeClick, ScreenSpot-Pro, VisualAgentBench, and MMBench-GUI L2 emphasize GUI grounding and screen element manipulation while largely omitting complex task planning involving multiple tools or apps, device-native function calling, multimodal conversational memory, and context understanding across images and text (Shi et al., 22 Oct 2025).

Within this design space, PhoneAgentBench is presented as a comprehensive, multimodal, phone-centric benchmark. It systematically evaluates MPAs along four core capabilities: complex task planning, device-native tool usage or function calling, multimodal memory and context, and screen or app context understanding together with app identification. In the broader experimental setup of DaMo, PhoneAgentBench serves as the primary domain-specific benchmark for mobile-agent capabilities, while BFCL-v3, MME-Perception, MME-Reasoning, and OCRBench function as general-purpose multimodal and tool-use benchmarks (Shi et al., 22 Oct 2025).

A notable aspect of the benchmark’s design is that it does not define phone-agent competence as GUI manipulation alone. Instead, it operationalizes mobile assistance as a combination of perception, planning, reference resolution, information extraction, and function invocation over a smartphone-oriented tool and app ecosystem. This suggests a deliberate shift from narrowly visual evaluation toward a capability profile closer to end-user mobile assistance.

2. Task suite and implicit capability taxonomy

PhoneAgentBench is composed of six datasets that are evaluated jointly to obtain an overall score. These six datasets define the benchmark itself.

Dataset	Evaluation ability	Data size
MT-Plan	Mulitmodal Task Planning	100
ACU	Agent Context Understand	100
APP-Rec	APP Recognition	100
MM-RR	Multimodal Reference Resolution	130
MM-NER	Multimodal Named Entity Recognition	376
Mobile-FC	Mobile Function Calling	429

The total size is 1,235 items. All six tasks are averaged to produce the overall PhoneAgentBench score, often reported as PAB Avg. in the DaMo experiments (Shi et al., 22 Oct 2025).

The six tasks target distinct but complementary aspects of mobile-agent behavior. MT-Plan takes an image and a natural-language query as input and requires a multi-step plan structured as a directed acyclic graph of subtasks and dependencies, with each subtask corresponding to a tool or app call. The scenarios are constructed so that the queries are short and colloquial, reflect daily user needs, and require at least two tools. Mobile-FC focuses on device-native and app-level operations in multi-round dialogue, requiring the model to select function names and parameter values from a predefined set of 50 mobile functions. ACU evaluates agent context understanding by requiring the rewriting of a final user question so that anaphora and implicit references are resolved using text history and, when applicable, image content. APP-Rec requires recognition of the app shown in a screenshot. MM-RR asks whether a question actually refers to image content, returning a binary label. MM-NER requires extraction of named entities from visually presented text in images, including time, location, person, phone number, tracking number, flight number, and train number (Shi et al., 22 Oct 2025).

The task suite also implies an internal taxonomy of phone-agent abilities. The benchmark descriptions explicitly distinguish planning versus execution, perception and recognition, context and memory, and tool grounding. MT-Plan and Mobile-FC jointly capture planning and execution; APP-Rec and MM-NER emphasize perception and recognition; ACU and MM-RR target contextual and referential processing; MT-Plan and Mobile-FC together assess grounding into mobile tools and functions. This implicit taxonomy is central to the benchmark’s role as a diagnostic instrument rather than a single-score leaderboard.

PhoneAgentBench is further characterized by its scenario basis. Its data are drawn from real industrial contexts, including screenshots from 100 different apps such as WeChat, QQ, Little Red Book, Weibo, Alipay, Pinduoduo, Taobao, and TikTok, along with phone OS interfaces and web images. As a result, it covers social messaging, social media and content platforms, payments and e-commerce, system utilities such as alarm and navigation, and text-heavy content such as documents and chat logs (Shi et al., 22 Oct 2025).

3. Data construction, sourcing, and annotation

The benchmark’s construction combines multiple image sources and manual authoring. Images come from real photos and mobile screenshots, public web data including Baidu image repositories, and screenshots captured manually on a phone with 100 installed apps. Text questions and dialogues are manually written by annotators, described as internal engineers and professional annotators. The benchmark therefore combines curated visual inputs with manually authored task formulations (Shi et al., 22 Oct 2025).

Task-specific annotation protocols are described in uneven but substantial detail. For MT-Plan, annotators use real photos or mobile screenshots and tools derived from OS or app APIs. They construct a concise, colloquial user query aligned with daily needs and a DAG-structured plan whose nodes are tools or subtasks. Each task must require at least two tool invocations. Quality control is explicit: three annotators perform cross-validation of the queries and plans to ensure accuracy. For MM-NER, the 376 image-only samples come from Baidu’s open data; raw images are manually filtered by professional annotators to retain clear, high-quality content, and the entity labels for the seven categories are applied manually. For Mobile-FC, manually constructed multi-round dialogues simulate realistic app usage scenarios, with 50 function call interfaces defined for cases such as alarm setting, weather querying, and navigation. Gold labels include function names and parameter names, and sometimes values (Shi et al., 22 Oct 2025).

For ACU, conversations are built manually from images and realistic chat scenarios, with emphasis on referential resolution. For APP-Rec, annotators manually install 100 apps on a phone and capture screenshots of different functional interfaces for each app. For MM-RR, images with textual content are sourced from the web, and annotators design positive questions that refer to the image text while creating negative examples by pairing the same question with unrelated images (Shi et al., 22 Oct 2025).

The benchmark documentation does not specify inter-annotator agreement statistics or a formal adjudication procedure beyond MT-Plan’s three-way cross-validation. It also notes that although some training datasets contain “thoughts” and “actions,” the evaluation benchmark primarily supervises final answers: plans, entities, function calls, rewrites, and labels. This places PhoneAgentBench closer to answer-based benchmarking than to trace-supervised reasoning evaluation.

4. Evaluation protocol and task-specific metrics

Each PhoneAgentBench subtask has its own evaluation protocol, and the overall benchmark score is the arithmetic mean of the six task scores. In the experimental tables, all metrics are normalized to percentages from 0 to 100 (Shi et al., 22 Oct 2025).

For MT-Plan, the benchmark uses the T-Eval planning evaluator. The predicted and gold plans are decomposed into ordered action sequences, and the evaluator finds the longest ordered action sequence that matches between them, where matching depends on tool and parameter similarity. The resulting planning score follows the T-Eval evaluator’s definition. MT-Plan is also characterized by explicit dataset statistics: its complexity is reported as $0.661$ and its diversity as $0.82$, compared with T-Eval planning at $0.122$ and $0.73$, respectively (Shi et al., 22 Oct 2025).

For MM-RR, scoring is standard binary accuracy over labels in $\{0,1\}$. For Mobile-FC, a prediction receives 1 only if both function name or names and parameter name or names match the gold labels exactly; otherwise it receives 0, and the final score is the average over examples. ACU is evaluated using BLEU similarity between the rewritten final question and the reference rewrite. MM-NER uses entity-level $F_1$-score. APP-Rec uses exact-match accuracy on the app name (Shi et al., 22 Oct 2025).

The evaluation setup assumes that the model receives images where appropriate, the text question or final user query, and dialogue history for ACU and Mobile-FC. Task-specific instructions are encoded in the prompt, such as requesting a pronoun-free rewrite or asking for function names and parameters. The resulting outputs are then compared with the gold answers using the task-specific metrics (Shi et al., 22 Oct 2025).

A significant methodological implication is that PhoneAgentBench does not use a single uniform metric. Its aggregate score pools planning alignment, exact classification accuracy, function-call exactness, BLEU-based rewriting quality, and entity extraction $F_1$. This makes the overall average interpretable only in conjunction with the per-task decomposition.

5. Empirical findings and use in multitask optimization

In the DaMo study, PhoneAgentBench functions both as an evaluation target and as a probe for data-mixture behavior in multitask supervised fine-tuning. The reported main results compare a baseline model without supervised fine-tuning, uniform and natural data mixtures, DML, and DaMo itself. On PAB Avg., the baseline without SFT scores 44.83, uniform mixing scores 61.52, natural mixing scores 65.33, DML scores 64.80, and DaMo reaches 68.18. Relative to the baseline, DaMo improves PhoneAgentBench performance by 23.35 percentage points; relative to DML, it improves by 3.38 points (Shi et al., 22 Oct 2025).

The per-task pattern is as important as the aggregate. Without SFT, the base model is already strong on some generic tasks, scoring 84.08 on MM-NER and 65.38 on MM-RR, and 68.77 on OS Avg. However, it is weak on phone-specific tasks such as MT-Plan at 20.00 and APP-Rec at 6.00. After fine-tuning, PhoneAgentBench scores rise sharply, and DaMo yields the top reported performance on MT-Plan at 55.50 and MM-NER at 83.34, with competitive performance on ACU and MM-RR. APP-Rec rises to 51.00 under DaMo, while Mobile-FC remains comparatively difficult at 47.79 (Shi et al., 22 Oct 2025).

These results support several benchmark-specific interpretations stated in the paper. First, PhoneAgentBench is described as challenging but tractable: strong pre-trained MLLMs do not perform well on several phone-specific tasks without targeted fine-tuning. Second, planning, app recognition, and mobile function calling appear to be the most difficult components and the most sensitive to training-data composition. Third, general multimodal benchmarks such as MME and OCRBench do not fully capture the phone-specific skills emphasized by PhoneAgentBench (Shi et al., 22 Oct 2025).

PhoneAgentBench also enters DaMo’s optimization objective directly. The downstream task-performance vector used in the framework includes the six PhoneAgentBench tasks together with external benchmarks. The paper further analyzes training dynamics through the lens of PhoneAgentBench subtasks. Training on the MMU dataset enhances ACU, conflicts with APP-Rec, has neutral impact on MM-NER, and leads to overfitting on MT-Plan, with initial gains followed by sharp decline. When APP-Rec and MMU are mixed, the APP-Rec performance surface over training steps and APP-Rec ratio is described as highly non-convex and non-monotonic. In this sense, PhoneAgentBench serves not only as a benchmark but also as an instrument for studying synergy, conflict, and sensitivity in multitask SFT (Shi et al., 22 Oct 2025).

6. Position in the benchmark landscape, limitations, and access

PhoneAgentBench occupies a specific niche within the mobile-agent evaluation landscape. In the DaMo paper, it is contrasted with text-oriented agent benchmarks that isolate planning, tool use, or memory, and with GUI-centric mobile benchmarks that emphasize grounding or element manipulation. Its distinctive contribution is the integration of planning, function calling, multimodal memory, screen understanding, and app recognition into a single phone-centric suite (Shi et al., 22 Oct 2025).

This position becomes clearer when contrasted with later benchmarks that emphasize different evaluation axes. SPA-Bench is a comprehensive smartphone-agent benchmark operating in an interactive Android environment, with 340 tasks, 11 distinct agents, and 7 metrics related to task completion and resource consumption, including multilingual coverage in English and Chinese (Chen et al., 2024). PhoneHarness Bench evaluates verifiable mobile workflows over mixed GUI, CLI, and host-side tool actions, with success defined by observable side effects and measured primarily by task pass rate on an annotated 124-task evaluation split (Li et al., 12 Jun 2026). Relative to these systems, PhoneAgentBench remains more compact and more answer-centric: it evaluates multimodal phone-agent capabilities through structured prediction tasks rather than long-horizon executed interaction traces. A plausible implication is that it is especially suited to studying model capability composition and fine-tuning effects, while benchmarks such as SPA-Bench and PhoneHarness Bench emphasize interactive execution, side-effect verification, or resource-aware evaluation.

Several limitations are acknowledged or strongly implied. The benchmark has 1,235 evaluation items, which is substantial for a curated suite but smaller than large static benchmarks. Many apps and contexts come from the Chinese ecosystem, including WeChat, QQ, Little Red Book, Pinduoduo, and Taobao, and the benchmark is not positioned as globally multilingual. Screenshots also reflect a single or limited set of phones and likely an Android-based ecosystem from a specific manufacturer. In addition, PhoneAgentBench does not model full long-horizon interactive episodes in which the agent executes actions on a device; it focuses on perception, planning, and function-call prediction (Shi et al., 22 Oct 2025).

PhoneAgentBench is released with the DaMo code and datasets at https://github.com/OPPO-Mente-Lab/DaMo.git. The repository is described as including the six PhoneAgentBench subtasks, the training datasets used in the experiments, and code for the DaMo data-mixing optimizer together with evaluation pipelines. The text does not specify licensing terms, which are instead likely provided in the repository (Shi et al., 22 Oct 2025).