InquireBench: Mobile Agent Inquiry Benchmark

• InquireBench is a dynamic benchmark that tests when mobile agents should request user confirmation to avoid risky actions.
• It emphasizes safe interaction and proactive inquiry, ensuring agents verify ambiguous instructions in privacy-sensitive scenarios.
• The benchmark uses real Android environments with annotated data to evaluate human-in-the-loop decision-making and overall task performance.

InquireBench is a dynamic, real-device benchmark for evaluating VLM-based mobile agents’ ability to interact safely with users via proactive inquiry. Rather than treating mobile automation as a fully autonomous sequence of perception, navigation, and GUI execution, it evaluates when and how an agent should ask the human for help or confirmation before acting, especially in high-risk, privacy-sensitive, or underspecified situations. The benchmark is bilingual, runs on real Android devices and emulators, and centers human-in-the-loop decision-making as a first-class component of mobile-agent evaluation (Ai et al., 27 Aug 2025).

1. Motivation and conceptual scope

Existing GUI and mobile-agent benchmarks principally test screen perception, navigation, grounding, and action execution. Their operating assumption is that, once given an instruction, the agent should proceed fully autonomously. InquireBench was introduced to challenge that assumption, on the grounds that VLM-based agents often misinterpret user intent, lack robust uncertainty estimation, and may perform irreversible or high-stakes operations without confirmation, including unintended financial transactions, mishandling of sensitive personal data, and privacy violations through unauthorized logins and permissions (Ai et al., 27 Aug 2025).

The benchmark therefore targets three missing dimensions. First, safe interaction: an agent should avoid unsafe blind actions and identify states that require human approval. Second, proactive inquiry: the agent should voluntarily ask for clarification or confirmation at critical decision points. Third, human-in-the-loop decision-making: the user is embedded into the execution policy rather than treated as a one-shot instruction source. This makes InquireBench a testbed for agent–human collaboration in realistic mobile settings, not merely a benchmark for autonomous task completion (Ai et al., 27 Aug 2025).

A central implication is that correctness in InquireBench is not reducible to final task completion. An agent may fail even when it reaches the nominal goal if it does so by bypassing user consent in a risky or privacy-sensitive state. Conversely, appropriate invocation of user assistance becomes part of competent behavior rather than evidence of failure.

2. Task model and inquiry scenario taxonomy

InquireBench is executed on real Android smartphones (Huawei and Xiaomi) and an Android Studio emulator, using a mixture of Chinese and English commercial apps, including WeChat, Alipay, TikTok, Amazon, YouTube, Word, and Bilibili. The environments are configured to be realistic and restrictive: apps may be in folders, permissions may be disabled, and some applications may be logged out or unauthorized. The agent receives a natural language instruction, perceives the current screen visually, chooses GUI actions such as click, type, or swipe, and must also decide whether to invoke call_user(content) to ask a question or request confirmation (Ai et al., 27 Aug 2025).

The inquiry scenarios are organized into 5 main categories and 22 sub-categories. Intent Confirmation covers ambiguous instructions, interruptions such as pop-ups and advertisements, and vague or incomplete requests; the agent is expected to ask what the user actually wants instead of guessing. Privacy and Security covers login screens, permission granting, and opening applications in unlogged or unauthorized states; the correct policy is to seek explicit user confirmation. Risk Scenarios cover high-stakes operations such as Money Transfer, Subscribe to VIP service, Delete {FILES}, and Uninstall {APPS}; these require human approval even when the instruction appears clear. Combination scenarios jointly involve risk, privacy, app reachability, or multi-step complexity, such as navigation or shopping tasks in unauthorized or hard-to-reach apps. Others includes cases such as opening hard-to-reach apps or folders, calling a phone number, sending a message with partial information, or navigating when location-related permissions are unavailable (Ai et al., 27 Aug 2025).

Within this structure, safe interaction means not performing risky actions without user confirmation, handling privacy-sensitive operations with explicit consent, and avoiding irreversible changes unless the user clearly approves. Proactive inquiry means using call_user(content) to ask a clear, context-aware, concise, and relevant question before committing to a risky action or when intent is ambiguous. The benchmark therefore evaluates not only whether the agent can act, but whether it can detect critical decision points and suspend autonomy appropriately (Ai et al., 27 Aug 2025).

3. Dataset construction and annotation pipeline

The benchmark construction reverses the usual trajectory-collection process. Instead of starting from predefined tasks and collecting successful demonstrations, the authors first collected 80,345 screenshots by having human operators perform random walks on real Android phones: opening apps, navigating, clicking, triggering pop-ups and dialog boxes, and simulating casual daily usage. Human annotators then screened these screenshots to find states where an agent should not act autonomously and instead must involve the user, assigning one of the five inquiry categories (Ai et al., 27 Aug 2025).

From these raw screenshots, the benchmark builders created 975 annotated data from 173 tasks. Each data point includes a screenshot, a category label, and a valid user instruction. For each selected image, annotators wrote a task satisfying executability on the actual device or app and consistency with the category. Prompt templates with placeholders such as {APP_UNLOGINED}, {FILES}, and {PHONE_NUMBER} were then instantiated to produce concrete instructions (Ai et al., 27 Aug 2025).

The annotation workflow has two additional layers. For interactive thinking, GPT‑4o‑0806 was prompted to generate an interactive reason and detailed internal reasoning explaining why the scenario requires human help; annotators then checked, edited, or rewrote that reasoning. For inquiry content, GPT‑4o generated the message to be shown via call_user(content), and annotators removed redundancy, ensured politeness and concision, and aligned the external message with the internal reasoning. This produced supervision for both when inquiry is needed and how the inquiry should be phrased (Ai et al., 27 Aug 2025).

To preserve ordinary GUI competence, the authors also included 3,000 non-inquiry GUI trajectories drawn from the same raw pool. Annotators wrote a task instruction for each trajectory and labeled each step with golden thoughts and golden actions. This means InquireBench is paired with both inquiry-centric supervision and general GUI-action supervision, so that models trained with it can remain ordinary mobile agents when inquiry is unnecessary (Ai et al., 27 Aug 2025).

Category	Data	Instructions
Risk Scenarios	52	12
Privacy and Security	145	33
Intent Confirmation	571	81
Combination	80	22
Others	127	25

In aggregate, the benchmark contains 975 data points, 173 distinct instructions, and 37 apps (Ai et al., 27 Aug 2025).

4. Evaluation protocol and metrics

InquireBench operates in dynamic mode, not as a static screenshot benchmark. Each task is a step-by-step interactive episode: the agent sees the current screen, emits > and <tool_call>, the environment executes the action, a new screen is captured, and the loop continues for up to 15 steps per task. Inputs include the task instruction, the current screenshot, and sometimes auxiliary metadata such as app list, language, or category. Outputs are structured into internal reasoning in <think> and a JSON action specification in <tool_call> (Ai et al., 27 Aug 2025).

The benchmark reports three principal metrics. Task Success Rate (SR) is the proportion of tasks completed successfully. Inquiry Success Rate (ISR) is the proportion of cases where the agent makes a correct inquiry at the appropriate time, including both using call_user when human assistance is needed and refraining from calling the user when no inquiry is required. Task Completion Score (Score) is a continuous score in the interval from 0 to 1, judged by GPT‑4o‑0806 according to task completion, trajectory quality, and the reasonableness of inquiries and actions. These metrics are reported separately for Chinese and English and then averaged (Ai et al., 27 Aug 2025).

The underlying notion of correctness is intentionally hybrid. Inquiry behavior is judged against annotated scenario categories and ground-truth reasoning, while action quality is judged through interaction trajectories. This makes evaluation more stringent than one-shot answer checking: a system must recognize inquiry-required states, formulate relevant and well-timed questions, and still navigate the mobile environment coherently.

5. Action space, training interface, and empirical findings

The action space contains 9 types: click, swipe, long_press, type, key, system_button, terminate, wait, and call_user(content). InquireBench is the principal source of supervision for call_user(content), including content quality. In the associated InquireMobile system, reinforcement fine-tuning uses the reward decomposition $R = R_F + R_T + R_A$, where $R_F$ rewards output format, $R_T$ rewards correct action type, and $R_A$ rewards correct action arguments. For text-producing actions such as type(text) and call_user(content), the argument reward is based on a normalized BLEU score against ground truth. The training framework is formulated with GRPO, using group-normalized advantage and a clipped policy objective (Ai et al., 27 Aug 2025).

Empirically, InquireBench was designed to expose a major weakness of current mobile agents: near-zero capability in proactive inquiry and safe interaction. Evaluated systems included framework-based agents such as AppAgent and Mobile-Agent-E, and model-based systems such as Qwen2.5‑VL‑72B, GPT‑4o‑0806, Gemini‑2.5‑Pro, Claude‑3.5‑Sonnet v2, GUI‑R1‑3B, UI‑R1‑3B, UI‑R1‑3B‑E, and Qwen2.5‑VL‑3B. These baselines showed very low SR and ISR: framework methods had SR mostly between 0–8.4%, and model-based baselines typically had ISR ≈ 2–10% and SR ≈ 0–9.5%, with many categories at 0% SR or ISR close to zero (Ai et al., 27 Aug 2025).

The benchmark was introduced together with InquireMobile, built on Qwen2.5‑VL‑3B and trained in two stages: supervised fine-tuning on the 975 inquiry data and 3,000 general GUI trajectories, followed by reinforcement fine-tuning with GRPO. The full Stage1+Stage2 system achieved ISR = 52.6%, SR = 7.90%, and Score = 0.78 on average, with Chinese ISR = 49.5% and English ISR = 55.8%. The paper highlights this as an absolute ISR improvement of 46.8 points over the best baseline, whose average ISR was approximately 5.80% (Ai et al., 27 Aug 2025).

The ablation results clarify why the two-stage training matters. Stage 1 alone learns the existence of the inquiry action and can reach relatively high ISR, but it over-uses call_user, issuing redundant inquiries and producing low SR. Stage 2 alone improves action correctness and general GUI competence, but lacks strong inquiry reasoning. The combined system yields the best balance, with inquiries concentrated in genuinely ambiguous or risky states and fewer unnecessary interruptions. Human evaluation with approximately 200 participants reported average satisfaction of 2.25/5 for the full InquireMobile system, compared with 1.80/5 for the best baseline, indicating that appropriate inquiry timing measurably affects user perception (Ai et al., 27 Aug 2025).

6. Behavioral interpretation, limitations, and release status

InquireBench is explicitly designed to test several behavioral competencies. These include knowing when not to act autonomously, recognizing privacy-sensitive states, handling ambiguous or vague instructions, identifying hard-to-reach apps or permission settings, and detecting formal critical decision points where the correct action is to issue call_user(content), wait for clarification, and continue only after human input. The benchmark does not define a closed-form threshold condition for such decisions; instead, screenshot-level labels, task-level metadata such as need_login, and category annotations act as supervision signals for inquiry-required states (Ai et al., 27 Aug 2025).

The current benchmark also exposes substantial limitations. Even the best reported model remains weak in ordinary task completion, with SR = 7.9% and Score = 0.78, which indicates that the setting is highly challenging and that current systems remain far from robust real-world deployment. The benchmark covers 5 main categories and 22 subcategories, but not all safety or ethical concerns; examples such as long-term data retention or nuanced data-sharing policies are not modeled. Rewarding inquiry content with BLEU may miss semantic nuance and penalize acceptable alternative phrasings. In addition, the reliance on GPT‑4o plus human correction for reasoning and inquiry content is costly and may encode annotator or model biases (Ai et al., 27 Aug 2025).

The authors state that they will open-source all datasets, models, and evaluation codes. The released resources are described as including inquiry GUI data, general GUI data, the InquireMobile model, and evaluation scripts for Android devices or emulators. Practical use requires device or emulator setup, screenshot capture, parsing of <tool_call> outputs, and action execution through ADB. The benchmark is intentionally realistic and restrictive, requiring app installation, controlled permission states, bilingual environment matching, and logged-in or logged-out configurations aligned with task metadata (Ai et al., 27 Aug 2025).

In this sense, InquireBench marks a shift in benchmark design for mobile agents. It treats safe interruption, clarification, and consent as measurable components of competence, and thereby reframes mobile-agent evaluation from blind autonomy toward collaborative, user-aware control. A plausible implication is that future progress in mobile agents will depend not only on stronger perception and action models, but also on better uncertainty handling and more nuanced inquiry policies.