---
title: 'InquireBench: Mobile Agent Inquiry Benchmark'
url: https://www.emergentmind.com/topics/inquirebench
type: topic
---

# InquireBench: Mobile Agent Inquiry Benchmark

InquireBench is a dynamic, real-device benchmark for evaluating **VLM-based mobile agents’ ability to interact safely with users via proactive inquiry**. Rather than treating mobile automation as a fully autonomous sequence of perception, navigation, and GUI execution, it evaluates **when and how an agent should ask the human for help or confirmation** before acting, especially in high-risk, privacy-sensitive, or underspecified situations. The benchmark is bilingual, runs on real Android devices and emulators, and centers **human-in-the-loop decision-making** as a first-class component of mobile-agent evaluation [2508.19679].

## 1. Motivation and conceptual scope

Existing GUI and mobile-agent benchmarks principally test screen perception, navigation, grounding, and action execution. Their operating assumption is that, once given an instruction, the agent should proceed **fully autonomously**. InquireBench was introduced to challenge that assumption, on the grounds that VLM-based agents often misinterpret user intent, lack robust uncertainty estimation, and may perform **irreversible or high-stakes operations** without confirmation, including unintended financial transactions, mishandling of sensitive personal data, and privacy violations through unauthorized logins and permissions [2508.19679].

The benchmark therefore targets three missing dimensions. First, **safe interaction**: an agent should avoid unsafe blind actions and identify states that require human approval. Second, **proactive inquiry**: the agent should voluntarily ask for clarification or confirmation at critical decision points. Third, **human-in-the-loop decision-making**: the user is embedded into the execution policy rather than treated as a one-shot instruction source. This makes InquireBench a testbed for **agent–human collaboration** in realistic mobile settings, not merely a benchmark for autonomous task completion [2508.19679].

A central implication is that correctness in InquireBench is not reducible to final task completion. An agent may fail even when it reaches the nominal goal if it does so by bypassing user consent in a risky or privacy-sensitive state. Conversely, appropriate invocation of user assistance becomes part of competent behavior rather than evidence of failure.

## 2. Task model and inquiry scenario taxonomy

InquireBench is executed on **real Android smartphones (Huawei and Xiaomi)** and an **Android Studio emulator**, using a mixture of **Chinese and English** commercial apps, including WeChat, Alipay, TikTok, Amazon, YouTube, Word, and Bilibili. The environments are configured to be realistic and restrictive: apps may be in folders, permissions may be disabled, and some applications may be logged out or unauthorized. The agent receives a **natural language instruction**, perceives the current screen visually, chooses GUI actions such as click, type, or swipe, and must also decide whether to invoke **`call_user(content)`** to ask a question or request confirmation [2508.19679].

The inquiry scenarios are organized into **5 main categories** and **22 sub-categories**. **Intent Confirmation** covers ambiguous instructions, interruptions such as pop-ups and advertisements, and vague or incomplete requests; the agent is expected to ask what the user actually wants instead of guessing. **Privacy and Security** covers login screens, permission granting, and opening applications in unlogged or unauthorized states; the correct policy is to seek explicit user confirmation. **Risk Scenarios** cover high-stakes operations such as **Money Transfer**, **Subscribe to VIP service**, **Delete {FILES}**, and **Uninstall {APPS}**; these require human approval even when the instruction appears clear. **Combination** scenarios jointly involve risk, privacy, app reachability, or multi-step complexity, such as navigation or shopping tasks in unauthorized or hard-to-reach apps. **Others** includes cases such as opening hard-to-reach apps or folders, calling a phone number, sending a message with partial information, or navigating when location-related permissions are unavailable [2508.19679].

Within this structure, **safe interaction** means not performing risky actions without user confirmation, handling privacy-sensitive operations with explicit consent, and avoiding irreversible changes unless the user clearly approves. **Proactive inquiry** means using `call_user(content)` to ask a clear, context-aware, concise, and relevant question **before** committing to a risky action or when intent is ambiguous. The benchmark therefore evaluates not only whether the agent can act, but whether it can detect **critical decision points** and suspend autonomy appropriately [2508.19679].

## 3. Dataset construction and annotation pipeline

The benchmark construction reverses the usual trajectory-collection process. Instead of starting from predefined tasks and collecting successful demonstrations, the authors first collected **80,345 screenshots** by having human operators perform **random walks** on real Android phones: opening apps, navigating, clicking, triggering pop-ups and dialog boxes, and simulating casual daily usage. Human annotators then screened these screenshots to find states where an agent **should not act autonomously** and instead must involve the user, assigning one of the five inquiry categories [2508.19679].

From these raw screenshots, the benchmark builders created **975 annotated data** from **173 tasks**. Each data point includes a screenshot, a category label, and a valid user instruction. For each selected image, annotators wrote a task satisfying **executability** on the actual device or app and **consistency with the category**. Prompt templates with placeholders such as `{APP_UNLOGINED}`, `{FILES}`, and `{PHONE_NUMBER}` were then instantiated to produce concrete instructions [2508.19679].

The annotation workflow has two additional layers. For **interactive thinking**, GPT‑4o‑0806 was prompted to generate an interactive reason and detailed internal reasoning explaining why the scenario requires human help; annotators then checked, edited, or rewrote that reasoning. For **inquiry content**, GPT‑4o generated the message to be shown via `call_user(content)`, and annotators removed redundancy, ensured politeness and concision, and aligned the external message with the internal reasoning. This produced supervision for both **when** inquiry is needed and **how** the inquiry should be phrased [2508.19679].

To preserve ordinary GUI competence, the authors also included **3,000 non-inquiry GUI trajectories** drawn from the same raw pool. Annotators wrote a task instruction for each trajectory and labeled each step with golden thoughts and golden actions. This means InquireBench is paired with both inquiry-centric supervision and general GUI-action supervision, so that models trained with it can remain ordinary mobile agents when inquiry is unnecessary [2508.19679].

| Category | Data | Instructions |
|---|---:|---:|
| Risk Scenarios | 52 | 12 |
| Privacy and Security | 145 | 33 |
| Intent Confirmation | 571 | 81 |
| Combination | 80 | 22 |
| Others | 127 | 25 |

In aggregate, the benchmark contains **975 data points**, **173 distinct instructions**, and **37 apps** [2508.19679].

## 4. Evaluation protocol and metrics

InquireBench operates in **dynamic mode**, not as a static screenshot benchmark. Each task is a step-by-step interactive episode: the agent sees the current screen, emits `<think>` and `<tool_call>`, the environment executes the action, a new screen is captured, and the loop continues for up to **15 steps per task**. Inputs include the task instruction, the current screenshot, and sometimes auxiliary metadata such as app list, language, or category. Outputs are structured into internal reasoning in `<think>` and a JSON action specification in `<tool_call>` [2508.19679].

The benchmark reports three principal metrics. **Task Success Rate (SR)** is the proportion of tasks completed successfully. **Inquiry Success Rate (ISR)** is the proportion of cases where the agent makes a correct inquiry at the appropriate time, including both using `call_user` when human assistance is needed and refraining from calling the user when no inquiry is required. **Task Completion Score (Score)** is a continuous score in the interval from 0 to 1, judged by **GPT‑4o‑0806** according to task completion, trajectory quality, and the reasonableness of inquiries and actions. These metrics are reported separately for Chinese and English and then averaged [2508.19679].

The underlying notion of correctness is intentionally hybrid. Inquiry behavior is judged against annotated scenario categories and ground-truth reasoning, while action quality is judged through interaction trajectories. This makes evaluation more stringent than one-shot answer checking: a system must recognize inquiry-required states, formulate relevant and well-timed questions, and still navigate the mobile environment coherently.

## 5. Action space, training interface, and empirical findings

The action space contains **9 types**: `click`, `swipe`, `long_press`, `type`, `key`, `system_button`, `terminate`, `wait`, and **`call_user(content)`**. InquireBench is the principal source of supervision for `call_user(content)`, including content quality. In the associated InquireMobile system, reinforcement fine-tuning uses the reward decomposition **\(R = R_F + R_T + R_A\)**, where \(R_F\) rewards output format, \(R_T\) rewards correct action type, and \(R_A\) rewards correct action arguments. For text-producing actions such as `type(text)` and `call_user(content)`, the argument reward is based on a normalized BLEU score against ground truth. The training framework is formulated with **GRPO**, using group-normalized advantage and a clipped policy objective [2508.19679].

Empirically, InquireBench was designed to expose a major weakness of current mobile agents: **near-zero capability in proactive inquiry and safe interaction**. Evaluated systems included framework-based agents such as **AppAgent** and **Mobile-Agent-E**, and model-based systems such as **Qwen2.5‑VL‑72B**, **GPT‑4o‑0806**, **Gemini‑2.5‑Pro**, **Claude‑3.5‑Sonnet v2**, **GUI‑R1‑3B**, **UI‑R1‑3B**, **UI‑R1‑3B‑E**, and **Qwen2.5‑VL‑3B**. These baselines showed very low SR and ISR: framework methods had SR mostly between **0–8.4%**, and model-based baselines typically had **ISR ≈ 2–10%** and **SR ≈ 0–9.5%**, with many categories at **0% SR** or **ISR close to zero** [2508.19679].

The benchmark was introduced together with **InquireMobile**, built on **Qwen2.5‑VL‑3B** and trained in two stages: supervised fine-tuning on the **975 inquiry data** and **3,000 general GUI trajectories**, followed by reinforcement fine-tuning with **GRPO**. The full **Stage1+Stage2** system achieved **ISR = 52.6%**, **SR = 7.90%**, and **Score = 0.78** on average, with **Chinese ISR = 49.5%** and **English ISR = 55.8%**. The paper highlights this as an **absolute ISR improvement of 46.8 points** over the best baseline, whose average ISR was approximately **5.80%** [2508.19679].

The ablation results clarify why the two-stage training matters. **Stage 1** alone learns the existence of the inquiry action and can reach relatively high ISR, but it **over-uses `call_user`**, issuing redundant inquiries and producing low SR. **Stage 2** alone improves action correctness and general GUI competence, but lacks strong inquiry reasoning. The combined system yields the best balance, with inquiries concentrated in genuinely ambiguous or risky states and fewer unnecessary interruptions. Human evaluation with approximately **200 participants** reported average satisfaction of **2.25/5** for the full InquireMobile system, compared with **1.80/5** for the best baseline, indicating that appropriate inquiry timing measurably affects user perception [2508.19679].

## 6. Behavioral interpretation, limitations, and release status

InquireBench is explicitly designed to test several behavioral competencies. These include **knowing when not to act autonomously**, **recognizing privacy-sensitive states**, **handling ambiguous or vague instructions**, **identifying hard-to-reach apps or permission settings**, and detecting formal **critical decision points** where the correct action is to issue `call_user(content)`, wait for clarification, and continue only after human input. The benchmark does not define a closed-form threshold condition for such decisions; instead, screenshot-level labels, task-level metadata such as `need_login`, and category annotations act as supervision signals for inquiry-required states [2508.19679].

The current benchmark also exposes substantial limitations. Even the best reported model remains weak in ordinary task completion, with **SR = 7.9%** and **Score = 0.78**, which indicates that the setting is highly challenging and that current systems remain far from robust real-world deployment. The benchmark covers **5 main categories** and **22 subcategories**, but not all safety or ethical concerns; examples such as long-term data retention or nuanced data-sharing policies are not modeled. Rewarding inquiry content with BLEU may miss semantic nuance and penalize acceptable alternative phrasings. In addition, the reliance on **GPT‑4o** plus human correction for reasoning and inquiry content is costly and may encode annotator or model biases [2508.19679].

The authors state that they will **open-source all datasets, models, and evaluation codes**. The released resources are described as including inquiry GUI data, general GUI data, the InquireMobile model, and evaluation scripts for Android devices or emulators. Practical use requires device or emulator setup, screenshot capture, parsing of `<tool_call>` outputs, and action execution through **ADB**. The benchmark is intentionally **realistic and restrictive**, requiring app installation, controlled permission states, bilingual environment matching, and logged-in or logged-out configurations aligned with task metadata [2508.19679].

In this sense, InquireBench marks a shift in benchmark design for mobile agents. It treats safe interruption, clarification, and consent as measurable components of competence, and thereby reframes mobile-agent evaluation from blind autonomy toward **collaborative, user-aware control**. A plausible implication is that future progress in mobile agents will depend not only on stronger perception and action models, but also on better uncertainty handling and more nuanced inquiry policies.

Source: https://www.emergentmind.com/topics/inquirebench