---
title: 'InquireMobile: Safe Interactive Mobile Agent'
url: https://www.emergentmind.com/topics/inquiremobile
type: topic
---

# InquireMobile: Safe Interactive Mobile Agent

InquireMobile is a safety-oriented mobile agent built on a Vision-Language Model (VLM) and designed for real smartphone graphical user interfaces in which fully autonomous execution is potentially unsafe. Its central premise is that a mobile agent should not merely perceive the screen and execute actions such as click, swipe, or type, but should also proactively request human confirmation or clarification when it encounters ambiguity, privacy-sensitive operations, missing permissions or login state, or high-risk actions such as payment, file deletion, app uninstall, and permission granting [2508.19679]. The system is introduced together with InquireBench, a benchmark for safe interaction and proactive inquiry in mobile environments, and it exemplifies a broader shift in mobile inquiry research toward context-sensitive, human-in-the-loop decision support rather than uninterrupted autonomy [2508.19679].

## 1. Problem setting and safety rationale

The problem addressed by InquireMobile arises from the dominant design assumption in recent VLM-based mobile agents: after receiving a user instruction, the agent is expected to complete the task entirely on its own by perceiving the phone screen and issuing GUI actions. InquireMobile rejects that assumption in settings where model understanding or reasoning is insufficient, arguing that erroneous action on a mobile device can produce irreversible consequences including financial loss, unwanted deletion, privacy exposure, or unintended account and permission changes [2508.19679].

The benchmark scenarios that motivate this design include payments and money transfer, file deletion, app uninstall, privacy-sensitive operations such as login or permission granting, ambiguous user intent, and unexpected pop-ups, advertisements, or hard-to-reach apps. In these cases, the relevant failure is not merely low task success; it is unsafe execution under uncertainty. InquireMobile therefore treats inquiry as part of the action policy rather than as an external fallback [2508.19679].

This safety framing is consistent with the broader literature on Mobile Information Retrieval, which characterizes mobile interaction as context-sensitive, time-sensitive, and often embedded in real-world constraints rather than isolated from them. In that literature, relevance extends beyond topical matching to include time, location, activity, social context, and even real-world entities, making explicit user clarification a natural mechanism when the system cannot safely infer intent from screen state alone [1902.01790].

## 2. InquireBench and the taxonomy of inquiry situations

InquireBench is the evaluation framework introduced alongside InquireMobile to measure whether a mobile agent can recognize situations requiring human help, formulate an appropriate inquiry, and continue interacting safely after asking. It is constructed from **80,345 raw screenshots** collected by simulating **random walks** on real Android phones. From this pool, the benchmark contains **975 annotated inquiry samples**, **173 tasks / instructions**, and **37 apps**. A separate evaluation subset contains **190 carefully curated instructions**, split into **95 Chinese** and **95 English** items [2508.19679].

The construction process has two phases. First, random exploratory interactions are used to collect screenshots, and human annotators identify screens where an agent should ask for help rather than act blindly. Second, for each labeled screen, annotators write a valid task, GPT-4o-0806 generates the **interactive reason** and **interactive content**, and human annotators review and revise those outputs. The resulting benchmark therefore includes both the situation labels and the desired reasoning and inquiry language [2508.19679].

InquireBench organizes inquiry cases into **five main categories** and **22 sub-categories**.

| Category | Definition | Representative cases |
|---|---|---|
| Intent Confirmation | The agent cannot determine the next action and needs clarification | Pop-ups, ads, ambiguous instructions |
| Privacy and Security | Actions require explicit user confirmation | Login, permission granting |
| Risk Scenarios | Actions are high-stakes or irreversible | Payment, file deletion, app uninstall, subscription, money transfer |
| Combination | Multiple inquiry reasons co-occur | Ambiguity plus risk, unlogged app plus task execution |
| Others | Cases outside the first four groups | Hard-to-reach apps, calls, messages, unauthorized map usage |

The appendix’s **22 subcategories** include privacy cases such as opening an unlogged or unauthorized app; intention cases such as “My clothes have worn out,” “Movie is a great movie,” and updating a personal signature on social media; risk scenarios such as money transfer, subscribing to VIP service, deleting files, and uninstalling apps; combination cases such as navigating via an unauthorized map app or booking the cheapest plane ticket; and other cases such as opening system tools in a hard-to-reach folder or sending a message to a phone number [2508.19679].

The benchmark is intentionally realistic. It uses real phones as the main test environment, includes apps placed in folders or on non-initial screens, disables permissions in some cases, sets account status to match whether login help is needed, and supports both Chinese and English. Its three metrics are **Task Success Rate (SR)**, **Inquiry Success Rate (ISR)**, and **Task Completion Score**, the last of which is judged by GPT-4o [2508.19679].

## 3. Agent architecture and interactive action model

InquireMobile is a VLM-based mobile agent designed to be interactive and safety-aware rather than purely autonomous. At the behavioral level, it behaves like a conventional GUI agent when execution is straightforward, but it can pause and contact the user when it judges that the current state is risky, privacy-sensitive, or unclear [2508.19679].

Its main architectural innovation is an **interactive pre-action reasoning** mechanism. Before taking an action, the model produces a `<think>` section for internal reasoning and then a `<tool_call>` section for the final action. If the model concludes that the next step should not be executed autonomously, the final action can be `call_user(content)`, which presents an inquiry to the user for confirmation or clarification [2508.19679].

The action space contains **nine actions**: **key**, **click**, **swipe**, **long press**, **type**, **call_user**, **system_button**, **terminate**, and **wait**. Inquiry is therefore not treated as a separate classifier attached to an executor, but as one action choice within the same policy that also performs ordinary GUI operations. This integration is significant because the system learns not only when to ask, but how to trade off inquiry against normal task execution under uncertainty [2508.19679].

The screens on which inquiry is expected include cases where the instruction is ambiguous, the next step cannot be safely inferred, the screen contains privacy or security sensitive operations, the action is high-risk, the app state or page context is unclear, or multiple conditions jointly indicate uncertainty or risk. The paper reports that most existing VLM-based agents exhibit near-zero performance on InquireBench, indicating that these inquiry conditions are not handled well by standard mobile-agent policies [2508.19679].

## 4. Two-stage training strategy and reward design

InquireMobile is trained using a two-stage strategy. **Stage 1** is **Supervised Fine-Tuning (SFT)** on **975 inquiry samples** and **3,000 general GUI samples**. The role of the general GUI data is to preserve ordinary mobile-agent capability so that inquiry behavior does not collapse into indiscriminate caution. The paper reports, however, that SFT alone makes the model too cautious: it often asks for confirmation unnecessarily, which lowers task success and creates redundant interaction loops [2508.19679].

**Stage 2** uses **Group Relative Policy Optimization (GRPO)** with rule-based rewards. GRPO is used instead of PPO, and the stated objective is to retain useful GUI skills, reduce unnecessary inquiries, ask only when appropriate, and improve reasoning under uncertainty. The relative advantage for a group of sampled outputs is defined as

$$
A_i = \frac{r_i - \mathrm{mean}(\{r_1,\dots,r_N\})}{\mathrm{std}(\{r_1,\dots,r_N\})}.
$$

The total reward is

$$
R = R_F + R_T + R_A,
$$

where \(R_F\) is a format reward, \(R_T\) is an action-type reward, and \(R_A\) is an action-argument reward [2508.19679].

The format reward explicitly favors the presence of `<think>` and `<tool_call>` with a JSON action object; it is \(1\) if the format matches and \(-1\) for errors. The action-type reward is \(1\) when the predicted action type exactly matches ground truth and \(0\) otherwise. The argument reward is action-dependent: for coordinate actions such as click and swipe, the reward is \(1\) if the predicted coordinate falls within the ground-truth target bounding box and \(0\) otherwise; for text actions such as type and call_user, the reward is a BLEU score normalized to \([0,1]\) [2508.19679].

The base model is **Qwen2.5-VL-3B**. Stage 1 uses **LoRA** with **rank 8**, **2 epochs**, and a learning rate of \(1.0 \times 10^{-4}\). Stage 2 uses **GRPO**, **2 epochs**, **4 generations per sample**, **temperature = 1**, and **max steps per task = 15**. Training uses **4 H100 GPUs**. Evaluation uses a **Huawei phone** for testing, with Android Studio emulator and two physical Android phones in the broader development and training pipeline, and GPT-4o-0806 serves as the judge model for completion scoring and evaluation automation [2508.19679].

## 5. Experimental results, ablations, and failure modes

InquireMobile is evaluated against **AppAgent**, **Mobile-Agent-E**, **Qwen2.5-VL-3B**, **GUI-R1-3B**, **UI-R1-3B**, and **UI-R1-3B-E**. Its headline result is an **Inquiry Success Rate (ISR) of 52.6%**, which is reported as **46.8 percentage points higher than the best baseline**. It also achieves the **best overall success rate** and the **best trajectory score** among the compared methods. The tabled results report **Task Success Rate (SR) = 7.9%** and **Task Completion Score = 0.78** [2508.19679].

The ablation study distinguishes the roles of the two training stages. **Stage 1 only** yields high ISR but poor SR and completion score because it over-inquires. **Stage 2 only** improves task completion relative to Stage 1 but exhibits weaker inquiry behavior. **Stage 1 + Stage 2** provides the best balance, supporting the interpretation that SFT teaches the model to recognize inquiry-worthy situations while GRPO teaches it to ask more selectively and act more effectively [2508.19679].

A human satisfaction study with about **200 participants** in public places reports that the combined Stage 1 + Stage 2 system achieves the best scores: **2.3/5 English**, **2.2/5 Chinese**, and **2.25/5 average**. The paper states that this is the only model above **2.0** on average, which it interprets as a preference for the more balanced inquiry behavior produced by the two-stage model [2508.19679].

The authors are explicit that performance remains far from ideal. Observed failure modes include **grounding and navigation errors** such as invalid clicks and poor localization in dynamic apps, **app-level priors** leading to opening the wrong app, **in-app page comprehension failures** involving icons or page structure, **over-inquiry in Stage 1**, and a broader **limited inquiry ability overall**. The low absolute task completion rate is presented as evidence that inquiry recognition is only one component of reliable mobile agency; robust visual grounding and navigation remain major bottlenecks [2508.19679].

## 6. Intellectual context and adjacent lines of research

InquireMobile belongs to a longer research trajectory in which mobile systems are expected to support information needs that arise in situ, under uncertainty, and often through social or human assistance rather than solitary autonomous computation. A direct precursor appears in the study of shopping uncertainty in mobile contexts, which found that **32%** of participants sought extra information on web pages or forums and/or asked peers for advice, **38%** reported suboptimal strategies such as buying all items, not buying, or choosing randomly, and **70%** consequently had social or information needs when making purchasing decisions. That study explicitly motivated mobile applications that allow consumers to ask shopping questions to their social network while on the move [0906.2307].

Work on mobile information acquisition also clarifies why inquiry on the move is difficult. A shadowing and technology-probe study identified **glance**, **inspect**, and **drift** as three visual behaviors, found that **drift** best supports mobile information acquisition, and showed that opportunity use depends on **situational awareness**, **switching costs**, **ongoing cognitive processes**, and **awareness of opportunities**. It further reported that phones impose **2–6 second transition times**, whereas OHMDs can support transitions on the order of a **fraction of second**. This suggests that any practical human-in-the-loop mobile agent must control the temporal and cognitive overhead of asking for help, not merely the correctness of the inquiry decision [2202.02748].

The broader Mobile IR literature provides the conceptual backdrop for this emphasis on context and interaction. Mobile IR is defined as enabling users to perform classical IR operations on mobile devices, but it departs from desktop IR because relevance is shaped by context, time, location, activity, and social factors. InquireMobile can be read as a specialized mobile interaction system in which uncertainty about those factors is resolved by escalating to the user at the point of action [1902.01790].

Several adjacent systems also illuminate the design space around InquireMobile. **CO-oPS** embeds community oversight into smartphone privacy and security decisions by allowing trusted community members to review installed apps and granted permissions, message one another, and discuss concerns in a community feed; this demonstrates that mobile decision support can be explicitly social and trust-mediated [2404.10258]. **Omicron** provides an extensible Android platform for collecting mobile queries, sensor data, and interaction traces in mobile user studies, while **Crepe** provides a no-code Android app for collecting screen content using a Graph Query over accessibility-based UI structures; together, these systems suggest an empirical instrumentation stack through which future inquiry-aware mobile agents could be studied in realistic settings [2001.11913] [2406.16173].

Within this landscape, InquireMobile’s distinctive contribution is to place inquiry inside the action policy of a VLM-based mobile agent and to benchmark that capability explicitly. This suggests a move from mobile systems that merely surface information or social feedback toward agents that can decide, in real time, when autonomous execution should yield to human judgment [2508.19679].

Source: https://www.emergentmind.com/topics/inquiremobile