InquireMobile: Safe Interactive Mobile Agent
- InquireMobile is a safety-oriented mobile agent that leverages a Vision-Language Model to initiate human inquiries in ambiguous mobile interactions.
- It integrates proactive inquiry into its pre-action reasoning to ensure that high-risk tasks like payments or file deletion are executed with user confirmation.
- Its accompanying benchmark, InquireBench, uses real smartphone data to evaluate inquiry success and task completion under various contextual challenges.
InquireMobile is a safety-oriented mobile agent built on a Vision-LLM (VLM) and designed for real smartphone graphical user interfaces in which fully autonomous execution is potentially unsafe. Its central premise is that a mobile agent should not merely perceive the screen and execute actions such as click, swipe, or type, but should also proactively request human confirmation or clarification when it encounters ambiguity, privacy-sensitive operations, missing permissions or login state, or high-risk actions such as payment, file deletion, app uninstall, and permission granting (Ai et al., 27 Aug 2025). The system is introduced together with InquireBench, a benchmark for safe interaction and proactive inquiry in mobile environments, and it exemplifies a broader shift in mobile inquiry research toward context-sensitive, human-in-the-loop decision support rather than uninterrupted autonomy (Ai et al., 27 Aug 2025).
1. Problem setting and safety rationale
The problem addressed by InquireMobile arises from the dominant design assumption in recent VLM-based mobile agents: after receiving a user instruction, the agent is expected to complete the task entirely on its own by perceiving the phone screen and issuing GUI actions. InquireMobile rejects that assumption in settings where model understanding or reasoning is insufficient, arguing that erroneous action on a mobile device can produce irreversible consequences including financial loss, unwanted deletion, privacy exposure, or unintended account and permission changes (Ai et al., 27 Aug 2025).
The benchmark scenarios that motivate this design include payments and money transfer, file deletion, app uninstall, privacy-sensitive operations such as login or permission granting, ambiguous user intent, and unexpected pop-ups, advertisements, or hard-to-reach apps. In these cases, the relevant failure is not merely low task success; it is unsafe execution under uncertainty. InquireMobile therefore treats inquiry as part of the action policy rather than as an external fallback (Ai et al., 27 Aug 2025).
This safety framing is consistent with the broader literature on Mobile Information Retrieval, which characterizes mobile interaction as context-sensitive, time-sensitive, and often embedded in real-world constraints rather than isolated from them. In that literature, relevance extends beyond topical matching to include time, location, activity, social context, and even real-world entities, making explicit user clarification a natural mechanism when the system cannot safely infer intent from screen state alone (Crestani et al., 2019).
2. InquireBench and the taxonomy of inquiry situations
InquireBench is the evaluation framework introduced alongside InquireMobile to measure whether a mobile agent can recognize situations requiring human help, formulate an appropriate inquiry, and continue interacting safely after asking. It is constructed from 80,345 raw screenshots collected by simulating random walks on real Android phones. From this pool, the benchmark contains 975 annotated inquiry samples, 173 tasks / instructions, and 37 apps. A separate evaluation subset contains 190 carefully curated instructions, split into 95 Chinese and 95 English items (Ai et al., 27 Aug 2025).
The construction process has two phases. First, random exploratory interactions are used to collect screenshots, and human annotators identify screens where an agent should ask for help rather than act blindly. Second, for each labeled screen, annotators write a valid task, GPT-4o-0806 generates the interactive reason and interactive content, and human annotators review and revise those outputs. The resulting benchmark therefore includes both the situation labels and the desired reasoning and inquiry language (Ai et al., 27 Aug 2025).
InquireBench organizes inquiry cases into five main categories and 22 sub-categories.
| Category | Definition | Representative cases |
|---|---|---|
| Intent Confirmation | The agent cannot determine the next action and needs clarification | Pop-ups, ads, ambiguous instructions |
| Privacy and Security | Actions require explicit user confirmation | Login, permission granting |
| Risk Scenarios | Actions are high-stakes or irreversible | Payment, file deletion, app uninstall, subscription, money transfer |
| Combination | Multiple inquiry reasons co-occur | Ambiguity plus risk, unlogged app plus task execution |
| Others | Cases outside the first four groups | Hard-to-reach apps, calls, messages, unauthorized map usage |
The appendix’s 22 subcategories include privacy cases such as opening an unlogged or unauthorized app; intention cases such as “My clothes have worn out,” “Movie is a great movie,” and updating a personal signature on social media; risk scenarios such as money transfer, subscribing to VIP service, deleting files, and uninstalling apps; combination cases such as navigating via an unauthorized map app or booking the cheapest plane ticket; and other cases such as opening system tools in a hard-to-reach folder or sending a message to a phone number (Ai et al., 27 Aug 2025).
The benchmark is intentionally realistic. It uses real phones as the main test environment, includes apps placed in folders or on non-initial screens, disables permissions in some cases, sets account status to match whether login help is needed, and supports both Chinese and English. Its three metrics are Task Success Rate (SR), Inquiry Success Rate (ISR), and Task Completion Score, the last of which is judged by GPT-4o (Ai et al., 27 Aug 2025).
3. Agent architecture and interactive action model
InquireMobile is a VLM-based mobile agent designed to be interactive and safety-aware rather than purely autonomous. At the behavioral level, it behaves like a conventional GUI agent when execution is straightforward, but it can pause and contact the user when it judges that the current state is risky, privacy-sensitive, or unclear (Ai et al., 27 Aug 2025).
Its main architectural innovation is an interactive pre-action reasoning mechanism. Before taking an action, the model produces a > section for internal reasoning and then a <tool_call> section for the final action. If the model concludes that the next step should not be executed autonomously, the final action can be call_user(content), which presents an inquiry to the user for confirmation or clarification (Ai et al., 27 Aug 2025).
The action space contains nine actions: key, click, swipe, long press, type, call_user, system_button, terminate, and wait. Inquiry is therefore not treated as a separate classifier attached to an executor, but as one action choice within the same policy that also performs ordinary GUI operations. This integration is significant because the system learns not only when to ask, but how to trade off inquiry against normal task execution under uncertainty (Ai et al., 27 Aug 2025).
The screens on which inquiry is expected include cases where the instruction is ambiguous, the next step cannot be safely inferred, the screen contains privacy or security sensitive operations, the action is high-risk, the app state or page context is unclear, or multiple conditions jointly indicate uncertainty or risk. The paper reports that most existing VLM-based agents exhibit near-zero performance on InquireBench, indicating that these inquiry conditions are not handled well by standard mobile-agent policies (Ai et al., 27 Aug 2025).
4. Two-stage training strategy and reward design
InquireMobile is trained using a two-stage strategy. Stage 1 is Supervised Fine-Tuning (SFT) on 975 inquiry samples and 3,000 general GUI samples. The role of the general GUI data is to preserve ordinary mobile-agent capability so that inquiry behavior does not collapse into indiscriminate caution. The paper reports, however, that SFT alone makes the model too cautious: it often asks for confirmation unnecessarily, which lowers task success and creates redundant interaction loops (Ai et al., 27 Aug 2025).
Stage 2 uses Group Relative Policy Optimization (GRPO) with rule-based rewards. GRPO is used instead of PPO, and the stated objective is to retain useful GUI skills, reduce unnecessary inquiries, ask only when appropriate, and improve reasoning under uncertainty. The relative advantage for a group of sampled outputs is defined as
The total reward is
where is a format reward, is an action-type reward, and is an action-argument reward (Ai et al., 27 Aug 2025).
The format reward explicitly favors the presence of
<think>and<tool_call>with a JSON action object; it is $1$ if the format matches and for errors. The action-type reward is $1$ when the predicted action type exactly matches ground truth and $0$ otherwise. The argument reward is action-dependent: for coordinate actions such as click and swipe, the reward is $1$ if the predicted coordinate falls within the ground-truth target bounding box and 0 otherwise; for text actions such as type and call_user, the reward is a BLEU score normalized to 1 (Ai et al., 27 Aug 2025).The base model is Qwen2.5-VL-3B. Stage 1 uses LoRA with rank 8, 2 epochs, and a learning rate of 2. Stage 2 uses GRPO, 2 epochs, 4 generations per sample, temperature = 1, and max steps per task = 15. Training uses 4 H100 GPUs. Evaluation uses a Huawei phone for testing, with Android Studio emulator and two physical Android phones in the broader development and training pipeline, and GPT-4o-0806 serves as the judge model for completion scoring and evaluation automation (Ai et al., 27 Aug 2025).
5. Experimental results, ablations, and failure modes
InquireMobile is evaluated against AppAgent, Mobile-Agent-E, Qwen2.5-VL-3B, GUI-R1-3B, UI-R1-3B, and UI-R1-3B-E. Its headline result is an Inquiry Success Rate (ISR) of 52.6%, which is reported as 46.8 percentage points higher than the best baseline. It also achieves the best overall success rate and the best trajectory score among the compared methods. The tabled results report Task Success Rate (SR) = 7.9% and Task Completion Score = 0.78 (Ai et al., 27 Aug 2025).
The ablation study distinguishes the roles of the two training stages. Stage 1 only yields high ISR but poor SR and completion score because it over-inquires. Stage 2 only improves task completion relative to Stage 1 but exhibits weaker inquiry behavior. Stage 1 + Stage 2 provides the best balance, supporting the interpretation that SFT teaches the model to recognize inquiry-worthy situations while GRPO teaches it to ask more selectively and act more effectively (Ai et al., 27 Aug 2025).
A human satisfaction study with about 200 participants in public places reports that the combined Stage 1 + Stage 2 system achieves the best scores: 2.3/5 English, 2.2/5 Chinese, and 2.25/5 average. The paper states that this is the only model above 2.0 on average, which it interprets as a preference for the more balanced inquiry behavior produced by the two-stage model (Ai et al., 27 Aug 2025).
The authors are explicit that performance remains far from ideal. Observed failure modes include grounding and navigation errors such as invalid clicks and poor localization in dynamic apps, app-level priors leading to opening the wrong app, in-app page comprehension failures involving icons or page structure, over-inquiry in Stage 1, and a broader limited inquiry ability overall. The low absolute task completion rate is presented as evidence that inquiry recognition is only one component of reliable mobile agency; robust visual grounding and navigation remain major bottlenecks (Ai et al., 27 Aug 2025).
6. Intellectual context and adjacent lines of research
InquireMobile belongs to a longer research trajectory in which mobile systems are expected to support information needs that arise in situ, under uncertainty, and often through social or human assistance rather than solitary autonomous computation. A direct precursor appears in the study of shopping uncertainty in mobile contexts, which found that 32% of participants sought extra information on web pages or forums and/or asked peers for advice, 38% reported suboptimal strategies such as buying all items, not buying, or choosing randomly, and 70% consequently had social or information needs when making purchasing decisions. That study explicitly motivated mobile applications that allow consumers to ask shopping questions to their social network while on the move (0906.2307).
Work on mobile information acquisition also clarifies why inquiry on the move is difficult. A shadowing and technology-probe study identified glance, inspect, and drift as three visual behaviors, found that drift best supports mobile information acquisition, and showed that opportunity use depends on situational awareness, switching costs, ongoing cognitive processes, and awareness of opportunities. It further reported that phones impose 2–6 second transition times, whereas OHMDs can support transitions on the order of a fraction of second. This suggests that any practical human-in-the-loop mobile agent must control the temporal and cognitive overhead of asking for help, not merely the correctness of the inquiry decision (Janaka et al., 2022).
The broader Mobile IR literature provides the conceptual backdrop for this emphasis on context and interaction. Mobile IR is defined as enabling users to perform classical IR operations on mobile devices, but it departs from desktop IR because relevance is shaped by context, time, location, activity, and social factors. InquireMobile can be read as a specialized mobile interaction system in which uncertainty about those factors is resolved by escalating to the user at the point of action (Crestani et al., 2019).
Several adjacent systems also illuminate the design space around InquireMobile. CO-oPS embeds community oversight into smartphone privacy and security decisions by allowing trusted community members to review installed apps and granted permissions, message one another, and discuss concerns in a community feed; this demonstrates that mobile decision support can be explicitly social and trust-mediated (Akter et al., 2024). Omicron provides an extensible Android platform for collecting mobile queries, sensor data, and interaction traces in mobile user studies, while Crepe provides a no-code Android app for collecting screen content using a Graph Query over accessibility-based UI structures; together, these systems suggest an empirical instrumentation stack through which future inquiry-aware mobile agents could be studied in realistic settings (Costa et al., 2020, Lu et al., 2024).
Within this landscape, InquireMobile’s distinctive contribution is to place inquiry inside the action policy of a VLM-based mobile agent and to benchmark that capability explicitly. This suggests a move from mobile systems that merely surface information or social feedback toward agents that can decide, in real time, when autonomous execution should yield to human judgment (Ai et al., 27 Aug 2025).