MobileUse: Autonomous Mobile Operation

• MobileUse is a GUI agent for autonomous mobile operation that leverages multimodal large language models to execute user tasks on Android apps.
• It employs a hierarchical reflection mechanism to detect and recover from errors at multiple temporal scales, ensuring sustained task progress.
• Its proactive exploration module builds app-specific knowledge before task execution, improving success rates on benchmarks like AndroidWorld and AndroidLab.

Searching arXiv for papers on MobileUse and related mobile-agent benchmarks. MobileUse is a GUI agent for autonomous mobile operation that uses multimodal LLMs to execute user instructions on Android apps more robustly than prior agents. It was introduced to address two difficulties that dominate real mobile automation: robustness in long-horizon tasks and dynamic environments, and the cold-start problem in unfamiliar interfaces. Its defining mechanisms are a hierarchical reflection architecture for error detection and recovery at multiple temporal scales, and a proactive exploration module that accumulates reusable environment knowledge before task execution. Evaluations on AndroidWorld and AndroidLab report success rates of 62.9% and 44.2%, respectively, and the work also releases an out-of-the-box toolkit for automated task execution on physical mobile devices (Li et al., 21 Jul 2025).

1. Research setting and problem formulation

MobileUse is situated in the recent line of research on autonomous mobile agents, where the central problem is not only visual grounding over mobile screenshots but also sustained control under uncertainty. The paper explicitly argues that mobile tasks are not just a perception-and-action problem; they are also a long-horizon reasoning problem. In real mobile environments, an agent can make small mistakes that compound over time, enter repetitive loops, or terminate before the user goal is actually completed (Li et al., 21 Jul 2025).

This framing differs from earlier mobile-use research in HCI and mobile systems, where “mobile use” often referred to human behavior across device ecologies, opportunistic feature adoption, and cross-device trade-offs. Survey work on knowledge workers emphasized that device use is full of compromises among portability, functionality, convenience, synchronization burden, and task appropriateness (0801.4423). Monitoring-based studies of smartphones and tablets likewise found that multidevice behavior is common and that 50.9% of total interaction time can be classified as multidevice usage (Finley et al., 2018). MobileUse addresses a different layer of the problem: not how people distribute tasks across devices, but how an agent can execute those tasks autonomously on the device itself.

The immediate benchmark context for MobileUse was AndroidWorld and AndroidLab. A later benchmark, MobileWorld, made this landscape more demanding by introducing 201 tasks across 20 applications, long-horizon cross-application workflows, vague user instructions, and MCP-augmented tasks, with the best agentic framework reaching 51.7% overall success and the best end-to-end model 20.9% (Kong et al., 22 Dec 2025). This suggests that the problems MobileUse targeted—error recovery, trajectory stability, and unfamiliar-environment adaptation—are central to the broader trajectory of mobile-agent research rather than confined to one benchmark.

2. System architecture and control loop

Architecturally, MobileUse is a multi-agent framework organized into two stages: proactive exploration and autonomous mobile operation. In the exploration stage, the agent interacts with apps without being given a specific task, with the aim of accumulating general knowledge about the environment, such as app-specific interaction conventions or common UI patterns. In the execution stage, the agent receives a user instruction and acts in a loop involving an Operator, Hierarchical Reflectors, a Progressor, and previously collected knowledge (Li et al., 21 Jul 2025).

The Operator produces the next action from the instruction, retrieved task-relevant knowledge, the current screenshot, prior actions, the previous reflection, and the previous progress summary:

$$a^t = (a_t^t, a_a^t, a_d^t) = O(I, K_I, s^t, a^{:t-1}, r^{t-1}, p^{t-1}).$$

Here, $a_t^t$ is the thought, $a_a^t$ is the structured action, and $a_d^t$ is the natural-language description. After execution, hierarchical reflection generates feedback $r^t$, and the Progressor updates the running summary:

$p^t = P(a^{:t-1}, p^{t-1}, a^t, r^t).$

This design gives MobileUse both local control over the current step and an explicit memory of task progress (Li et al., 21 Jul 2025).

A notable design choice is that MobileUse is not a single-pass predictor. The Progressor compresses execution history into a running summary, while the reflectors selectively interrogate failures at different temporal scales. That combination is presented as the mechanism by which the system maintains coherence over long trajectories instead of relying on the backbone model alone (Li et al., 21 Jul 2025).

3. Hierarchical reflection and reflection-on-demand

The most distinctive mechanism in MobileUse is hierarchical reflection. The paper argues that action-level checking alone is insufficient for long tasks, because some failures arise not from immediate execution errors but from drift in the overall trajectory or from premature completion. MobileUse therefore uses three reflectors: Action Reflector, Trajectory Reflector, and Global Reflector (Li et al., 21 Jul 2025).

The Action Reflector provides immediate step-level feedback by comparing the screenshot before and after an action and checking whether the intended effect occurred. To support this, MobileUse includes a perception module $Pe$ that highlights visual differences between screenshots using bounding boxes. The reflection is formulated as:

$$r_a^t = R_{\text{action}}(I, s^t, Pe(s^{t+1}), a^t).$$

This reflector is designed to catch errors such as clicking the wrong app, typing into the wrong field, or failing to trigger the expected screen change (Li et al., 21 Jul 2025).

The Trajectory Reflector operates on a short recent window, typically the last 3–5 actions and their associated action-level reflections:

$$r_t^t = R_{\text{trajectory}}(I, p^{t-1}, (a^i, r_a^i), \dots, (a^t, r_a^t)).$$

Its role is to detect repeated actions, repeated screenshots, or accumulated action-level errors indicating that the agent is stuck or drifting away from the instruction. The Global Reflector is invoked when the Operator decides to terminate, and validates whether the user instruction has actually been fulfilled using historical actions and the latest screenshots:

$$r_g^t = R_{\text{global}}(I, (a^0, r_a^0, r_t^0), \dots, (a^t, r_a^t, r_t^t), s^j, \dots, s^t),$$

with $a_t^t$0 in the implementation to balance efficiency (Li et al., 21 Jul 2025).

MobileUse does not invoke reflection unconditionally. The paper introduces a “reflection-on-demand” strategy because reflecting at every step can add latency and can sometimes be harmful when the reflector hallucinates or overreacts. The Operator computes a confidence score for the predicted action type:

$a_t^t$1

where $a_t^t$2 are the tokens representing the action type, usually with $a_t^t$3. The Action Reflector is called only if $a_t^t$4. The Trajectory Reflector is triggered by repeated actions, repeated screenshots, or accumulated action-level errors rather than by a confidence threshold (Li et al., 21 Jul 2025).

The reported ablations and threshold analyses make the efficiency claim precise. Starting from Operator + Progressor, average success rate on AndroidWorld was 49.5%. Adding the Action Reflector raised this to 55.17%, the Trajectory Reflector to 56.1%, and the Global Reflector to 58.6%. Adding Reflection-on-Demand increased average success rate to 61.6%, and adding Proactive Exploration produced the final 62.9%. With a threshold such as $a_t^t$5, more than 85% of reflections can be omitted while the success rate drops by less than 1.5%, and a looser threshold such as $a_t^t$6 can even improve performance because it filters out harmful reflections (Li et al., 21 Jul 2025).

4. Proactive exploration and environment knowledge

The second major contribution is proactive exploration, introduced to address cold-start behavior in unfamiliar apps. Before solving a user task, the agent is allowed to explore the environment freely and collect reusable knowledge. The paper explicitly describes this as mimicking human exploration in a new app. Under a proactive exploration instruction rather than a task-specific goal, the agent generates trajectories

$a_t^t$7

from which a summary agent extracts useful operation patterns and compresses them into reusable experiences:

$a_t^t$8

A critic agent guides the action agent to explore more effectively by using the accumulated summaries (Li et al., 21 Jul 2025).

The practical significance of this module is that MobileUse does not wait until task execution to discover the interface structure. Instead, it constructs prior knowledge about app-specific conventions and common UI patterns that can later be retrieved as task-relevant knowledge $a_t^t$9. The paper’s case studies report that this helps in issues such as identifying high-priority tasks in an unfamiliar app (Li et al., 21 Jul 2025).

A plausible implication is that proactive exploration partially changes the ontology of mobile control from purely reactive instruction following to a two-phase regime: environment familiarization followed by goal-conditioned execution. That is consistent with the benchmark evidence reported for later systems, where failures increasingly arise from long-horizon state tracking, ambiguity handling, and tool-use decisions rather than from isolated taps alone (Kong et al., 22 Dec 2025).

5. Empirical performance and released toolkit

MobileUse was evaluated on AndroidWorld and AndroidLab. AndroidWorld contains 116 tasks across 20 apps with randomized parameters and many possible task variants. AndroidLab contains 138 tasks from 9 apps and reports success rate, sub-goal success rate, reversed redundancy ratio, and reasonable operation ratio. The main backbone model used in the reported experiments was Qwen2.5-VL-72B-Instruct with temperature set to 0 (Li et al., 21 Jul 2025).

On AndroidWorld, MobileUse achieved a 62.9% success rate. The paper states that this exceeds V-Droid at 59.5% by 3.4 percentage points, Agent-S2 at 54.3% by 8.6 points, and the plain Qwen2.5-VL-72B-Instruct baseline at 35.0% by 27.9 points. On AndroidLab, MobileUse reached 44.2% success rate and 50.01% sub-goal success rate, improving over V-Droid’s 38.30% success rate by 5.9 points. The same evaluation reports reversed redundancy ratio of 74.40 and reasonable operation ratio of 88.50. The authors note that MobileUse is not optimal on those auxiliary metrics, but argue that hierarchical reflection trades more execution for higher task completion (Li et al., 21 Jul 2025).

The paper also includes an analysis of reflection behavior itself. Hierarchical reflection corrected 18 failed tasks in one confusion-matrix analysis, corresponding to a 30.51% correction rate with only 7.02% misjudgment. It also reports reductions in multiple failure modes, especially perception and navigation failures. These numbers are central because they support the claim that MobileUse is not merely improving action prediction but stabilizing execution across several failure categories (Li et al., 21 Jul 2025).

In addition to benchmark evaluation, the work releases a MobileUse Toolkit for physical devices. The toolkit is described as out-of-the-box, lightweight, and modular; it is built on Android Debug Bridge and uses a Gradio-based web interface. A phone is connected via ADB, the toolkit is launched locally, the web UI is opened at http://127.0.0.1:7860, a VLM service is configured, and the task is entered for execution. The interface displays the execution process in real time and supports activation, deactivation, and customization of the reflection modules (Li et al., 21 Jul 2025).

6. Scope, relation to adjacent mobile-use systems, and open issues

MobileUse belongs to a larger shift in mobile computing from mobile devices as human-operated endpoints to mobile devices as sites of autonomous execution. Earlier systems had already used handheld devices as configurable control surfaces—for example, a Windows Phone communicating over Bluetooth with a Lego Mindstorms NXT robot in managed C#.NET (Tucker, 2014). Research infrastructure such as Omicron likewise treated the mobile phone as an instrumented study platform for collecting mobile query logs, sensor data, app usage statistics, and interaction traces in task-based laboratory and field studies (Costa et al., 2020). MobileUse differs in that it targets end-to-end task execution on the device rather than remote control or observational logging.

Its present scope is also bounded by the benchmarks on which it was evaluated. MobileUse was validated on AndroidWorld and AndroidLab, both of which focus on dynamic Android environments, but a later benchmark, MobileWorld, adds agent-user interaction and MCP-augmented tasks, with 22.4% of tasks in the former category and 19.9% in the latter. MobileWorld’s analysis reports recurring weaknesses in ambiguity detection and clarification, MCP context management, long-term memory and state tracking, complex logic reasoning, and temporal-spatial context awareness (Kong et al., 22 Dec 2025). This suggests that MobileUse’s hierarchical reflection directly addresses some of the identified bottlenecks—especially long-term memory drift and premature completion—while leaving open the question of how the same framework would behave when clarification and tool invocation become first-class actions.

A common misconception is to treat mobile agents as if they were only GUI navigators. The benchmark literature argues against that view. MobileWorld explicitly characterizes realistic mobile use as involving long-horizon tasks, cross-application interactions, vague user instructions, and hybrid tool usage (Kong et al., 22 Dec 2025). MobileUse is therefore best understood not as a simple tapping policy, but as a control architecture that couples perception, memory, task-progress summarization, selective reflection, and pre-task environment acquisition (Li et al., 21 Jul 2025).

Within that interpretation, its main significance is methodological. MobileUse shows that robust mobile automation can be improved by structuring failure analysis across three temporal scales and by learning the interface before task execution. The later benchmark trends suggest that these ingredients are likely to remain relevant as the field moves from single-app GUI manipulation toward collaborative, tool-augmented, and cross-application mobile intelligence (Kong et al., 22 Dec 2025).