MobiAgent: Modular Mobile GUI Automation

• MobiAgent is a modular mobile GUI agent framework that integrates visual perception, planning, execution, and memory for automated multi-step smartphone interactions.
• It employs hierarchical designs with specialized roles such as Manager, Operator, and Reflector, ensuring robust task planning and execution across diverse apps.
• Benchmarking results and self-evolution techniques highlight significant improvements in success rates, efficiency, and decision accuracy on complex mobile tasks.

MobiAgent is a name associated with a line of mobile GUI agent systems that automate multi-step smartphone interactions through explicit decomposition of perception, planning, execution, reflection, memory, and, in later variants, retrieval or replay. In the most specific usage, Mobile-Agent-v2 is “also referred to as MobiAgent,” while a later paper formalizes “MobiAgent” as a full-stack framework comprising the MobiMind-series agent models, the AgentRR acceleration framework, and the MobiFlow benchmarking suite (Wang et al., 2024, Zhang et al., 30 Aug 2025). Across related papers, the architecture evolves from the vision-centric single-agent Mobile-Agent, to multi-agent planning–decision–reflection systems, to hierarchical Manager–Operator designs with long-term memory, and finally to retrieval-augmented and record-replay systems intended for long-horizon, cross-application mobile automation (Wang et al., 2024, Wang et al., 20 Jan 2025, Zhou et al., 15 Nov 2025).

1. Terminology, lineage, and scope

The term “mobile agent” long predates contemporary GUI automation. In earlier distributed-systems literature, a MobileAgent was a software entity that “carries user-specified goals, partial results, and an itinerary plan,” migrates across firms via SupervisorAgents, queries ResourceAgents, and returns filtered results to an InterfaceAgent in B2B information retrieval workflows (Giza et al., 2014). That older usage is architecturally distinct from present mobile GUI agents, which act on screenshots, UI elements, and device-control APIs rather than traversing network hosts.

The modern lineage begins with “Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception,” which introduced an autonomous multi-modal mobile device agent that operates “in a vision-centric way,” without relying on XML files of apps or mobile system metadata (Wang et al., 2024). Mobile-Agent-v2 then restructured the system into Planning, Decision, and Reflection agents, explicitly targeting “task progress navigation” and “focus content navigation” under multi-step mobile-device operation (Wang et al., 2024). Mobile-Agent-E extended this direction with a hierarchical Manager plus subordinate Perceptor, Operator, Action Reflector, and Notetaker, together with self-evolution through long-term memory containing Tips and Shortcuts (Wang et al., 20 Jan 2025). Mobile-Agent-RAG, in turn, “builds on the familiar MobiAgent architecture” by introducing a two-level hierarchy—Manager and Operator—each with its own RAG module, plus specialized knowledge bases and the Mobile-Eval-RAG benchmark (Zhou et al., 15 Nov 2025). Finally, the later “MobiAgent: A Systematic Framework for Customizable Mobile Agents” recasts MobiAgent as a comprehensive system with dedicated agent models, acceleration, benchmarking, and an AI-assisted agile data collection pipeline (Zhang et al., 30 Aug 2025).

A useful editorial reading is that “MobiAgent” functions both as a specific system name and as a family resemblance among hierarchical mobile GUI agents. This suggests that the term now denotes not merely a single model, but a modular design pattern for mobile automation centered on role separation and structured external support.

2. Architectural decomposition

The original Mobile-Agent pipeline consists of a Visual Perception Module, a Multimodal LLM using GPT-4V as backbone, a task planner or self-planning loop, and an execution engine. At each step, the system captures screenshot $I_t$, detects text and icon boxes, sends the instruction, screenshot, detections, and history to GPT-4V, receives one action from $$\{\text{OpenApp, ClickText, ClickIcon, Type, Scroll, Back, Exit, Stop}\}$$, and converts that action into Android input events (Wang et al., 2024). The key design claim is vision-centric adaptability: the agent operates solely on pixels and thus avoids system-specific customizations.

Mobile-Agent-v2, also called MobiAgent, introduces a stricter modular split. Iteration $t$ is organized around a Visual Perception Module producing structured perception $P_t$, a Planning Agent generating a compressed text state called task progress, a Decision Agent choosing one low-level operation from a six-action space, and a Reflection Agent verifying the effect of the action by comparing pre- and post-operation screens. The Planning Agent updates

$TP_t = PA(Ins, O_{t-1}, TP_{t-1}, FC_{t-1}),$

while the Decision Agent produces

$O_t = DA(Ins, TP_t, FC_{t-1}, R_{t-1}, S_t, P_t).$

This decomposition is motivated by the claim that single-agent systems suffer from token-length and interleaved text-image context explosion (Wang et al., 2024).

Mobile-Agent-E deepens the hierarchy. The Manager computes an overall plan $W_P^{(t)}$ and immediate subgoal $W_S^{(t)}$; the Perceptor extracts UI elements; the Operator selects an atomic operation or Shortcut; the Action Reflector determines whether execution succeeded, partially succeeded, or failed; and the Notetaker aggregates task-relevant information into working memory $W_N^{(t)}$. The architecture is explicitly hierarchical in the sense that high-level planning is separated from low-level action execution (Wang et al., 20 Jan 2025).

Mobile-Agent-RAG simplifies the upper structure to a clear two-level hierarchy. Manager-RAG retrieves high-level human-validated task demonstrations to produce robust long-horizon plans, whereas Operator-RAG retrieves app-specific low-level exemplars grounded in real screenshots for accurate atomic actions. Supporting modules—Perceptor, Action Reflector, and Notetaker—remain in the loop, producing what the paper describes as a “plan, act, reflect, note” cycle (Zhou et al., 15 Nov 2025).

The systematic MobiAgent framework of 2025 reorganizes the stack once more around three model roles: Planner (4 B parameters), Decider (7 B), and Grounder (3 B). Here the Planner produces a high-level task plan $P$, the Decider issues action primitives $$\{\text{OpenApp, ClickText, ClickIcon, Type, Scroll, Back, Exit, Stop}\}$$0 in natural language, and the Grounder maps primitives to bounding boxes $$\{\text{OpenApp, ClickText, ClickIcon, Type, Scroll, Back, Exit, Stop}\}$$1 (Zhang et al., 30 Aug 2025). Relative to earlier systems, this version foregrounds model specialization and a full supporting infrastructure.

3. Perception, grounding, and device control

Visual perception is foundational throughout the MobiAgent line, but the implementation details vary. In Mobile-Agent, text localization uses off-the-shelf OCR such as CRAFT + CRNN, while icon localization uses Grounding DINO plus CLIP. The model represents fused multimodal state abstractly as

$$\{\text{OpenApp, ClickText, ClickIcon, Type, Scroll, Back, Exit, Stop}\}$$2

and maps abstract actions to Android ADB calls such as adb input tap, adb input text, and adb swipe (Wang et al., 2024).

Mobile-Agent-v2 inserts a lightweight Visual Perception Module that extracts structured text and icon coordinates from screenshots. The module applies OCR with ConvNextViT-document and icon detection with GroundingDINO + Qwen-VL-Int4, after which the Decision Agent chooses among Open app(app_name), Tap(x,y), Swipe((x1,y1),(x2,y2)), Type(text), Home, and Stop. Device control is executed “in real time over ADB (Android Debug Bridge) or Harmony OS equivalent” (Wang et al., 2024).

In Mobile-Agent-E, the Perceptor outputs a list of (icon,text,coords) using DBNet+ConvNextViT for OCR, GroundingDINO for icon grounding, and Qwen-VL-Plus for icon captioning. The Operator’s action space expands to $$\{\text{OpenApp, ClickText, ClickIcon, Type, Scroll, Back, Exit, Stop}\}$$3, where $$\{\text{OpenApp, ClickText, ClickIcon, Type, Scroll, Back, Exit, Stop}\}$$4 is the set of learned Shortcuts (Wang et al., 20 Jan 2025). The inclusion of Shortcuts means that low-level execution may invoke reusable multistep routines rather than only atomic operations.

The 2025 systematic MobiAgent makes grounding an explicit learned subproblem. Grounder training includes a reward with IoU and center-based terms,

$$\{\text{OpenApp, ClickText, ClickIcon, Type, Scroll, Back, Exit, Stop}\}$$5

and the Decider can be trained using the Grounder as a reward model for click actions (Zhang et al., 30 Aug 2025). This marks a shift from relying primarily on external MLLMs/VLMs toward explicit role-specialized agent models.

A common misconception is that all MobiAgent systems are purely vision-only. The early Mobile-Agent emphasizes operation solely on pixels and avoids XML or system metadata (Wang et al., 2024), but later systems and adjacent frameworks explicitly incorporate XML-based bbox generation, OmniParser, or structured perception outputs (Zhang et al., 30 Aug 2025). The family is therefore better characterized by modularization of GUI interaction than by a single invariant sensory interface.

4. Memory, self-evolution, retrieval, and replay

Memory appears first in compact task-focused form in Mobile-Agent-v2. The Memory Unit $$\{\text{OpenApp, ClickText, ClickIcon, Type, Scroll, Back, Exit, Stop}\}$$6 stores “focus content,” such as earlier match results or unread messages, and is updated by

$$\{\text{OpenApp, ClickText, ClickIcon, Type, Scroll, Back, Exit, Stop}\}$$7

Its purpose is to retain task-relevant content without forcing the agent to rescan a long sequence of screenshots (Wang et al., 2024).

Mobile-Agent-E turns memory into a persistent self-evolution mechanism. Long-term memory is partitioned into Tips $$\{\text{OpenApp, ClickText, ClickIcon, Type, Scroll, Back, Exit, Stop}\}$$8, which are textual heuristics, and Shortcuts $$\{\text{OpenApp, ClickText, ClickIcon, Type, Scroll, Back, Exit, Stop}\}$$9, which are JSON records containing a name, a screen-state precondition, argument mappings, and a sequence of atomic operations. After a task finishes, two Experience Reflector agents update these memories:

$t$0

During execution, the Manager conditions on Shortcuts in planning, while the Operator conditions on both Shortcuts and Tips when selecting actions and rectifying errors (Wang et al., 20 Jan 2025).

Mobile-Agent-RAG replaces internally accumulated experience with retrieval-oriented external knowledge. At the planning stage, Manager-RAG queries a high-level knowledge base $t$1 using the instruction $t$2, embeds documents with a contrastive text encoder $t$3, and ranks them by cosine similarity,

$t$4

The top-$t$5 retrieved pairs become few-shot exemplars for long-horizon planning. At execution time, Operator-RAG retrieves a single app-specific triplet $t$6 from $t$7, and action generation is biased by

$t$8

where $t$9 scores the fit of action $P_t$0 to context $P_t$1 (Zhou et al., 15 Nov 2025).

The two knowledge bases are deliberately specialized. The high-level base $P_t$2 is constructed from a “representative 50\% subset of Mobile-Eval-RAG tasks,” filtered for shortest successful traces and manually curated into 50 documents. The low-level per-app base $P_t$3 stores verified triplets of atomic subtasks, screenshots, and correct ADB commands, with “17–28 high-quality triplets” per app (Zhou et al., 15 Nov 2025). The paper’s explicit claim is that planning and execution require fundamentally distinct knowledge types.

The later systematic MobiAgent formalizes another form of externalized memory via AgentRR. AgentRR captures “task plan, action primitive, concrete UI operation” into an ActTree whose nodes are UI states and whose edges carry actions and associated tasks. It then uses latent memory models, cosine-similarity filtering, reranking, invalidation and eviction, and speculative replay via shortcuts to decide when cached actions can be replayed instead of invoking the VLM/LLM (Zhang et al., 30 Aug 2025). A plausible implication is that the family’s design trajectory moves from ephemeral context compression, to persistent experiential memory, to explicit retrieval stores, and finally to cache-like execution reuse.

5. Evaluation regimes and reported performance

The benchmark landscape co-evolves with the architectures. Mobile-Agent introduced Mobile-Eval, consisting of 10 mainstream Android apps plus 1 multi-app category, with 3 instructions per app and $P_t$4 distinct scenarios. Its metrics are Success (Su), Process Score (PS), Relative Efficiency (RE), and Completion Rate (CR). The reported aggregate results are “Average Success $P_t$5,” “Mean Process Score PS $P_t$6,” “Mean Relative Efficiency RE $P_t$7,” “Mean Completion Rate CR $P_t$8,” and in the multi-app category “100% on all metrics” (Wang et al., 2024).

Mobile-Agent-v2 uses “dynamic evaluation” on Harmony OS and Android OS, with 5 system apps, 5 external apps, app-specific basic and advanced instructions, and multi-app tasks, for a total of 88 instructions per scenario. It reports Success Rate (SR), Completion Rate (CR), Decision Accuracy (DA), and Reflection Accuracy (RA). In the non-English scenario, SR rises from 5/10 to 9/10 on System App Basic, from 3/10 to 6/10 on System App Advanced, from 2/10 to 8/10 on External App Basic, and from 1/2 to 2/2 on Multi-App Basic, for an “overall average SR improvement ≈ 30 %.” The English scenario shows “a similar uplift of ≈ 27 % SR,” with DA improving from “≈ 70 % to > 90 % in many subcategories,” and RA exceeding 90% (Wang et al., 2024).

Mobile-Agent-E introduces Mobile-Eval-E, with 25 tasks across Restaurant Recommendation, Information Researching, Online Shopping, What’s Trending, and Travel Planning. The benchmark is characterized by “76% multi-app tasks vs < 10% in prior benchmarks” and “Avg #Ops per task: 14.56 (⃗> 2× prior).” Using GPT-4o, the paper reports the following progression: AppAgent $P_t$9, Mobile-Agent-v1 $TP_t = PA(Ins, O_{t-1}, TP_{t-1}, FC_{t-1}),$0, Mobile-Agent-v2 $TP_t = PA(Ins, O_{t-1}, TP_{t-1}, FC_{t-1}),$1, Mobile-Agent-E $TP_t = PA(Ins, O_{t-1}, TP_{t-1}, FC_{t-1}),$2, and “+ Self-Evolution” $TP_t = PA(Ins, O_{t-1}, TP_{t-1}, FC_{t-1}),$3. The corresponding Action Accuracy rises to $TP_t = PA(Ins, O_{t-1}, TP_{t-1}, FC_{t-1}),$4, Reflection Accuracy to $TP_t = PA(Ins, O_{t-1}, TP_{t-1}, FC_{t-1}),$5, and Termination Error falls to $TP_t = PA(Ins, O_{t-1}, TP_{t-1}, FC_{t-1}),$6 (Wang et al., 20 Jan 2025).

Mobile-Agent-RAG introduces Mobile-Eval-RAG, a 50-task benchmark divided into “simple” and “complex” categories, with tasks averaging “14–19 steps across 2–3 apps.” Its six metrics are Success Rate (SR), Completion Rate (CR), Operator Accuracy (OA), Reflector Accuracy (RA), Steps, and Efficiency. Against prior multi-agent baselines, Mobile-Agent-RAG on Gemini-1.5-Pro raises “CR from 61.2% to 75.7% (+14.5 pp)” and “Efficiency from 2.81 to 4.03 (+43%).” “Overall task completion (SR) climbs from 56% to 76%,” with CR improving “by 23.0% (from 63.4% to 78.0%)” on simple tasks and “by 35.2% (from 54.9% to 74.2%)” on complex tasks (Zhou et al., 15 Nov 2025).

The systematic MobiAgent framework evaluates on MobiFlow, a DAG-based benchmarking suite with multi-level validation. On MobiFlow, the reported “overall average task score” is approximately 83% for MobiAgent, compared with approximately 72% for UI-TARS-1.5-7B, approximately 65% for Gemini-2.5 Pro, and approximately 60% for GPT-5. Easy tasks reach “~ 92%,” hard tasks “~ 74%,” and “termination reliability” is reported as “GPT failed to terminate in 11 categories; Gemini in 3; MobiAgent none.” AgentRR yields replay rates of “30%–60%” under uniform task distributions and “60%–85%” under realistic power-law distributions, with “Replay correctness > 99%” and end-to-end latency reduction of “~ 2×” on simple tasks and “~ 3×” on complex tasks (Zhang et al., 30 Aug 2025).

System	Benchmark	Reported result
Mobile-Agent	Mobile-Eval	Average Success $TP_t = PA(Ins, O_{t-1}, TP_{t-1}, FC_{t-1}),$7; CR $TP_t = PA(Ins, O_{t-1}, TP_{t-1}, FC_{t-1}),$8
Mobile-Agent-v2 / MobiAgent	Dynamic evaluation	Overall average SR improvement $TP_t = PA(Ins, O_{t-1}, TP_{t-1}, FC_{t-1}),$9
Mobile-Agent-E + Self-Evolution	Mobile-Eval-E	SS $O_t = DA(Ins, TP_t, FC_{t-1}, R_{t-1}, S_t, P_t).$0; AA $O_t = DA(Ins, TP_t, FC_{t-1}, R_{t-1}, S_t, P_t).$1; RA $O_t = DA(Ins, TP_t, FC_{t-1}, R_{t-1}, S_t, P_t).$2; TE $O_t = DA(Ins, TP_t, FC_{t-1}, R_{t-1}, S_t, P_t).$3
Mobile-Agent-RAG	Mobile-Eval-RAG	SR $O_t = DA(Ins, TP_t, FC_{t-1}, R_{t-1}, S_t, P_t).$4; CR $O_t = DA(Ins, TP_t, FC_{t-1}, R_{t-1}, S_t, P_t).$5
MobiAgent	MobiFlow	Overall average task score $O_t = DA(Ins, TP_t, FC_{t-1}, R_{t-1}, S_t, P_t).$6

These results should be interpreted with some care. One paper explicitly notes that no “formal convergence proofs or guaranteed optimality theorems” are provided for Mobile-Agent-E (Wang et al., 20 Jan 2025), and the systematic MobiAgent paper states that no explicit statistical significance tests are reported (Zhang et al., 30 Aug 2025).

6. Data pipelines, customization, and related training directions

The 2025 systematic MobiAgent places unusual emphasis on data construction and customization. Its AI-assisted agile data collection pipeline combines a lightweight recorder, XML-based bbox generation via OmniParser, Gemini-2.5-based reasoning reconstruction under the ReAct paradigm, and post-processing steps including task concatenation, prefix sampling, history augmentation, prompt generalization, and mixing single-step corner cases. The paper states that this pipeline reduces human annotation cost by “~70% compared to fully manual labelling,” while simultaneously supporting MobiMind training and MobiFlow evaluation (Zhang et al., 30 Aug 2025).

Training within MobiMind is staged. The Decider and Grounder are first warmed up by supervised fine-tuning; this is followed by “Two-Stage Curriculum Reinforcement Learning (GRPO),” including Grounding GRPO, Grounder-as-RM for Decider, curriculum learning over easy and hard actions, and self-evolution in which failed trajectories are corrected and merged into future training data (Zhang et al., 30 Aug 2025). Customization, in this formulation, is therefore not limited to prompt-level variation: it includes data curation, role-specific model design, acceleration policies, and task-template instantiation.

A related but distinct direction is MobileA3gent, a federated framework for training VLM-based mobile GUI agents from decentralized self-sourced data. It consists of Auto-Annotation, which generates low-level step descriptions and episode-level high-level instructions from users’ natural phone interactions, and FedVLM-A, which modifies federated aggregation through

$O_t = DA(Ins, TP_t, FC_{t-1}, R_{t-1}, S_t, P_t).$7

The framework emphasizes that “raw user data (screenshots, actions) never leaves device,” and reports “99.97% cost reduction” for Auto-Annotation relative to human labeling, as well as strong federated results under IID and non-IID settings (Wang et al., 5 Feb 2025). Although MobileA3gent is not itself called MobiAgent, it addresses the same bottleneck identified by later MobiAgent work: the scarcity and cost of high-quality mobile-agent training data.

This suggests that the contemporary MobiAgent research space is bifurcating into two complementary problems: inference-time system architecture, exemplified by planning, reflection, retrieval, and replay; and data-generation or training-time infrastructure, exemplified by AI-assisted collection and federated self-sourced supervision.

7. Limitations, misconceptions, and open problems

The papers identify several recurring failure modes. In the original Mobile-Agent, OCR may miss “very small or stylized fonts,” and CLIP matching can confuse visually similar icons, causing hallucinated text, misgrounding, or stalls (Wang et al., 2024). Mobile-Agent-v2 diagnoses a different source of failure: long token sequences and interleaved text-image histories degrade navigation performance, which is why the architecture externalizes task progress and focus content into short pure-text summaries (Wang et al., 2024). Mobile-Agent-RAG attributes two critical failure points in SoTA agents to excessive reliance on static internal MLLM knowledge: “strategic hallucinations in high-level planning” and “operational errors during low-level execution on user interfaces (UI)” (Zhou et al., 15 Nov 2025).

Several papers also qualify their own guarantees. Mobile-Agent-E states that reasoning and self-evolution rely on prompting frozen LMMs and that the system offers “empirical robustness rather than analytic guarantees” (Wang et al., 20 Jan 2025). The systematic MobiAgent paper lists “Environment drift,” the possibility that “app updates may invalidate experiences more rapidly than high-level plans,” the risk that offline trace evaluation can produce false negatives if agents discover new valid trajectories, and the “lack of end-to-end latency vs. energy trade-off analysis on real edge devices” (Zhang et al., 30 Aug 2025).

A further misconception is to equate MobiAgent with a single fixed interface strategy. The literature instead spans vision-only control, XML-assisted grounding, long-term textual memory, retrieval-oriented knowledge bases, and ActTree-based replay. The stable commonality is the explicit separation of roles—planning, action selection, validation, and contextual support—rather than any one perception stack or model family (Wang et al., 2024, Wang et al., 2024, Zhang et al., 30 Aug 2025).

Open directions are correspondingly diverse. Mobile-Agent proposed few-shot finetuning of visual detectors on UI data, richer gesture suites, and porting to iOS by replacing the execution engine (Wang et al., 2024). Mobile-Agent-RAG suggests growing knowledge bases via live user feedback and combining RAG with improved visual perception (Zhou et al., 15 Nov 2025). The systematic MobiAgent points to continual learning for UI change adaptation, cross-device and cross-platform extensions, richer multimodal signals such as touch pressure and haptics, and formal statistical significance analysis (Zhang et al., 30 Aug 2025). Taken together, these directions indicate that MobiAgent is evolving from a mobile-operation assistant into a broader research program on long-horizon, context-aware, and customizable mobile automation.