---
title: 'MobiAgent: Modular Mobile GUI Automation'
url: https://www.emergentmind.com/topics/mobiagent
type: topic
---

# MobiAgent: Modular Mobile GUI Automation

MobiAgent is a name associated with a line of mobile GUI agent systems that automate multi-step smartphone interactions through explicit decomposition of perception, planning, execution, reflection, memory, and, in later variants, retrieval or replay. In the most specific usage, Mobile-Agent-v2 is “also referred to as MobiAgent,” while a later paper formalizes “MobiAgent” as a full-stack framework comprising the MobiMind-series agent models, the AgentRR acceleration framework, and the MobiFlow benchmarking suite [2406.01014] [2509.00531]. Across related papers, the architecture evolves from the vision-centric single-agent Mobile-Agent, to multi-agent planning–decision–reflection systems, to hierarchical Manager–Operator designs with long-term memory, and finally to retrieval-augmented and record-replay systems intended for long-horizon, cross-application mobile automation [2401.16158] [2501.11733] [2511.12254].

## 1. Terminology, lineage, and scope

The term “mobile agent” long predates contemporary GUI automation. In earlier distributed-systems literature, a MobileAgent was a software entity that “carries user-specified goals, partial results, and an itinerary plan,” migrates across firms via SupervisorAgents, queries ResourceAgents, and returns filtered results to an InterfaceAgent in B2B information retrieval workflows [1406.0296]. That older usage is architecturally distinct from present mobile GUI agents, which act on screenshots, UI elements, and device-control APIs rather than traversing network hosts.

The modern lineage begins with “Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception,” which introduced an autonomous multi-modal mobile device agent that operates “in a vision-centric way,” without relying on XML files of apps or mobile system metadata [2401.16158]. Mobile-Agent-v2 then restructured the system into Planning, Decision, and Reflection agents, explicitly targeting “task progress navigation” and “focus content navigation” under multi-step mobile-device operation [2406.01014]. Mobile-Agent-E extended this direction with a hierarchical Manager plus subordinate Perceptor, Operator, Action Reflector, and Notetaker, together with self-evolution through long-term memory containing Tips and Shortcuts [2501.11733]. Mobile-Agent-RAG, in turn, “builds on the familiar MobiAgent architecture” by introducing a two-level hierarchy—Manager and Operator—each with its own RAG module, plus specialized knowledge bases and the Mobile-Eval-RAG benchmark [2511.12254]. Finally, the later “MobiAgent: A Systematic Framework for Customizable Mobile Agents” recasts MobiAgent as a comprehensive system with dedicated agent models, acceleration, benchmarking, and an AI-assisted agile data collection pipeline [2509.00531].

A useful editorial reading is that “MobiAgent” functions both as a specific system name and as a family resemblance among hierarchical mobile GUI agents. This suggests that the term now denotes not merely a single model, but a modular design pattern for mobile automation centered on role separation and structured external support.

## 2. Architectural decomposition

The original Mobile-Agent pipeline consists of a Visual Perception Module, a Multimodal Large Language Model using GPT-4V as backbone, a task planner or self-planning loop, and an execution engine. At each step, the system captures screenshot \(I_t\), detects text and icon boxes, sends the instruction, screenshot, detections, and history to GPT-4V, receives one action from \(\{\text{OpenApp, ClickText, ClickIcon, Type, Scroll, Back, Exit, Stop}\}\), and converts that action into Android input events [2401.16158]. The key design claim is vision-centric adaptability: the agent operates solely on pixels and thus avoids system-specific customizations.

Mobile-Agent-v2, also called MobiAgent, introduces a stricter modular split. Iteration \(t\) is organized around a Visual Perception Module producing structured perception \(P_t\), a Planning Agent generating a compressed text state called task progress, a Decision Agent choosing one low-level operation from a six-action space, and a Reflection Agent verifying the effect of the action by comparing pre- and post-operation screens. The Planning Agent updates
$$
TP_t = PA(Ins, O_{t-1}, TP_{t-1}, FC_{t-1}),
$$
while the Decision Agent produces
$$
O_t = DA(Ins, TP_t, FC_{t-1}, R_{t-1}, S_t, P_t).
$$
This decomposition is motivated by the claim that single-agent systems suffer from token-length and interleaved text-image context explosion [2406.01014].

Mobile-Agent-E deepens the hierarchy. The Manager computes an overall plan \(W_P^{(t)}\) and immediate subgoal \(W_S^{(t)}\); the Perceptor extracts UI elements; the Operator selects an atomic operation or Shortcut; the Action Reflector determines whether execution succeeded, partially succeeded, or failed; and the Notetaker aggregates task-relevant information into working memory \(W_N^{(t)}\). The architecture is explicitly hierarchical in the sense that high-level planning is separated from low-level action execution [2501.11733].

Mobile-Agent-RAG simplifies the upper structure to a clear two-level hierarchy. Manager-RAG retrieves high-level human-validated task demonstrations to produce robust long-horizon plans, whereas Operator-RAG retrieves app-specific low-level exemplars grounded in real screenshots for accurate atomic actions. Supporting modules—Perceptor, Action Reflector, and Notetaker—remain in the loop, producing what the paper describes as a “plan, act, reflect, note” cycle [2511.12254].

The systematic MobiAgent framework of 2025 reorganizes the stack once more around three model roles: Planner (4 B parameters), Decider (7 B), and Grounder (3 B). Here the Planner produces a high-level task plan \(P\), the Decider issues action primitives \(a_t\) in natural language, and the Grounder maps primitives to bounding boxes \(\mathrm{bbox}_t\) [2509.00531]. Relative to earlier systems, this version foregrounds model specialization and a full supporting infrastructure.

## 3. Perception, grounding, and device control

Visual perception is foundational throughout the MobiAgent line, but the implementation details vary. In Mobile-Agent, text localization uses off-the-shelf OCR such as CRAFT + CRNN, while icon localization uses Grounding DINO plus CLIP. The model represents fused multimodal state abstractly as
$$
\mathbf{h} = \mathrm{Transformer}\bigl[{\rm Enc}_{\rm img}(I), {\rm Enc}_{\rm txt}(T)\bigr],
$$
and maps abstract actions to Android ADB calls such as `adb input tap`, `adb input text`, and `adb swipe` [2401.16158].

Mobile-Agent-v2 inserts a lightweight Visual Perception Module that extracts structured text and icon coordinates from screenshots. The module applies OCR with ConvNextViT-document and icon detection with GroundingDINO + Qwen-VL-Int4, after which the Decision Agent chooses among `Open app(app_name)`, `Tap(x,y)`, `Swipe((x1,y1),(x2,y2))`, `Type(text)`, `Home`, and `Stop`. Device control is executed “in real time over ADB (Android Debug Bridge) or Harmony OS equivalent” [2406.01014].

In Mobile-Agent-E, the Perceptor outputs a list of `(icon,text,coords)` using DBNet+ConvNextViT for OCR, GroundingDINO for icon grounding, and Qwen-VL-Plus for icon captioning. The Operator’s action space expands to
\(\{\text{Open\_App, Tap, Swipe, Type, Enter, Switch\_App, Back, Home, Wait}\} \cup L_S\),
where \(L_S\) is the set of learned Shortcuts [2501.11733]. The inclusion of Shortcuts means that low-level execution may invoke reusable multistep routines rather than only atomic operations.

The 2025 systematic MobiAgent makes grounding an explicit learned subproblem. Grounder training includes a reward with IoU and center-based terms,
$$
R = R_{\mathrm{iou}} + R_{\mathrm{center}},
$$
and the Decider can be trained using the Grounder as a reward model for click actions [2509.00531]. This marks a shift from relying primarily on external MLLMs/VLMs toward explicit role-specialized agent models.

A common misconception is that all MobiAgent systems are purely vision-only. The early Mobile-Agent emphasizes operation solely on pixels and avoids XML or system metadata [2401.16158], but later systems and adjacent frameworks explicitly incorporate XML-based bbox generation, OmniParser, or structured perception outputs [2509.00531]. The family is therefore better characterized by modularization of GUI interaction than by a single invariant sensory interface.

## 4. Memory, self-evolution, retrieval, and replay

Memory appears first in compact task-focused form in Mobile-Agent-v2. The Memory Unit \(FC_t\) stores “focus content,” such as earlier match results or unread messages, and is updated by
$$
FC_t = FC_{t-1} \cup ExtractFocus(P_t, Ins).
$$
Its purpose is to retain task-relevant content without forcing the agent to rescan a long sequence of screenshots [2406.01014].

Mobile-Agent-E turns memory into a persistent self-evolution mechanism. Long-term memory is partitioned into Tips \(L_T\), which are textual heuristics, and Shortcuts \(L_S\), which are JSON records containing a name, a screen-state precondition, argument mappings, and a sequence of atomic operations. After a task finishes, two Experience Reflector agents update these memories:
$$
L_T \leftarrow \mathcal{A}_{ET}(\cdots), \qquad
L_S \leftarrow \mathcal{A}_{ES}(\cdots).
$$
During execution, the Manager conditions on Shortcuts in planning, while the Operator conditions on both Shortcuts and Tips when selecting actions and rectifying errors [2501.11733].

Mobile-Agent-RAG replaces internally accumulated experience with retrieval-oriented external knowledge. At the planning stage, Manager-RAG queries a high-level knowledge base \(K_{MR}\) using the instruction \(q=I\), embeds documents with a contrastive text encoder \(\phi(\cdot)\), and ranks them by cosine similarity,
$$
s(q,d)=\dfrac{\langle\phi(q),\phi(d)\rangle}{\|\phi(q)\|\;\|\phi(d)\|}.
$$
The top-\(k\) retrieved pairs become few-shot exemplars for long-horizon planning. At execution time, Operator-RAG retrieves a single app-specific triplet \((T_{OR}, S_{OR}, A_{OR})\) from \(K_{OR}^{\mathrm{app}}\), and action generation is biased by
$$
P(a\mid c)\propto \exp\!\bigl(f(c,a)/\tau\bigr),
$$
where \(f(c,a)\) scores the fit of action \(a\) to context \(c\) [2511.12254].

The two knowledge bases are deliberately specialized. The high-level base \(K_{MR}\) is constructed from a “representative 50\% subset of Mobile-Eval-RAG tasks,” filtered for shortest successful traces and manually curated into 50 documents. The low-level per-app base \(K_{OR}^{\mathrm{app}}\) stores verified triplets of atomic subtasks, screenshots, and correct ADB commands, with “17–28 high-quality triplets” per app [2511.12254]. The paper’s explicit claim is that planning and execution require fundamentally distinct knowledge types.

The later systematic MobiAgent formalizes another form of externalized memory via AgentRR. AgentRR captures “task plan, action primitive, concrete UI operation” into an ActTree whose nodes are UI states and whose edges carry actions and associated tasks. It then uses latent memory models, cosine-similarity filtering, reranking, invalidation and eviction, and speculative replay via shortcuts to decide when cached actions can be replayed instead of invoking the VLM/LLM [2509.00531]. A plausible implication is that the family’s design trajectory moves from ephemeral context compression, to persistent experiential memory, to explicit retrieval stores, and finally to cache-like execution reuse.

## 5. Evaluation regimes and reported performance

The benchmark landscape co-evolves with the architectures. Mobile-Agent introduced Mobile-Eval, consisting of 10 mainstream Android apps plus 1 multi-app category, with 3 instructions per app and \(11 \times 3 = 33\) distinct scenarios. Its metrics are Success (Su), Process Score (PS), Relative Efficiency (RE), and Completion Rate (CR). The reported aggregate results are “Average Success \(\approx 85\%\),” “Mean Process Score PS \(\approx 0.83\),” “Mean Relative Efficiency RE \(\approx 1.17\),” “Mean Completion Rate CR \(\approx 93\%\),” and in the multi-app category “100% on all metrics” [2401.16158].

Mobile-Agent-v2 uses “dynamic evaluation” on Harmony OS and Android OS, with 5 system apps, 5 external apps, app-specific basic and advanced instructions, and multi-app tasks, for a total of 88 instructions per scenario. It reports Success Rate (SR), Completion Rate (CR), Decision Accuracy (DA), and Reflection Accuracy (RA). In the non-English scenario, SR rises from 5/10 to 9/10 on System App Basic, from 3/10 to 6/10 on System App Advanced, from 2/10 to 8/10 on External App Basic, and from 1/2 to 2/2 on Multi-App Basic, for an “overall average SR improvement ≈ 30 %.” The English scenario shows “a similar uplift of ≈ 27 % SR,” with DA improving from “≈ 70 % to > 90 % in many subcategories,” and RA exceeding 90% [2406.01014].

Mobile-Agent-E introduces Mobile-Eval-E, with 25 tasks across Restaurant Recommendation, Information Researching, Online Shopping, What’s Trending, and Travel Planning. The benchmark is characterized by “76% multi-app tasks vs < 10% in prior benchmarks” and “Avg #Ops per task: 14.56 (⃗> 2× prior).” Using GPT-4o, the paper reports the following progression: AppAgent \(SS=25.2\%\), Mobile-Agent-v1 \(45.5\%\), Mobile-Agent-v2 \(53.0\%\), Mobile-Agent-E \(75.1\%\), and “+ Self-Evolution” \(\mathbf{86.9\%}\). The corresponding Action Accuracy rises to \(\mathbf{90.4\%}\), Reflection Accuracy to \(\mathbf{97.8\%}\), and Termination Error falls to \(\mathbf{12.0\%}\) [2501.11733].

Mobile-Agent-RAG introduces Mobile-Eval-RAG, a 50-task benchmark divided into “simple” and “complex” categories, with tasks averaging “14–19 steps across 2–3 apps.” Its six metrics are Success Rate (SR), Completion Rate (CR), Operator Accuracy (OA), Reflector Accuracy (RA), Steps, and Efficiency. Against prior multi-agent baselines, Mobile-Agent-RAG on Gemini-1.5-Pro raises “CR from 61.2% to 75.7% (+14.5 pp)” and “Efficiency from 2.81 to 4.03 (+43%).” “Overall task completion (SR) climbs from 56% to 76%,” with CR improving “by 23.0% (from 63.4% to 78.0%)” on simple tasks and “by 35.2% (from 54.9% to 74.2%)” on complex tasks [2511.12254].

The systematic MobiAgent framework evaluates on MobiFlow, a DAG-based benchmarking suite with multi-level validation. On MobiFlow, the reported “overall average task score” is approximately 83% for MobiAgent, compared with approximately 72% for UI-TARS-1.5-7B, approximately 65% for Gemini-2.5 Pro, and approximately 60% for GPT-5. Easy tasks reach “~ 92%,” hard tasks “~ 74%,” and “termination reliability” is reported as “GPT failed to terminate in 11 categories; Gemini in 3; MobiAgent none.” AgentRR yields replay rates of “30%–60%” under uniform task distributions and “60%–85%” under realistic power-law distributions, with “Replay correctness > 99%” and end-to-end latency reduction of “~ 2×” on simple tasks and “~ 3×” on complex tasks [2509.00531].

| System | Benchmark | Reported result |
|---|---|---|
| Mobile-Agent | Mobile-Eval | Average Success \(\approx 85\%\); CR \(\approx 93\%\) |
| Mobile-Agent-v2 / MobiAgent | Dynamic evaluation | Overall average SR improvement \(\approx 30\%\) |
| Mobile-Agent-E + Self-Evolution | Mobile-Eval-E | SS \(86.9\%\); AA \(90.4\%\); RA \(97.8\%\); TE \(12.0\%\) |
| Mobile-Agent-RAG | Mobile-Eval-RAG | SR \(56\%\to76\%\); CR \(61.2\%\to75.7\%\) |
| MobiAgent | MobiFlow | Overall average task score \(\approx 83\%\) |

These results should be interpreted with some care. One paper explicitly notes that no “formal convergence proofs or guaranteed optimality theorems” are provided for Mobile-Agent-E [2501.11733], and the systematic MobiAgent paper states that no explicit statistical significance tests are reported [2509.00531].

## 6. Data pipelines, customization, and related training directions

The 2025 systematic MobiAgent places unusual emphasis on data construction and customization. Its AI-assisted agile data collection pipeline combines a lightweight recorder, XML-based bbox generation via OmniParser, Gemini-2.5-based reasoning reconstruction under the ReAct paradigm, and post-processing steps including task concatenation, prefix sampling, history augmentation, prompt generalization, and mixing single-step corner cases. The paper states that this pipeline reduces human annotation cost by “~70% compared to fully manual labelling,” while simultaneously supporting MobiMind training and MobiFlow evaluation [2509.00531].

Training within MobiMind is staged. The Decider and Grounder are first warmed up by supervised fine-tuning; this is followed by “Two-Stage Curriculum Reinforcement Learning (GRPO),” including Grounding GRPO, Grounder-as-RM for Decider, curriculum learning over easy and hard actions, and self-evolution in which failed trajectories are corrected and merged into future training data [2509.00531]. Customization, in this formulation, is therefore not limited to prompt-level variation: it includes data curation, role-specific model design, acceleration policies, and task-template instantiation.

A related but distinct direction is MobileA3gent, a federated framework for training VLM-based mobile GUI agents from decentralized self-sourced data. It consists of Auto-Annotation, which generates low-level step descriptions and episode-level high-level instructions from users’ natural phone interactions, and FedVLM-A, which modifies federated aggregation through
$$
n_k^* = \lambda\,n_k^{epi} + n_k, \qquad
\omega_k = \frac{n_k^*}{\sum_{i\in\mathcal{S}^l} n_i^*}.
$$
The framework emphasizes that “raw user data (screenshots, actions) never leaves device,” and reports “99.97% cost reduction” for Auto-Annotation relative to human labeling, as well as strong federated results under IID and non-IID settings [2502.02982]. Although MobileA3gent is not itself called MobiAgent, it addresses the same bottleneck identified by later MobiAgent work: the scarcity and cost of high-quality mobile-agent training data.

This suggests that the contemporary MobiAgent research space is bifurcating into two complementary problems: inference-time system architecture, exemplified by planning, reflection, retrieval, and replay; and data-generation or training-time infrastructure, exemplified by AI-assisted collection and federated self-sourced supervision.

## 7. Limitations, misconceptions, and open problems

The papers identify several recurring failure modes. In the original Mobile-Agent, OCR may miss “very small or stylized fonts,” and CLIP matching can confuse visually similar icons, causing hallucinated text, misgrounding, or stalls [2401.16158]. Mobile-Agent-v2 diagnoses a different source of failure: long token sequences and interleaved text-image histories degrade navigation performance, which is why the architecture externalizes task progress and focus content into short pure-text summaries [2406.01014]. Mobile-Agent-RAG attributes two critical failure points in SoTA agents to excessive reliance on static internal MLLM knowledge: “strategic hallucinations in high-level planning” and “operational errors during low-level execution on user interfaces (UI)” [2511.12254].

Several papers also qualify their own guarantees. Mobile-Agent-E states that reasoning and self-evolution rely on prompting frozen LMMs and that the system offers “empirical robustness rather than analytic guarantees” [2501.11733]. The systematic MobiAgent paper lists “Environment drift,” the possibility that “app updates may invalidate experiences more rapidly than high-level plans,” the risk that offline trace evaluation can produce false negatives if agents discover new valid trajectories, and the “lack of end-to-end latency vs. energy trade-off analysis on real edge devices” [2509.00531].

A further misconception is to equate MobiAgent with a single fixed interface strategy. The literature instead spans vision-only control, XML-assisted grounding, long-term textual memory, retrieval-oriented knowledge bases, and ActTree-based replay. The stable commonality is the explicit separation of roles—planning, action selection, validation, and contextual support—rather than any one perception stack or model family [2401.16158] [2406.01014] [2509.00531].

Open directions are correspondingly diverse. Mobile-Agent proposed few-shot finetuning of visual detectors on UI data, richer gesture suites, and porting to iOS by replacing the execution engine [2401.16158]. Mobile-Agent-RAG suggests growing knowledge bases via live user feedback and combining RAG with improved visual perception [2511.12254]. The systematic MobiAgent points to continual learning for UI change adaptation, cross-device and cross-platform extensions, richer multimodal signals such as touch pressure and haptics, and formal statistical significance analysis [2509.00531]. Taken together, these directions indicate that MobiAgent is evolving from a mobile-operation assistant into a broader research program on long-horizon, context-aware, and customizable mobile automation.

Source: https://www.emergentmind.com/topics/mobiagent