MagicGUI: Mobile GUI Agent Architecture

• MagicGUI is a foundational mobile GUI agent that combines scalable data pipelines, unified action space, and planning-oriented reasoning for autonomous GUI interaction.
• It leverages a two-stage training process—continued pre-training and reinforcement fine-tuning—to achieve high performance on diverse benchmark tasks.
• MagicGUI-RMS extends the framework with a multi-agent reward model system for automated feedback, corrective action proposals, and continual self-improvement.

MagicGUI is a foundational mobile GUI agent designed to address critical challenges in perception, grounding, and reasoning within real-world mobile GUI environments, while MagicGUI-RMS is a multi-agent reward model system for self-evolving GUI agents via automated feedback reflux that extends this line of work with adaptive trajectory evaluation, corrective feedback, and continual improvement (Tang et al., 19 Jul 2025, Li et al., 19 Jan 2026). Taken together, the two systems define a technical stack in which large-scale GUI-centric pre-training, planning-oriented action generation, reinforcement fine-tuning, and hierarchical reward modeling are integrated to improve autonomous interaction and reliable task execution across diverse applications.

1. System definition and overall organization

MagicGUI is presented as an end-to-end mobile GUI agent built around six core components: Scalable GUI Data Pipeline, Perception & Grounding, Unified Action Space, Planning-Oriented Reasoning, Two-Stage Training (CPT & RFT), and Benchmark Evaluation & Deployment (Tang et al., 19 Jul 2025). At each time step it consumes a screenshot $o_n$, a user instruction or task description $\text{task}$, and the history of past actions $a_{0:n-1}$, optionally augmented with past reasoning traces $t_{0:n-1}$, and produces either the next action $a_n$ alone or the pair $(t_n,a_n)$ when planning is enabled. The basic formulation is

$$a_n=\mathrm{VLM}\bigl(\text{task},a_{0:n-1},o_n\bigr),$$

and the planning-oriented form is

$$(t_n,a_n)=\mathrm{VLM}\bigl(\text{task},a_{0:n-1},t_{0:n-1},o_n\bigr).$$

MagicGUI-RMS operates on top of a base UI agent in a three-stage pipeline: Action Proposal, Hierarchical Reward Evaluation, and Dual-Loop Data Reflux (Li et al., 19 Jan 2026). Given task instruction $x$, current screen state $s$, and historical trajectory $\text{task}$0, the base UI Agent proposes

$\text{task}$1

The Domain-Specific Reward Model (DS-RM) then evaluates the action under deterministic UI rules, and the General-Purpose Reward Model (GP-RM) judges semantic correctness, predicts task completion, and scores action preferences. The selected action is fed back into the training loop, while DS-RM/GP-RM disagreements are used for retraining DS-RM.

Component	MagicGUI	MagicGUI-RMS
Primary role	Foundational mobile GUI agent	Multi-agent reward model system
Core inputs	Screenshot, task, action history, optional reasoning traces	Task instruction, screen state, historical trajectory, candidate action
Main output	Next action, optionally with intermediate meta-plan	Reward judgments, correction proposal, preferred action

A common misconception is to treat MagicGUI-RMS as a replacement for the GUI agent itself. The architecture described in the paper instead positions it as an auxiliary reward-model layer around a base UI agent, with the action proposal still originating from $\text{task}$2. A plausible implication is that the two papers should be read as complementary: the first establishes the agent substrate, and the second adds a reward-centered self-improvement loop.

2. Data construction and supervision at scale

MagicGUI introduces a four-stage GUI Data Pipeline consisting of Raw Data Collection, Data Preprocessing, Hierarchical Task Annotation, and Data Refinement (Tang et al., 19 Jul 2025). Raw data collection draws from open-source repositories including RICO, OS-ATLAS, AMEX, GUIAct, AndroidControl, and GUI-Odyssey, as well as automated crawling on real devices using Cloud Real Device Platform and UIAutomator2, plus manual collection of corner cases such as nested popups, dynamic elements, long-press, and swipe-only flows. Data preprocessing applies rule-based noise filtering, model-based semantic filtering, duplicate removal via perceptual-hash, tree-string comparison, and embedding similarity, and a unified element taxonomy with 20 types and unified action space. Hierarchical task annotation covers five perception tasks and four action-task levels, while reasoning traces are injected via rejection-sampling from chain-of-thought VLMs.

The CPT dataset contains 7.8 M samples: 3.1 M in-house, 2.8 M open-source general, and 1.9 M open-source GUI navigation (Tang et al., 19 Jul 2025). For the in-house portion, the reported proportions are Element Referring 17.2%, Element Grounding 24.1%, Element Description 10.2%, Screen Caption 23.0%, Screen VQA 11.8%, and Navigation Action 13.7%. Data refinement includes NovelSelect, which greedily picks the most “novel” samples according to

$\text{task}$3

with $\text{task}$4, and the paper reports that using only 50% of the full set matches full-data performance.

MagicGUI-RMS adds a structured data construction pipeline for reward learning that synthesizes a balanced reward dataset of 72 K samples through Rule-Based Verification, Structured Perturbation, and Intention-Centric Grounding Correction (Li et al., 19 Jan 2026). Rule-Based Verification enforces type alignment, spatial validity, and semantic equivalence to separate positive samples $\text{task}$5 from hard negatives $\text{task}$6. Structured Perturbation produces easy negatives $\text{task}$7 by instruction substitution and trajectory stitching. Intention-Centric Grounding Correction queries open-source UI agents $\text{task}$8, repairs spatial grounding when high-level intention is correct to generate additional positives, and labels intention failures as moderate negatives $\text{task}$9. The resulting dataset is described as balanced across difficulty—easy, moderate, and hard—and across in- and out-of-distribution domains.

These two data pipelines address different bottlenecks. MagicGUI’s 7.8 M-sample corpus is directed at broad capability acquisition in perception, grounding, and action modeling, whereas MagicGUI-RMS’s 72 K reward dataset is specialized for discriminating correct from incorrect actions and producing corrective feedback. This suggests a division between policy learning data and reward learning data rather than a single monolithic supervision source.

3. Perception, grounding, action formalization, and planning

MagicGUI improves fine-grained multimodal alignment by training simultaneously on five perception tasks: Element Referring, Element Grounding, Element Description, Screen Caption, and Screen VQA (Tang et al., 19 Jul 2025). Element Referring and Grounding require the model to classify an element into one of 20 types and predict its center coordinate $a_{0:n-1}$0. Screen Caption and VQA support holistic scene comprehension, and the CPT objective is expressed as

$a_{0:n-1}$1

where $a_{0:n-1}$2 is token length and $a_{0:n-1}$3 the vocab size. The visual encoder, from Qwen-VL, is unfrozen, and image resolution is dynamically varied.

The action interface is standardized through a unified action space. The paper states that MagicGUI standardizes 13 discrete action types, each represented as a token sequence in the action vocabulary $a_{0:n-1}$4, and that every action $a_{0:n-1}$5 is of the form ActionType$a_{0:n-1}$6 or ActionType$a_{0:n-1}$7 (Tang et al., 19 Jul 2025). The listed operations are: tap$a_{0:n-1}$8, scroll$a_{0:n-1}$9, text$t_{0:n-1}$0, navigate_back$t_{0:n-1}$1, navigate_home$t_{0:n-1}$2, long_press$t_{0:n-1}$3, finish$t_{0:n-1}$4, wait$t_{0:n-1}$5, enter$t_{0:n-1}$6, take_over$t_{0:n-1}$7, drag$t_{0:n-1}$8, call_api$t_{0:n-1}$9, screen_shot$a_n$0 / long_screen_shot$a_n$1, and no_answer$a_n$2. Tap, scroll, input, and drag carry grounding coordinates; text and call_api carry additional string arguments.

Planning-oriented reasoning is implemented by interleaving a plan update and an action execution within a single output using special tokens > … followed by <answer>…</answer> (Tang et al., 19 Jul 2025). The paper explicitly states that there is no explicit planning cost function; rather, the MLP head jointly generates plan and action, and the RFT reward guides both aspects implicitly. This framing places planning as a generative intermediate representation rather than a separately optimized symbolic planner.

MagicGUI-RMS adds a second representational layer around actions by decomposing reward evaluation into deterministic rule compliance and higher-level semantic judgment (Li et al., 19 Jan 2026). DS-RM outputs $a_n$3, where $a_n$4 indicates rule compliance, $a_n$5 is a textual rationale, and when $a_n$6, the model also proposes a corrected action and explanation. GP-RM then ingests the action together with DS-RM outputs and produces $a_n$7 for semantic correctness, $a_n$8 for task completion, and $a_n$9 for action-level preference. The system therefore separates local UI legality from global task semantics.

4. Optimization procedures and reward design

MagicGUI is trained in two stages: Continued Pre-training (CPT) and Reinforcement Fine-Tuning (RFT) (Tang et al., 19 Jul 2025). The base model is Qwen-VL (7B or 32B). CPT uses approximately 1 B tokens of GUI-centric dialogue data, and annealing on 350k high-quality, upsampled GUI-specific samples with learning rate reduced by 50% yields +1.0% on key tasks. RFT models the task as an MDP $(t_n,a_n)$0 with policy $(t_n,a_n)$1 and objective

$(t_n,a_n)$2

The RFT reward is a composite reward

$(t_n,a_n)$3

The paper specifies $(t_n,a_n)$4 if the action syntax is correct and $(t_n,a_n)$5 otherwise; $(t_n,a_n)$6 if action type, coordinates, and content match ground truth within a 14% bounding circle or $(t_n,a_n)$7 for text and $(t_n,a_n)$8 otherwise; and $(t_n,a_n)$9 if $$a_n=\mathrm{VLM}\bigl(\text{task},a_{0:n-1},o_n\bigr),$$0, else 0. Optimization uses DF-GRPO (Dual Filtering Group Relative Policy Optimization), with static filtering removing states where all sampled rollouts are uniformly correct or incorrect, and dynamic filtering ensuring both reward polarities appear in each group.

MagicGUI-RMS introduces a separate optimization regime for reward modeling (Li et al., 19 Jan 2026). For DS-RM, the step-level scalar reward is defined as

$$a_n=\mathrm{VLM}\bigl(\text{task},a_{0:n-1},o_n\bigr),$$1

where $$a_n=\mathrm{VLM}\bigl(\text{task},a_{0:n-1},o_n\bigr),$$2 is the ground-truth correctness. The supervised DS-RM objective combines cross-entropy for $$a_n=\mathrm{VLM}\bigl(\text{task},a_{0:n-1},o_n\bigr),$$3 and maximum-likelihood terms for corrections and rationales: $$a_n=\mathrm{VLM}\bigl(\text{task},a_{0:n-1},o_n\bigr),$$4 A second reinforcement-fine-tuning stage maximizes expected $$a_n=\mathrm{VLM}\bigl(\text{task},a_{0:n-1},o_n\bigr),$$5, for example with PPO or policy gradient. In the RMS paper’s implementation, GP-RM is treated as a high-capacity semantic arbiter and GPT-4o serves this role early on; GP-RM receives no gradient updates in the current pipeline.

The UI agent in MagicGUI-RMS is supervised on MagicGUI-Agent-39k and AndroidControl with AdamW, learning rate $$a_n=\mathrm{VLM}\bigl(\text{task},a_{0:n-1},o_n\bigr),$$6, batch size 8, and is then reinforcement fine-tuned with reward functions from MagicGUI plus DS-RM’s binary reward (Li et al., 19 Jan 2026). DS-RM is trained in Stage 1 on the 72K reward dataset and in Stage 2 with PPO/PG using AdamW, learning rate $$a_n=\mathrm{VLM}\bigl(\text{task},a_{0:n-1},o_n\bigr),$$7, batch size 16, followed by two rounds of disagreement-based reflux. This optimization structure connects the foundational reward design of MagicGUI with a second-layer reward discrimination system specialized for corrective feedback.

5. Automated feedback reflux and self-evolution

MagicGUI-RMS centers on an automated feedback reflux mechanism that couples UI Agent Data Reflux and RMS Data Reflux in a step-level loop (Li et al., 19 Jan 2026). For each step in an episode, the agent predicts $$a_n=\mathrm{VLM}\bigl(\text{task},a_{0:n-1},o_n\bigr),$$8, DS-RM evaluates it and may generate $$a_n=\mathrm{VLM}\bigl(\text{task},a_{0:n-1},o_n\bigr),$$9, GP-RM scores the alternatives, and the selected action is

$$(t_n,a_n)=\mathrm{VLM}\bigl(\text{task},a_{0:n-1},t_{0:n-1},o_n\bigr).$$0

The tuple $$(t_n,a_n)=\mathrm{VLM}\bigl(\text{task},a_{0:n-1},t_{0:n-1},o_n\bigr).$$1 is appended to the UI agent training set, and if $$(t_n,a_n)=\mathrm{VLM}\bigl(\text{task},a_{0:n-1},t_{0:n-1},o_n\bigr).$$2, the disagreement case is appended to the DS-RM training set. The environment then executes $$(t_n,a_n)=\mathrm{VLM}\bigl(\text{task},a_{0:n-1},t_{0:n-1},o_n\bigr).$$3 and proceeds to the next state.

The paper describes this as a dual-loop mechanism. UI Agent Reflux uses the GP-endorsed action $$(t_n,a_n)=\mathrm{VLM}\bigl(\text{task},a_{0:n-1},t_{0:n-1},o_n\bigr).$$4 as imitation supervision, thereby converting reward-model judgments into additional high-quality training data. RMS Reflux collects DS-RM versus GP-RM disagreements for high-priority retraining of DS-RM, refining its decision boundary over time. The reported intent is continual self-improvement without depending on manual annotation or static rule-based verification alone.

The functional rationale is stated explicitly. DS-RM captures precise interface rules; GP-RM enforces global semantics; the structured data pipeline ensures balanced, diverse training signals; and automated reflux closes the loop for continual self-improvement (Li et al., 19 Jan 2026). Observed failure modes also clarify the division of labor: early DS-RM may approve locally valid clicks that violate long-horizon intent, which GP-RM can correct, while GP-RM’s reliance on an external model limits end-to-end scalability. This makes the reflux mechanism both a training method and a diagnostic lens for identifying whether an error is primarily rule-level or semantic.

6. Empirical results, limitations, and research significance

MagicGUI reports competitive performance on one proprietary benchmark and seven public benchmarks (Tang et al., 19 Jul 2025). On Screen-QA, MagicGUI-CPT achieves 94.6 VQA score, second to UI-TARS-7B at 95.4 and above GPT-4o at 90.3 and Gemini at 90.4. On ScreenSpot v2 mobile, MagicGUI-CPT reaches 90.2 grounding accuracy, above UI-TARS-7B at 88.6 and Qwen2-VL at 70.7. On Os-Atlas-mobile, MagicGUI-CPT scores 95.2 versus UI-TARS-7B at 82.5. On the Chinese Magic-RICH benchmark with 4k low-level tasks, MagicGUI-CPT averages step success around 90–97% across Routine, Instruction, and Complex subsets, while MagicGUI-RFT boosts step success to 97.5%, 94.0%, and 74.1% respectively and retains approximately 92% on exception scenarios. On AndroidControl-Low, MagicGUI-RFT reaches 93.5% step success versus UI-TARS-7B at 91.8%; on AndroidControl-High, it reaches 76.3% versus 74.4%; and on GUI-Odyssey it records 74.3%, second to OS-ATLAS at 76.8% despite a different split. In reasoning ablations on single-step tap actions, “spatial” accuracy rises from 69.4% for CPT-full-act without reasoning to 72.1% for CPT-half-act-think, 79.3% for CPT-full-act-think, and 81.5% with additional RFT.

MagicGUI-RMS reports step-level action-prediction gains, reward discrimination gains, and self-evolution gains (Li et al., 19 Jan 2026). On AndroidControl-Low, MagicGUI-Agent achieves TM/EM = 97.2 / 93.5 versus UI-TARS-7B at 95.2 / 91.8. On AndroidControl-High, it records 84.7 / 76.3, which is +3.1 / +1.9 over UI-TARS. On MagicGUI-Agent-39k, it reaches 88.7 / 74.1, with the paper noting reduced subtle grounding errors. On RMS-72k reward discrimination, MagicGUI-RMS versus GPT-4o scores 93.6% versus 87.6% on Easy, 96.1% versus 54.6% on Moderate, and 68.0% versus 33.5% on Hard, with in-domain and out-of-domain splits showing especially large gains on hard cases of more than 30 points. In the self-evolution experiments, step-level success for MagicGUI-Agent improves from 74.1 to 76.6 to 78.6 across rounds 0, 1, and 2, while DS-RM improves from 73.6 to 76.5 to 78.3. Ablations show that DS-RM alone lifts hard-case accuracy from 29.5 to 66.7, GP-RM adds a further 2–3 points on hard and out-of-distribution cases, and injecting Explicit Operational Knowledge into DS-RM yields up to 96.1% on hard cases.

The limitations are also explicit. In MagicGUI-RMS, early DS-RM may approve locally valid clicks that violate long-horizon intent, and GP-RM’s reliance on an external model limits end-to-end scalability (Li et al., 19 Jan 2026). Future directions proposed in the paper are to train GP-RM end-to-end to reduce dependence on proprietary evaluators, extend explicit operational knowledge to multi-modal planning over desktop and web, and incorporate uncertainty-aware reflux to focus learning on high-value disagreement samples. For MagicGUI more broadly, the empirical pattern indicates that large-scale GUI-centric pre-training, unified perception/action modeling, and planning traces are most effective when coupled to spatially aware reward shaping and post hoc reward discrimination. A plausible implication is that the line of work represented by MagicGUI and MagicGUI-RMS is moving GUI agents from static supervised imitation toward closed-loop, reward-mediated self-refinement, while keeping the technical separation between policy generation and trajectory evaluation explicit.