MemGUI-Agent: Context-Aware Mobile GUI Agent
- MemGUI-Agent is a memory-augmented mobile GUI agent that integrates context management as an explicit action to preserve UI-derived facts across long, cross-application workflows.
- The ConAct formulation decomposes context into folded action history, durable memory storage, and recent step records, enabling proactive context compression and reducing token overhead.
- Evaluations on benchmarks like MemGUI-Bench and MobileWorld show that MemGUI-Agent reduces process and output hallucinations while achieving higher Pass metrics compared to ReAct-style prompting.
MemGUI-Agent denotes, in its most specific usage, the end-to-end long-horizon mobile GUI agent introduced with Context-as-Action (ConAct), and, in a broader research usage, a memory-augmented GUI agent design target for workflows in which intermediate facts must survive many interaction steps and app transitions. The specific ConAct-based system addresses a failure mode of ReAct-style prompting in mobile GUI control: per-step records are passively accumulated, prompts expand with horizon length, and critical cross-app facts become diluted or truncated. MemGUI-Agent instead treats context management as part of the policy itself, so that the same MLLM emits both UI actions and context-maintenance operations over structured memory fields (Liu et al., 18 Jun 2026).
1. Definition and problem setting
MemGUI-Agent is situated in the long-horizon mobile GUI regime, where the agent operates from screenshot observations in Android environments, without a reliable DOM tree, and must execute actions such as tapping, typing, swiping, pressing system buttons, waiting, and terminating. The tasks emphasized in this literature are multi-step and frequently cross-application: examples include product comparison across shopping, calculator, and note-taking apps, or contact extraction in one app followed by reconstruction and messaging in others. In the cited benchmark setting, trajectories average 28–36 steps, and MemGUI-Bench reports 36.2 golden steps on average with 78.1% of tasks requiring cross-application information transfer (Liu et al., 18 Jun 2026, Liu et al., 3 Feb 2026).
The motivating claim is that long-horizon unreliability is not reducible to immediate perception failure. Rather, the agent must preserve exact UI-derived facts—prices, dates, verification codes, contact strings, addresses, copied text, and intermediate decisions—over many screens and across application boundaries. ReAct-style history accumulation is described as creating two concrete pathologies: prompt explosion and dilution of critical facts. The MemGUI-Agent paper states that, on MemGUI-Bench tasks, ConAct’s proactive folding saves about 1.5k input tokens compared to ReAct at 150 steps, while its error analysis attributes large reductions in process and output hallucinations to proactive context management (Liu et al., 18 Jun 2026).
Related benchmark work strengthens the premise that memory is a first-order bottleneck rather than a secondary convenience. MemGUI-Bench was explicitly built because existing dynamic mobile benchmarks contained only 5.2–11.8% memory-related tasks and did not evaluate cross-session learning; it instead provides 128 tasks across 26 applications, with 89.8% challenging memory through cross-temporal and cross-spatial retention (Liu et al., 3 Feb 2026). This broader context explains why “MemGUI-Agent” is often used in adjacent papers as a generic shorthand for a memory-centric GUI agent design target, even when the concrete implementation differs.
2. ConAct formulation and system design
The defining technical contribution of MemGUI-Agent is ConAct, or Context-as-Action. The agent state is formalized as , where is the screenshot at step , is the instruction or goal, and is structured context. That context is decomposed into three fields, , with complementary roles (Liu et al., 18 Jun 2026).
| Context field | Meaning | Function |
|---|---|---|
| Folded Action History | Compressed trajectory summaries | |
| Folded UI State | Persistent, structured UI-derived facts | |
| Recent Step Record | Grounded record of the last step |
stores compact summaries of completed step spans or individual steps. 0 stores durable, structured memory items whose content is preserved verbatim, such as `"Samsung A55 8GB/256GB at 1L_tI_t$2, while context-facing actions are $I_t$3. A single MLLM policy emits both kinds of outputs: $I_t$4. Here $I_t$5 is a folding directive over step range $I_t$6 with summary $I_t$7, $I_t$8 is a grounded UI observation, and $I_t$9 is the next-step action intent (Liu et al., 18 Jun 2026).
The update operators are explicitly defined. Folded history is updated by $t$0. Memory is updated by add, update, or delete operators only when the chosen action is a memory action; otherwise $t$1 is preserved. The recent step record is $t$2, with $t$3 denoting the tool or environment result. UI actions change the screen through the environment, whereas memory actions do not change the screen and instead modify only $t$4. This distinction corrects a common misconception: memory mutation in MemGUI-Agent is not an implicit side effect of reasoning but an explicit action class with operational semantics (Liu et al., 18 Jun 2026).
A token-budgeted view is also given. The serialized context must satisfy $t$5. The paper presents fact retention conceptually as a constrained selection problem over candidate facts under a token budget, but notes that MemGUI-Agent does not hard-code an importance function $t$6; prioritization is learned implicitly through policy training and fold directives. This suggests that ConAct is not merely a prompt template but a learned context-allocation mechanism.
The model’s step output is serialized in a five-part schema:
<thinking>,
<folding>,
<tool_call>,
<ui_observation>,
and <action_intent>.
The environment executes exactly one action per step. This one-action-per-step constraint is later identified as a limitation, because UI progression and memory bookkeeping sometimes must be interleaved rather than co-issued (Liu et al., 18 Jun 2026).
3. Dataset construction and training regime
The supervised substrate for small- and mid-scale MemGUI-Agent models is MemGUI-3K, a dataset released as 2,956 successful trajectories across 26 Android apps and 7 categories, with zero overlap with MemGUI-Bench tasks. Its trajectories average 28.8 steps, median 25, and are described as approximately 1.9× longer than GUIOdyssey. The dataset is not a raw trajectory dump: step-level reasonableness filtering keeps 64,430 reasonable SFT samples, split into 57,951 train and 6,479 test, averaging 21.8 steps per trajectory (Liu et al., 18 Jun 2026).
MemGUI-3K is specifically organized around ConAct annotations rather than only action labels. The paper reports that 65.1% of trajectories contain memory actions, 88.7% use at least one span-level fold, and 23.8% of folds are span-level with mean span 6.25 steps. The trajectories are expanded from 128 MemGUI-Bench seeds through entity substitution, memory-operation augmentation, and simplification. A teacher model, Qwen3-VL-235B-Thinking with full ConAct prompting, generates candidate trajectories; MemGUI-Eval vets them; and unreasonable steps such as loops and counterproductive actions are filtered out (Liu et al., 18 Jun 2026).
Training of MemGUI-8B-SFT uses standard next-token cross-entropy over the full five-part output. No auxiliary loss is introduced for memory triggering or folding depth. The paper’s point is that ConAct behavior can emerge from consistent demonstration across the 64k-step corpus. The public MemGUI-8B-SFT model uses Qwen3-VL-8B-Instruct as backbone, LoRA SFT with ms-swift, and sequence length 32,768 tokens. Reported hyperparameters are: bfloat16 precision, 1 epoch, AdamW fused, cosine learning-rate schedule, warmup ratio 0.05, learning rate 7, weight decay 0.1, max grad norm 1.0, per-device batch size 2, gradient accumulation 8, 8×80GB GPUs, effective batch size 128, LoRA rank 8, alpha 32, dropout 0.05, target all linear LM projection modules, and training time of approximately 535 minutes (Liu et al., 18 Jun 2026).
A notable empirical claim accompanies this setup: zero-shot ConAct formatting helps only at very large scale. The paper states that gains from ConAct without supervision emerge only for Qwen3-VL-235B-Thinking, while smaller backbones regress, which is why MemGUI-3K is positioned as the mechanism that makes proactive context management learnable at 8B scale (Liu et al., 18 Jun 2026).
4. Empirical performance and ablation findings
MemGUI-Agent is evaluated on MemGUI-Bench and MobileWorld GUI-Only. MemGUI-Bench reports Pass@1, Pass@3, Information Retention Rate (IRR), and Memory-Task Proficiency Ratio (MTPR), where IRR is the benchmark’s fidelity measure for how many required information units are correctly recalled or used (Liu et al., 3 Feb 2026).
| System | MemGUI-Bench | MobileWorld GUI-Only |
|---|---|---|
| Same 235B backbone with ReAct | 24.2 / 46.9 / 30.0 (Pass@1 / Pass@3 / IRR) | 14.5 SR |
| MemGUI-Agent-235B, zero-shot ConAct | 37.5 / 62.5 / 46.8 | 29.1 SR |
| Qwen3-VL-8B-Instruct backbone | 9.4 / 20.3 / 15.1 | 9.4 SR implied by +8.5 gap |
| MemGUI-8B-SFT | 23.4 / 35.9 / 30.2 | 17.9 SR |
The headline result is that MemGUI-Agent-235B achieves 37.5% Pass@1, 62.5% Pass@3, and 46.8% IRR on MemGUI-Bench, compared to 24.2, 46.9, and 30.0 for the same backbone under ReAct prompting. On MobileWorld, the same comparison is 29.1% versus 14.5% success rate. The MemGUI-8B-SFT model is reported as the best open-data 8B result on MemGUI-Bench, with 23.4% Pass@1, 35.9% Pass@3, and 30.2% IRR, and it reaches 17.9% success rate on MobileWorld, described as second-best among open-data 8B systems and +8.5 over the backbone (Liu et al., 18 Jun 2026).
Component ablations isolate the contribution of the three ConAct ingredients. On MemGUI-Bench-40 with the 235B zero-shot setting, adding UI memory actions improves Pass@1 from 5.0 to 17.5, adding history folding reaches 22.5 but leaves limited Pass@3 because compression alone can drop facts, adding self-describing steps yields 25.0, and full ConAct reaches 40.0 Pass@1, 62.5 Pass@3, and 51.0 IRR. The paper interprets this as evidence that folding, durable memory, and grounded step self-description are complementary rather than interchangeable (Liu et al., 18 Jun 2026).
Offline step-level evaluation on the MemGUI-3K test split shows that the 8B SFT model improves UI action match from 29.2% to 36.3%, memory trigger F1 from 19.9% to 48.0%, deep folding ratio from 8.8% to 26.1%, deep-range accuracy to 58.9%, and format compliance to 99.9%. The largest gain is on memory trigger F1, which is consistent with the paper’s thesis that deciding when to write or update memory is a learnable skill distinct from perception or grounding alone (Liu et al., 18 Jun 2026).
The error analysis is similarly specific. Full ConAct reduces total failures by 41%, from 99 to 58 attempts, primarily through a 42% reduction in process hallucination and a 57% reduction in output hallucination. The remaining stable error classes are knowledge deficiency, intent misunderstanding, and environment robustness issues (Liu et al., 18 Jun 2026).
5. Position within memory-centric GUI-agent research
MemGUI-Agent belongs to a larger shift in GUI-agent research from raw history replay toward explicit memory structures, but its mechanism is distinct from several neighboring formulations.
SecAgent compresses longer-term dependencies into a single semantic context 8 and conditions on only one raw historical step, formalized as 9; its contribution is history summarization efficiency rather than context actions inside the policy (Xie et al., 9 Mar 2026). MGA, or Memory-Driven GUI Agent, formalizes each desktop interaction step as an independent state 0, where 1 is task-agnostic spatial-semantic structure and 2 is structured memory; this is observation-first and memory-driven, but memory is maintained by specialized modules rather than emitted as first-class action tokens (Cheng et al., 28 Oct 2025). MementoGUI reframes long-horizon control as learned multimodal memory control over working memory and episodic memory, with ROI-level visual evidence and on-demand retrieval from a frozen GUI backbone; its controller is a plug-in layer, not the same end-to-end policy that acts in the environment (Zeng et al., 18 May 2026). UI-Copilot, by contrast, decouples transient execution context 3 from persistent observations 4 and trains the policy to invoke Retriever and Calculator tools selectively, so memory retrieval is tool-mediated rather than internalized as context-management actions (Lu et al., 15 Apr 2026).
A separate strand emphasizes human-in-the-loop and slot-structured memory. ReInAgent organizes memory as clarified instruction, slots, subtask list, action trajectory, reflections, and summaries, and introduces a HELP meta-action for ambiguity, conflict, and sensitive operations; this is a context-aware mobile agent, but its emphasis is user collaboration and conflict resolution rather than token-budgeted folding and exact UI fact storage (Jia et al., 9 Oct 2025).
Benchmark studies explain why these memory designs matter. MemGUI-Bench shows that short-term memory is mandatory, long-term memory is beneficial but optional, and cross-app complexity is the main bottleneck; it also identifies five failure modes, including Partial Memory Hallucination and Process Memory Hallucination (Liu et al., 3 Feb 2026). MMBench-GUI, covering Windows, macOS, Linux, iOS, Android, and Web, argues that long-context memory, broad action space, and long-term reasoning play a critical role, but also stresses that precise grounding is a critical determinant of success and that efficiency remains underexplored, quantified by the Efficiency-Quality Area metric (Wang et al., 25 Jul 2025). In this landscape, MemGUI-Agent’s contribution is not to replace grounding or planning, but to redefine context maintenance as an explicit control problem inside the action policy.
6. Limitations, misconceptions, and future directions
A frequent simplification is to equate MemGUI-Agent with “a longer context window.” The reported results argue against that interpretation. The paper’s ablations show that proactive context management is not mere retention of additional tokens; it is a structured combination of span-level folding, verbatim memory storage, and self-described grounded steps. Conversely, the paper also rejects the inverse simplification that memory alone resolves long-horizon GUI control. Even with full ConAct, stable residual failures remain in knowledge deficiency, intent misunderstanding, and environment robustness (Liu et al., 18 Jun 2026).
Another misconception is that the method is scale-agnostic. The zero-shot results explicitly indicate otherwise: ConAct formatting alone helps only for Qwen3-VL-235B-Thinking, while smaller backbones degrade until supervised on MemGUI-3K. This makes the dataset a substantive part of the method, not merely an auxiliary resource (Liu et al., 18 Jun 2026).
The current scope is also narrow in platform terms. The paper evaluates Android mobile environments only; desktop, web, and iOS are not covered. This matters because related work suggests that different platforms stress different deficits: MMBench-GUI identifies cross-platform generalization gaps and substantial inefficiency even when tasks are eventually completed (Wang et al., 25 Jul 2025). The one-action-per-step design is another operational limitation, since memory writes and UI actions cannot be co-issued when both would be natural.
The authors’ stated future directions are RL fine-tuning with online success-weighted rewards for folding and memory timing, better importance modeling for fact selection, multimodal grounding to UI structure when UI trees are available, system and API integration such as clipboard or contacts, and scaling beyond 8B with instruction-tuned, memory-aware pretraining and multi-environment data (Liu et al., 18 Jun 2026). A plausible implication is that future MemGUI-Agent variants may converge toward hybrid architectures in which context-as-action, learned memory control, and tool or API augmentation coexist rather than compete.
In the broader conceptual vocabulary of GUI autonomy, memory and context tracking are increasingly treated as prerequisites for Level 3 and Level 4 behavior, although the GAL framework itself remains qualitative and does not define a formal autonomy metric (Feng et al., 12 Feb 2026). Within that broader trajectory, MemGUI-Agent is best understood as a concrete end-to-end attempt to make context maintenance an explicit, learnable, and evaluable component of GUI control rather than an accidental by-product of prompt accumulation.