MemGUI-Agent: An End-to-End Long-Horizon Mobile GUI Agent with Proactive Context Management

Abstract: MLLM-based mobile GUI agents have made substantial progress on short-horizon tasks, yet remain unreliable on long-horizon tasks that require retaining intermediate facts across many steps and app transitions. We attribute this limitation to ReAct-style prompting, which passively accumulates per-step records, leading to prompt explosion and dilution of critical cross-app facts. To address this, we introduce MemGUI-Agent, an end-to-end long-horizon mobile GUI agent with proactive context management. MemGUI-Agent is built on Context-as-Action (ConAct), which casts context management as first-class actions emitted by the same policy that selects UI actions. Instead of passively appending history, ConAct maintains three structured context fields: folded action history, folded UI state, and recent step record, preserving critical UI facts while keeping context compact. To make proactive context management learnable across model scales, we construct MemGUI-3K, a 2,956-trajectory dataset with full ConAct annotations for supervised training and offline analysis. Training an 8B model on MemGUI-3K produces MemGUI-8B-SFT, an 8B MemGUI-Agent that achieves the best open-data 8B performance on MemGUI-Bench and generalizes to the out-of-distribution MobileWorld benchmark. Code, data, and trained models will be released at https://memgui-agent.github.io/.

• The paper introduces the innovative context-as-action protocol that treats history folding, explicit memory operations, and self-description as first-class actions, ensuring robust retention of pivotal UI facts over long-horizon tasks.
• It details the use of the MemGUI-3K dataset, with 2,956 multi-step trajectories across diverse Android apps, to supervise and fine-tune the model for improved UI action matching and memory trigger performance.
• Empirical results demonstrate significant gains, reducing failure modes by up to 41% and achieving up to 62.5% Pass@3 on benchmark tests, underscoring the approach's effectiveness in long-horizon mobile automation.

MemGUI-Agent: An End-to-End Long-Horizon Mobile GUI Agent with Proactive Context Management

Motivation and Framework Overview

Mobile GUI agents leveraging large multimodal LLMs (MLLMs) have made substantial advances in automating tasks on mobile devices. However, existing solutions underperform in long-horizon scenarios, particularly where the retention of intermediate facts, progress tracking over many steps, and cross-app information transfer are critical. The paper identifies the primary bottleneck as prompt explosion and information loss due to passive context accumulation mechanisms commonly used in end-to-end models (e.g., Action-Thought, Multi-turn, Rule-based approaches). As task horizons increase, these strategies lead to either prohibitively large prompts or frequent loss of pivotal UI-derived facts, undermining task completion rates.

MemGUI-Agent introduces context-as-action (ConAct), a unified protocol where context manipulation—including history folding, explicit memory operations, and self-describing step outputs—is treated as first-class actions, predicted jointly with UI actions by the agent policy. This paradigm enables compact yet persistent working contexts and robust retention of critical task information, directly addressing the major failure modes observed in prior work.

(Figure 1)

Figure 1: MemGUI-Agent employs ConAct to flatten prompt growth across long-horizon tasks, yielding higher benchmark performance while retaining critical context.

ConAct Protocol and Policy Integration

ConAct is formalized as a structured decision protocol with three central fields: Folded Action History ($H_t$), Folded UI State ($M_t$), and Recent Step Record ($L_t$). At each step, the MLLM policy emits a tuple covering reasoning ($\tau$), folding directive ($\phi$), UI/memory action ($a$), grounded observation ($o$), and action intent ($\iota$). History folding compresses long trajectories through span-level summarization, memory actions store exact task-relevant content (e.g., prices, codes, copied text), and self-description ensures the context is reusable and unambiguous.

ConAct's design shifts critical context-maintenance decisions from mechanical prompt manipulation outside the model to explicit, policy-level choices, leveraging the agent’s task-level reasoning for context compression, fact retention, and recent-step description. The three context fields are updated through model actions, and subsequent environment/tool responses are integrated alongside these updates, ensuring context transitions are tightly controlled and information persists across steps, screens, and app transitions.

MemGUI-3K Dataset Construction and Supervised Training

Recognizing that the ConAct protocol cannot yield consistent gains by mere format change (zero-shot improvements are backbone-dependent), MemGUI-3K is curated to enable model-scale learning of proactive context management. This dataset comprises 2,956 trajectories (avg. 28.8 steps; median 25) over 26 Android apps and 7 functional categories, derived from task expansion (entity substitution, memory-operation augmentation, task simplification). Each trajectory is annotated with step-level reasonableness and impact, yielding 64,430 supervised samples amenable for SFT-style fine-tuning.

Figure 2: MemGUI-3K collection pipeline: expanded tasks are rolled out, evaluated, and filtered for step-level reasonableness, generating high-quality SFT samples.

MemGUI-3K supervises not only UI actions but also memory operations, history compression, and step-level self-description, supporting granular offline skill assessment and facilitating practical deployment of context-driven agents.

(Figure 3)

Figure 3: Representative MemGUI-3K trajectory: step-level annotations enable separation of reasonable and counterproductive behaviors for supervised learning.

Empirical Results: Benchmark, Ablation, and Transfer Performance

Zero-shot MemGUI-Agent-235B, using ConAct with Qwen3-VL-235B-Thinking, achieves Pass@3 of 62.5% on MemGUI-Bench—substantially outperforming agentic frameworks reliant on proprietary backbones (e.g., Gemini-2.5-Pro) and improving success and information retention rates by 15.6 and 16.8 points over the same backbone. On MobileWorld GUI-Only, MemGUI-Agent-235B secures 29.1% Pass@1, demonstrating strong generalization to disjoint environments and app sets.

Fine-tuning Qwen3-VL-8B-Instruct on MemGUI-3K produces MemGUI-8B-SFT, which delivers the best open-data 8B performance on MemGUI-Bench and transfers to MobileWorld (17.9% Pass@1, +8.5 over backbone). Gains are concentrated on medium/hard tasks, indicating effectiveness against long-horizon context drift.

Figure 4: Successful zero-shot MemGUI-Agent-235B trajectories on MemGUI-Bench and MobileWorld, illustrating robust cross-app context management and fact preservation.

Ablation studies confirm that all three ConAct mechanisms are necessary; memory actions triple Pass@1 but do not solve context drift alone; history folding is insufficient without persistent memory; self-describing steps enhance context reuse. Full ConAct achieves 40.0% Pass@1 and 62.5% Pass@3 on MemGUI-Bench-40, and reduces total failures by 41%, predominantly due to reductions in process and output hallucinations.

Figure 5: ConAct reduces process and output hallucination failure categories by substantial margins, supporting its primary gains in context-induced error mitigation.

Quantitative and Qualitative Skill Assessment

Step-level offline evaluation against gold SFT context on MemGUI-3K test split reveals that MemGUI-8B-SFT improves UI action matching (36.3% vs. 29.2% baseline), memory trigger F1 (48.0% vs. 19.9%), and deep folding ratio (26.1% vs. 8.8%). Format compliance exceeds 99.9%. These results show that MemGUI-3K supervision imparts actionable context-management decisions, not merely formatting skill, promoting critical transient observations to durable memory.

Implications and Future Directions

MemGUI-Agent demonstrates an effective integration of proactive context management into end-to-end multimodal mobile GUI agents, enabling robust long-horizon task execution and cross-app fact preservation. The context-as-action protocol is directly responsible for mitigating prompt explosion and information loss seen in previous work. By unifying history folding, explicit memory operations, and grounded self-description within the agent policy, MemGUI-Agent sets a rigorous new baseline for long-horizon mobile GUI automation.

The supervised dataset MemGUI-3K provides a scalable foundation for training future context-intensive agents, and the method transfers to unseen environments and app sets, as shown by performance on MobileWorld. The framework’s extensibility to iOS, desktop, and web interfaces remains an open problem; further scaling and generalization of context-management policies will be critical in advancing agent reliability.

Conclusion

MemGUI-Agent reframes context management as a core action modality inside end-to-end mobile GUI control. The unified ConAct protocol, paired with the high-quality MemGUI-3K dataset, enables agents to maintain compact, persistent, and actionable working contexts. Empirical and skill-based evaluations substantiate significant gains in success and information retention, with reductions in context-induced failure modes. The approach provides a rigorous foundation for long-horizon multimodal agent research, and its implications extend to practical deployment and theoretical advances in agent-based context curation.