- The paper introduces an Executable Agentic Memory (EAM) that transforms GUI planning by leveraging a knowledge graph for reliable, efficient, and scalable automation.
- It employs a DFS-based offline memory construction and MCTS-guided online path extraction, significantly reducing latency and token costs.
- Empirical results demonstrate that EAM outperforms state-of-the-art agents with higher success rates and provable theoretical guarantees in structured planning.
Summary of "Executable Agentic Memory for GUI Agent" (2605.12294)
Motivation and Problem Setting
Graphical User Interface (GUI) automation agents powered by LLMs have transformative potential for real-world applications, ranging from mobile task automation to accessibility enhancements. However, existing step-wise, model-centric interaction paradigms are not robust when faced with long-horizon tasks: context limitations, perceptual drift, and compounding reasoning errors lead to unreliable action sequences, frequent hallucinations, and inefficient resource usage. Current approaches based on in-context memory injection or free-form retrieval are unable to guarantee reliable reproduction of executable paths from historical knowledge. Furthermore, repeated LLM inference incurs significant latency and token cost, particularly in heterogeneous environments and complex user tasks.
Proposed Solution: Executable Agentic Memory (EAM)
The paper introduces Executable Agentic Memory (EAM)—a structured, environment-induced Knowledge Graph (KG) that explicitly captures GUI states, available actions, and deterministic state transitions. EAM transforms GUI planning from unconstrained action generation to a retrieval-and-execution paradigm grounded by the KG topology, thereby enabling reliable, efficient, and scalable agentic reasoning.
Offline Memory Construction
- Task-oriented, state-aware Depth-First Search (DFS) rapidly explores the environment, systematically covering task-relevant transitions while minimizing redundant interactions.
- State deduplication and action-group mining—leveraging statistical mechanisms inspired by Byte Pair Encoding (BPE)—compress frequent multi-step routines into high-level action nodes, reducing search complexity.
- The resulting KG encodes the UI as a state machine: alternates between state and action nodes, annotating them with semantic descriptions and merging trajectories via coarse/fine filtering.
Online Path Extraction and Planning
- Task execution is modeled as a finite-horizon episodic MDP over the KG, using deterministic transitions and binary terminal rewards.
- Monte Carlo Tree Search (MCTS), guided by a lightweight Q-model trained via iterative self-training and preference ranking, efficiently traverses the state-action space for optimal path extraction.
- The Q-model estimates success probabilities under a random policy, ensuring bias-consistency guarantees and fine-grained credit assignment.
- The agent requires only a single LLM cloud call for final path grounding, contrasting with the expensive step-wise inference in prior frameworks.
Theoretical Guarantees
- The paper rigorously derives bias consistency bounds for the learned Q-model on the critical set, showing that action rankings required for optimal path recovery are preserved as sample size increases.
- Sample complexity bounds are established: the number of MCTS simulations required to recover the optimal execution path scales polynomially with horizon, branching factor, and inversely with the effective action gap in Q-values.
Empirical Evaluation
Benchmarks include AndroidWorld, MobileMiniWob++, and DroidTask, covering complex tasks and diverse app environments. EAM is compared against both on-device and cloud-based baselines, including model-centric agents (UI-TARS-7B, AutoDroid-V2), cloud VLMs (GPT-4o, M3A), and knowledge-augmented baselines (AppAgentX, GUI-Explorer).
- Success Rate: EAM achieves up to 52.6% on AndroidWorld, 76.1% on MobileMiniWob++, and 86.1% on DroidTask—significantly outperforming UI-TARS-7B (+19.6% on AndroidWorld) and surpassing GPT-4o-based M3A (+7.6% AndroidWorld), while achieving parity with knowledge-enhanced agents.
- Efficiency: Average latency is 2.8s per task step, and token cost is reduced by approximately 6x compared to GPT-4o. The plan-then-execute paradigm, grounded in KG, ensures minimal API usage and high-speed path execution.
- Self-Training Dynamics: Iterative self-training consistently improves Q-model accuracy and path extraction margin, with larger models (3B) achieving best results. Action group mining and in-environment Q-model training are both critical for optimal performance and cross-environment generalization.
Component Analysis & Ablation
Component ablations demonstrate:
- Knowledge graph guidance yields substantial gains versus baseline model-centric execution.
- Action groups extracted via statistical mining, rather than LLM summarization, appreciably improve both success rate and latency.
- Path extraction via MCTS consistently outperforms greedy or best-of-N approaches, aligning with sample complexity theory.
- Fine-grained Q-value prediction (via soft targets) is markedly superior to binary reward training for credit assignment.
Implications and Future Directions
Practical Impact
EAM addresses key limitations of current GUI automation agents by fundamentally decoupling semantic generation from executable environment logic. Treating the KG as a state machine enables:
- Robust long-horizon planning with provable guarantees on execution correctness.
- On-device deployment with small models, bridging the gap to frontier cloud LLMs.
- Efficient real-time automation of multi-step tasks, potentially improving productivity and accessibility.
Theoretical Implications
The approach formalizes the agentic GUI automation task as structured planning over deterministic state machines, facilitating connections to classical tree search theory, tabular RL, and sample-efficient value estimation. The bias-consistency and sample complexity results illuminate pathways for scalable, reliable agentic reasoning on constrained domains.
Limitations and Speculative Future Work
The current EAM framework presumes relatively static UI environments and may require incremental updating or evolution mechanisms for knowledge graphs in highly dynamic scenarios. Further research into adaptive KG evolution, continual learning, and privacy-preserving agent memory construction is warranted. Additionally, integrating richer semantic reasoning, stochastic environments, and hybrid policy/value architectures could further enhance agent robustness and generalization.
Conclusion
The Executable Agentic Memory paradigm represents a significant formal advance in GUI automation methodology, combining sample-efficient structured memory with value-guided planning to achieve reliable, efficient, and scalable agentic reasoning. Empirical and theoretical results jointly demonstrate the feasibility of memory-grounded automation, defining a robust foundation for future AI agentic systems targeting long-horizon interaction in real-world digital environments.