Memory-Driven GUI Agent

• Memory-Driven GUI Agent is a system for automating cross-application GUI tasks using dynamic, self-evolving memory banks.
• The architecture integrates a Planner–Actor model with a Darwinian Memory System that prunes and mutates memory units for enhanced reliability.
• Empirical results show significant improvements in success rate, execution stability, and latency reduction compared to traditional memory paradigms.

A Memory-Driven GUI Agent (MGA) is a class of agentic system for graphical user interface (GUI) automation that leverages explicit, often self-evolving, memory systems in conjunction with LLM or multimodal LLM (MLLM) backbones. The central objective is to enable robust, long-horizon, cross-application task completion in dynamic and unpredictable GUI environments by augmenting standard reactive or model-centric planning architectures with offline or online memory banks. These memory systems are designed to address context window limitations, hallucinations from outdated or mismatched experiences, error propagation, execution inefficiency, and inadequate granularity between planning intent and low-level action. This entry presents the formal underpinnings, architectural patterns, algorithmic innovations, empirical advancements, and practical design guidance for MGAs, drawing on the Darwinian Memory System (DMS) and related research (Mi et al., 30 Jan 2026).

1. Historical Motivation and Core Challenges

Existing MLLM-based GUI agents are constrained by limited context windows, which prevent the reliable retention and exploitation of information needed for multi-step, cross-application workflows. Prior memory approaches (static episodic buffers, key–value stores indexed by goal, or Retrieval-Augmented Generation (RAG) style accumulators) suffer from several fundamental limitations:

• Static accumulation: Unlimited accumulation leads to "context pollution"—retrieval of outdated or toxic priors, inducing hallucinations or brittle execution.
• Granularity mismatch: Indexing by global task goal or saving full trajectories fails to bridge the semantic level of planning with the execution-level detail needed for robust recovery and compositional behavior.
• Staleness and rigidity: Memory not subject to continual updating or pruning becomes unreliable under interface drift and layout shifts.
• Error amplification: Replay of entire historical trajectories amplifies early mistakes at later steps, causing trajectory collapse.

A Memory-Driven GUI Agent targets these bottlenecks by constructing memory as a dynamic selection of reusable knowledge units, continuously filtered or evolved to maximize utility and minimize risk within the agent’s own planning and execution loop (Mi et al., 30 Jan 2026).

2. Architectural Patterns and Memory Bank Construction

The blueprint for an MGA consists of three tightly integrated components: (i) a Planner–Actor architecture, (ii) an evolutionary memory bank governed by survival pressure, and (iii) a set of continual maintenance operations. The canonical design from DMS is as follows (Mi et al., 30 Jan 2026):

Planner–Actor Backbone

• The Planner $P$ (generally an LLM or MLLM) decomposes the current global task $T$ and observation $o_t$ into natural-language sub-tasks $p_i$ (each typically 1–5 steps).
• The Actor $A$ executes these atomic actions $a_t$ per sub-task.

Darwinian Memory Bank

• Memory units $m = (p, \tau, s_\text{meta})$ are stored off-model, with $p = \langle$\emph{Precondition}, \emph{Goal}$\rangle$ as key, payload $\tau$ (trajectory), and survival metadata $T$0.
• Decomposition is driven by the Planner’s natural sub-goal boundaries; each completed sub-goal forms a single memory unit. Fragments of length $T$1 (trivial actions) are filtered out.
• The resulting memory pool forms an ecosystem: units compete, mutate, and are pruned by utility.

3. Utility-Driven Natural Selection Mechanisms

The DMS approach formalizes selection as follows (Mi et al., 30 Jan 2026):

• Each memory $T$2 maintains usage and reliability metadata:
  • $T$3: reuse count,
  • $T$4: time since last reuse,
  • $T$5: verification failure strikes.
• The composite survival value is:

$T$6

where $T$7 is a novelty bonus, $T$8 grows with use, and $T$9 parameterize decay and error penalty.

• Pruning (Homeostatic Elbow): Sort all memory units by $o_t$0, compute the second derivative to find the "elbow" $o_t$1, prune all units with rank $o_t$2.
• Risk-Aware Inhibition: Maintain a Beta-Binomial posterior over observed failures $o_t$3 and successes $o_t$4 for each plan key, compute the lower-confidence bound risk $o_t$5; suppress high-risk units ($o_t$6) from retrieval even if $o_t$7 is high.
• $o_t$8-Mutation & Replacement: With probability $o_t$9, ignore retrieved memory and generate a fresh trajectory. If the new trajectory is strictly shorter and successful, it overwrites the previous memory unit.

4. Retrieval, Composition, and Closed-Loop Integration

Retrieval is critical for compositional generalization and safety (Mi et al., 30 Jan 2026):

• Dual-Factor Retrieval: For candidate plan $p_i$0, compute:

$p_i$1

where $p_i$2 is a semantic encoder. Accept the top candidate if above a threshold and not risk-suppressed.

• Integration and Maintenance Routine: For each subtask:
  1. Planner proposes $p_i$3.
  2. DMS retrieves corresponding memory unit or triggers Actor to generate a new trajectory.
  3. Verifier checks execution outcome.
  4. DMS updates memory unit’s $p_i$4, $p_i$5, $p_i$6, and triggers pruning/mutation.
  5. All interactions utilize standard LLM prompts and tool calls; no retraining required.

• This design yields a self-evolving, lifelong-learning MGA with plug-and-play compatibility for any LLM-based Planner/Actor stack.

5. Empirical Evidence and Quantitative Performance

MGAs utilizing DMS have demonstrated significant gains on real-world multi-app GUI benchmarks (AndroidWorld, 116 tasks × 20 apps) (Mi et al., 30 Jan 2026):

Backbone	Baseline SR	DMS MGA SR	$p_i$7SR	SRR (Stability)	$p_i$8SRR	Latency Reduction
Qwen2.5-VL-72B	41.0%	66.4%	+25.4 pp	(not reported)		Up to 50%
Qwen3-VL-30B	25.0%	44.0%	+19.0 pp	36.5% $p_i$9 84.7%	+48.2 pp
GLM-4.5V	38.0%	50.4%	+12.4 pp	53.0% $A$0 87.0%	+34.0 pp
Seed1.6-VL	41.0%	56.0%	+15.0 pp

• Overall average gain: +18.0 pp in success rate; +33.9 pp in execution stability (success retention rate).
• Latency: Memory reuse increased sharply over five execution rounds, reducing effective task latency by up to 40–50%.
• Footprint: Pruning operations stabilized memory usage at $A$18 MB.

These results establish that training-free, utility-driven memory evolution endows MGAs with improved robustness, shorter latency, and higher long-horizon reliability compared to baseline episodic or rigid key–value systems.

6. Comparative Analysis with Prior Memory Paradigms

MGAs with DMS address several critical defects in prevailing memory paradigms (Mi et al., 30 Jan 2026):

Paradigm	Granularity Alignment	Pruning/Evolution	Context Pollution Avoidance	Robustness to Drift
Static episodic buffers	Poor	None	No	Low
Goal-only key–value stores	Poor	None	No	Low
RAG-style accumulators	N/A	None	No	Low
DMS MGA (Darwinian)	High (subgoal-level)	Yes	Yes	High (self-evolving)

• DMS-style memory units are aligned to the planner's effective sub-goal scope, avoiding negative transfer and over-fragmentation.
• The continuous “natural selection” logic prunes out low-value and risky units, unlike accumulators that cause context pollution.
• $A$2-mutation and in-place evolution prevent local minima and support system adaptation to interface drift.

7. Design Guidelines, Best Practices, and Constraints

MGA development with DMS or similar evolutionary memory systems is subject to specific design recommendations (Mi et al., 30 Jan 2026):

• Granularity alignment: Always align memory unit segmentation with the Planner’s natural sub-goal horizon.
• Expressive but minimal keys: Use $A$3\emph{Precondition}, \emph{Goal}$A$4 pairs for keying.
• Survival value tracking: Implement $A$5 update and ranking, combining use, recency, and reliability signals.
• Maintenance scheduling: Periodically prune using the homeostatic elbow, and suppress risk-prone units.
• Exploration rate: Tune $A$6-mutation for efficient exploration/exploitation.
• Verification depth: Maintain failure strike counts ($A$7) to balance reliability and update latency.
• Footprint regulation: Monitor and adjust $A$8 as needed to control memory bank size.
• Constraint tuning: Over-pruning risks eliminating rare but essential strategies; $A$9 rate affects convergence versus exploration.

Key parameters (as recommended in (Mi et al., 30 Jan 2026)): $a_t$0, $a_t$1, $a_t$2, $a_t$3, $a_t$4, $a_t$5.

---

In sum, Memory-Driven GUI Agents—exemplified by the Darwinian Memory System—embody a paradigm of self-evolving, utility-driven memory, tightly integrated with a semantic Planner–Actor loop. Empirically and theoretically, they deliver state-of-the-art generalization, robustness, and efficiency on long-horizon and cross-application GUI tasks, circumventing pitfalls of static episodic memory systems by introducing continuous pruning, survival-driven adaptation, and compositional memory recombination (Mi et al., 30 Jan 2026).