---
title: 'MemGUI-3K: Mobile GUI Context Management Dataset'
url: https://www.emergentmind.com/topics/memgui-3k
type: topic
---

# MemGUI-3K: Mobile GUI Context Management Dataset

MemGUI-3K is a supervised dataset of mobile GUI agent trajectories built specifically for Context-as-Action (ConAct), the proactive context-management mechanism introduced in "MemGUI-Agent: An End-to-End Long-Horizon Mobile GUI Agent with Proactive Context Management" [2606.19926]. It consists of **2,956 trajectories** with **full ConAct annotations**, and is designed to support supervised training and offline analysis of long-horizon mobile GUI interaction in which an agent must retain intermediate facts across many steps and app transitions. The dataset is collected in the MemGUI-Bench Android snapshot environment and is closely tied to the memory-centric evaluation setting introduced by "MemGUI-Bench: Benchmarking Memory of Mobile GUI Agents in Dynamic Environments" [2602.06075].

## 1. Definition and scope

MemGUI-3K is defined as **“a 2,956-trajectory dataset with full ConAct annotations, supporting supervised training and offline analysis of proactive context management across model scales”** [2606.19926]. Its immediate purpose is to make ConAct trainable, especially at the 8B scale. The dataset is not merely a larger corpus of demonstrations: each trajectory contains explicit supervision for standard UI actions, context-management actions, self-describing step outputs, and the resulting structured state transitions of the context fields.

The dataset addresses a gap identified in prior mobile GUI work. Existing mobile GUI datasets and benchmarks primarily supervise UI actions and often rely on ReAct-style histories or unstructured multi-turn dialogues, but they do **not** provide explicit supervision for context-management decisions such as when to summarize history, what to persist in memory, or how to maintain a compact but sufficient context [2606.19926]. In the broader benchmark context, MemGUI-Bench was introduced because existing mobile GUI agent benchmarks contain only **5.2–11.8%** memory-related tasks, lack **pass@k** protocols for cross-session learning, and rely on evaluation methods that do not scale well to long, cross-app trajectories [2602.06075]. MemGUI-3K is therefore best understood as the training substrate for the memory-centric regime that MemGUI-Bench exposed.

A common misconception is to treat MemGUI-3K as simply a larger benchmark. The primary paper instead presents it as a **supervised dataset** for ConAct. MemGUI-Bench remains the evaluation benchmark, whereas MemGUI-3K provides annotated trajectories from which a model can learn proactive context management [2606.19926].

## 2. ConAct formalism and annotation schema

MemGUI-3K is organized around the ConAct state and action formulation. The structured context state at step \(t\) is

\[
\mathcal{S}_t = (G, H_t, M_t, L_t),
\]

and the policy output is

\[
y_t = (\tau_t,\phi_t,a_t,o_t,\iota_t) \sim \pi_\theta(\cdot \mid I_t,\mathcal{S}_t).
\]

Here, \(G\) is the task goal, \(H_t\) is folded action history, \(M_t\) is folded UI state, \(L_t\) is the recent step record, and \(I_t\) is the screenshot [2606.19926].

For each step, MemGUI-3K includes the following inputs: the task goal \(G\), folded action history \(H_t\), folded UI state \(M_t\), recent step record \(L_t\), and current screenshot \(I_t\). It also includes the gold assistant output in a five-part ConAct structure: `<thinking>` for \(\tau_t\), `<folding>` for \(\phi_t\), `<tool_call>` for \(a_t\), `<ui_observation>` for \(o_t\), and `<action_intent>` for \(\iota_t\) [2606.19926].

Three context fields are central.

First, **Folded Action History** \(H_t\) is a list of folded records generated by prior folding directives. A folding directive is written as

\[
\phi_t = ([s_t, t], z_t),
\]

where \([s_t,t]\) specifies the interval to compress and \(z_t\) is a natural-language summary. The update rule is

\[
H_{t+1} = \mathrm{Fold}(H_t, \phi_t). \tag{3}
\]

This supports both **step-level distillation** (\(s_t=t\)) and **span-level abstraction** (\(s_t<t\)) [2606.19926].

Second, **Folded UI State** \(M_t\) stores persistent UI-derived facts. Each memory item is

\[
m = (\mathrm{id}, d, c),
\]

with a unique identifier, a short description, and full content. Memory actions are defined by

\[
\mathcal{A} = \mathcal{A}_{ui} \cup \mathcal{A}_{mem},\quad
\mathcal{A}_{mem} = \{a^{+},a^{\circ},a^{-}\} = \{\text{memory\_add},\text{memory\_update},\text{memory\_delete}\}. \tag{2}
\]

The update rule is

\[
M_{t+1}=
\begin{cases}
\mathrm{Add}(M_t,m), & a_t=a^{+},\\
\mathrm{Update}(M_t,\mathrm{id},c), & a_t=a^{\circ},\\
\mathrm{Delete}(M_t,\mathrm{id}), & a_t=a^{-},\\
M_t, & a_t\in\mathcal{A}_{ui}.
\end{cases} \tag{5}
\]

MemGUI-3K explicitly logs which UI facts are preserved, updated, or deleted [2606.19926].

Third, **Recent Step Record** \(L_t\) contains a grounded UI observation and an action intent. The observation includes “exact visible text, numbers, names, and task-relevant UI facts,” while the action intent summarizes what the step is meant to accomplish. Together with the executed action and tool result, the update is

\[
L_{t+1} = (o_t,\iota_t,a_t,r_t). \tag{6}
\]

The complete next context state is then

\[
\mathcal{S}_{t+1} = (G, H_{t+1}, M_{t+1}, L_{t+1}). \tag{8}
\]

This annotation design makes context management part of the same policy that selects UI actions, rather than an external or post-hoc mechanism [2606.19926].

## 3. Construction pipeline and data curation

MemGUI-3K is collected in the **snapshot-based Android environment of MemGUI-Bench**, with **26 apps** and the same device abstraction and standard touch actions used in that framework [2606.19926]. The teacher model is **Qwen3-VL-235B-Thinking** running the full ConAct protocol. Each selected task is executed once, with a step budget of

\[
B_{\text{MemGUI}}(g) = 2.5g + 1,
\]

where \(g\) is the golden-step count from MemGUI-Bench [2606.19926].

Task generation begins from the **128 seed tasks** of MemGUI-Bench. The authors apply three transformations. **Entity substitution** preserves task pipelines while replacing entities such as product names or search queries, producing **456 expanded tasks**. **Memory-operation augmentation** explicitly constructs tasks that require `memory_update` and `memory_delete` in addition to `memory_add`, producing **289 tasks**. **Task simplification** decomposes complex multi-app tasks into shorter, single-objective variants while preserving the need for context management, producing **6,558 simplified tasks** [2606.19926]. These processes yield **7,303 candidate tasks**, of which **5,293 are selected for rollout**.

The final dataset contains **2,566 simplified tasks**, **234 entity substitution tasks**, and **156 memory-operation tasks** [2606.19926]. Trajectory-level filtering then retains only rollouts judged correct by the MemGUI-Eval pipeline: **2,959 of 5,293** pass this stage. A subsequent sanity filter removes **one abnormal 321-step outlier** and **two tasks with extremely low-frequency apps**, leaving the final **2,956 trajectories** [2606.19926].

The dataset also applies step-level filtering. Even successful trajectories may contain exploratory or counterproductive steps, so each step is labeled with `reasonableness`, `impact`, and an explanation. Only steps labeled **reasonable** are used as supervised fine-tuning samples. This produces **64,430 reasonable-step samples**, or about **21.8 reasonable steps per trajectory**, corresponding to a **75.7%** ratio [2606.19926]. This suggests that MemGUI-3K is curated not only for success at the trajectory level but also for local action quality and context-management quality.

## 4. Scale, composition, and long-horizon properties

MemGUI-3K contains **2,956 successful rollouts**, split into **2,661 train trajectories** and **295 test trajectories**. At the step level it contains **64,430** reasonable-step samples, divided into **57,951 train** and **6,479 test** [2606.19926]. Every trajectory is a full interaction from initial state to termination in the MemGUI-Bench Android snapshot environment.

The average trajectory length is **28.8 steps** with **median 25**, which the paper reports as about **1.9× GUIOdyssey (15.3 steps)** [2606.19926]. Task difficulty inherits the MemGUI-Bench categorization: **Easy (1–20 golden steps), Medium (21–40), Hard (41+)**. The source benchmark is predominantly memory-intensive: **89.8%** of MemGUI-Bench tasks are explicitly memory-intensive and **78.1%** require cross-application information transfer [2602.06075]. MemGUI-3K is built from expansions of those tasks while remaining disjoint from the original **128 evaluation tasks** [2606.19926].

The dataset spans **26 Android apps** across **7 app categories** and includes both single-app and cross-app workflows, with many transitions among applications such as Amazon, Joplin, Bing, Calculator, Calendar, and Messages [2606.19926]. In the underlying benchmark, task structures include flows such as Amazon → Joplin → Calculator, Apartments.com → Bing → Citymapper → Joplin, and AP News → DeepL → Joplin, all designed to stress cross-temporal and cross-spatial retention [2602.06075].

The ConAct annotations reveal nontrivial context-management behavior. **23.8% of folds are span-level**, with **average span length 6.25 steps**, **median 4**, and a maximum of **118**; **88.7% of trajectories contain at least one span-level fold** [2606.19926]. Memory actions appear in **65.1% of trajectories**, with averages of **1.17 `memory_add`**, **0.03 `memory_update`**, and **0.04 `memory_delete`** per trajectory [2606.19926]. These statistics indicate that MemGUI-3K is not limited to one-step memoization or trivial logging; it encodes teacher behavior for deciding when to abstract history and when to persist task-relevant UI facts.

## 5. Training role and empirical significance

MemGUI-3K is used to train **MemGUI-8B-SFT**, an 8B MemGUI-Agent obtained by supervised fine-tuning on the dataset’s reasonable steps [2606.19926]. The base model is **Qwen3-VL-8B-Instruct**, trained with LoRA on **57,951** training steps for **1 epoch** [2606.19926]. The supervision target is the full assistant response, so the model is jointly trained to predict the next UI action, the folding directive, memory operations, UI observation, and action intent.

The paper emphasizes that proactive context management is **not merely a prompt-format change**. In zero-shot experiments, simply switching to ConAct helps only the strongest model, **Qwen3-VL-235B-Thinking**, and can hurt smaller backbones [2606.19926]. MemGUI-3K is therefore presented as the mechanism that makes ConAct learnable below that scale.

On MemGUI-Bench, the improvement attributed to **MemGUI-8B-SFT (ConAct + MemGUI-3K)** over the **Qwen3-VL-8B-Instruct** ReAct-style baseline is substantial. Pass@1 rises from **9.4%** to **23.4%**, pass@3 from **20.3%** to **35.9%**, and IRR from **15.1%** to **30.2%** [2606.19926]. On **Hard tasks (41+ golden steps)**, pass@1 increases from **2.6%** to **21.1%**, pass@3 from **7.9%** to **34.2%**, and IRR from **9.5%** to **32.6%** [2606.19926]. Since MemGUI-Bench was specifically designed to expose failures in memory-intensive, long-horizon, and cross-app tasks, these gains are directly relevant to the long-context problem formulation [2602.06075].

The same model also transfers to **MobileWorld GUI-Only (117 tasks)**, where success rate rises from **9.4%** for the baseline to **17.9%** for MemGUI-8B-SFT, exceeding **OpenMobile-8B (17.7%)** and **ClawGUI-2B (17.1%)** in that reported comparison [2606.19926]. This indicates that the learned behavior is not restricted to the MemGUI-Bench environment.

Offline analysis further shows that MemGUI-3K teaches context-control skills rather than only action syntax. With gold context, UI action match accuracy improves from **29.2%** to **36.3%**; memory timing trigger F1 improves from **19.9%** to **48.0%**; deep fold ratio rises from **8.8%** to **26.1%**; deep range accuracy improves from **45.2%** to **58.9%**; and full format compliance increases from **94.9%** to **99.9%** [2606.19926]. The largest improvement is specifically described as occurring in context-control decisions, especially “promoting transient observations to durable memory.”

## 6. Relation to MemGUI-Bench, distinctiveness, and limitations

MemGUI-3K is structurally downstream of MemGUI-Bench. The benchmark paper defines memory for mobile GUI agents as the ability to retain, process, and utilize contextual information within tasks and experiential knowledge across tasks to enhance decision-making and task performance over time [2602.06075]. It also introduces the core evaluation measures—SR, IRR, MTPR, pass@k SR, FRR, Average Step Ratio, Average Time per Step, and Average Cost per Step—and documents substantial memory deficits across evaluated agents [2602.06075]. MemGUI-3K does not replace these benchmark functions; it complements them by supplying annotated trajectories that operationalize proactive context management.

Relative to prior datasets, MemGUI-3K differs in two main respects. First, it is reported as **the longest open SFT dataset for mobile GUI agents currently reported**, with average length **28.8 steps** [2606.19926]. Second, it provides end-to-end supervision for **history folding ranges and summaries**, **structured UI memory actions inside the same policy**, and **explicit self-describing step outputs**, which the paper states are absent from other datasets and agent frameworks such as ReAct-, Action-Thought-, rule-based-, or external-memory-agent designs [2606.19926].

Several limitations are explicit. MemGUI-3K is restricted to **Android-style mobile GUI environments** and **26 apps**, with future work needed for **iOS, desktop, and web interfaces** [2606.19926]. Although the dataset extends horizon length, the paper notes that harder regimes such as more app transitions or multi-session tasks are not yet captured. The annotations are generated by a **235B model + MemGUI-Eval**, not by human experts, so residual suboptimal context actions or teacher bias may remain [2606.19926]. Memory editing is also sparse: `memory_update` and `memory_delete` average only **0.03** and **0.04** per trajectory, which may limit generalization to settings requiring aggressive memory revision [2606.19926].

A further point of clarification concerns naming. MemGUI-Bench contains a section that extrapolates to a hypothetical “MemGUI-3K” as a scaled-up successor benchmark, but explicitly states that this is **reasoned extrapolation** and that the paper does not mention “MemGUI-3K” as an actual benchmark artifact [2602.06075]. The concrete MemGUI-3K that exists in the literature is instead the ConAct-centric dataset introduced in the MemGUI-Agent work [2606.19926]. This suggests that the term now denotes a training dataset rather than a benchmark, even though the benchmark paper had earlier used it hypothetically as a name for a future large-scale evaluation suite.

Source: https://www.emergentmind.com/topics/memgui-3k