---
title: 'MAPLE: Mobile GUI Task Reasoning Framework'
url: https://www.emergentmind.com/topics/maple-mobile-gui-task-reasoning
type: topic
---

# MAPLE: Mobile GUI Task Reasoning Framework

MAPLE (Mobile Agent with Persistent Finite State Machines for Structured Task Reasoning) is a state-aware, multi-agent framework that enables autonomous completion of user-instructed tasks across mobile GUI environments. MAPLE abstracts app interactions as a dynamically constructed finite state machine (FSM), computationally modeling each UI screen as a distinct state and user actions as transitions. This facilitates structured representation of app navigation, robust error detection and recovery, and knowledge retention, supporting complex, cross-application mobile task automation [2505.23596].

## 1. System Design and Modular Architecture

MAPLE operates as a modular, multi-agent system layered over a physical or emulated Android device, controlled via the Android Debug Bridge (ADB). The core architecture is partitioned into four interdependent phases: Planning, Execution, Verification & Error Recovery, and Knowledge Retention. Each phase is managed by specialized agents that communicate through prompts to a Multimodal Large Language Model (MLLM). The Actor Agent executes low-level, atomic GUI operations such as Tap, Type, and Swipe via the mobile API.

**Phases and Responsible Agents:**
- **Planning:** The Planner Agent receives a user instruction $u$ (and optionally reusable knowledge $K$), producing a multi-step plan $\pi = (g_1, r_1), ..., (g_k, r_k)$, where each $g_i$ is a subgoal and $r_i$ its rationale. A two-stage workflow generates $n$ candidate plans, which the MLLM scores and selects.
- **Execution:** Three agents collaborate: the Screen Parser captures screenshots and performs OCR (DBNet + ConvNextViT-document), icon grounding (GroundingDINO), and icon captioning (Qwen-VL-Plus) to emit perception data $p_i$; the State Agent maintains and builds the FSM, prompting the MLLM for state summaries, predicted next screens, pre-conditions ($\mathrm{pre}^{\,i+1}$), and post-conditions ($\mathrm{post}^i$); the Actor Agent selects and executes UI actions $a_i$.
- **Verification & Error Recovery:** The Reflection Agent compares actual vs. predicted states post-action, classifies outcomes, and triggers rollback or replanning logic upon failure.
- **Knowledge Retention:** The Mentor Agent analyzes action histories and FSMs post-task, distilling reusable guidance and sequences into the persistent knowledge base $K$.

Orchestration is handled by a lightweight controller that alternates among high-level planning, execution, state-tracking, verification, and recovery.

## 2. Finite State Machine Formalism

MAPLE's FSM provides a structured, annotated representation of mobile app navigation in real time:

\[
\mathcal{M} = (S,\,A,\,T,\,s_0,\,G)
\]
- $S$: Set of discovered UI states, $s_i \in S$ each corresponding to a screen.
- $A$: Finite set of GUI actions (e.g., tap, type, swipe).
- $T \subseteq S \times A \times S$: Transition relation; $T(s_i, a_i) = (s_{i+1}, \mathrm{pre}^{\,i+1}, \mathrm{post}^i)$, denoting that action $a_i$ in $s_i$ leads to $s_{i+1}$, annotated with pre/post-conditions.
- $s_0 \in S$: Initial state, such as the home screen.
- $G \subseteq S$: Set of goal states marking subtask completions.

Each $s_i$ includes:
- Natural-language description $d_i$ (current screen).
- Prediction $d_{i+1}$ (expected next screen).
- Pre-condition $\mathrm{pre}^{\,i}$.
- Post-condition $\mathrm{post}^i$.

The FSM is incrementally constructed: upon each screen transition, the MLLM generates descriptions and conditions, after which the State Agent updates the FSM.

## 3. Specialized Agent Functions and Workflows

The system relies on five agent types, each with dedicated responsibilities and interdependence.

| Agent           | Inputs / Outputs                    | Roles and Workflow Highlights                                                       |
|-----------------|------------------------------------|-------------------------------------------------------------------------------------|
| Planner Agent   | $u$, $K$ → plan $\pi$              | Generates, scores, and selects multi-step task decompositions; uses MLLM as judge   |
| Screen Parser   | Screenshot $s_i$                   | Performs OCR, icon detection, and segmentation to produce perception $p_i$          |
| State Agent     | $p_i$, $g_i$                       | Updates FSM, prompts MLLM for state descriptors and pre/post-conditions             |
| Actor Agent     | $g_i$, $p_i$                       | Maps subgoals and perceptions to actions $a_i$; executes actions over ADB           |
| Reflection Agent| $p_{i+1}$, FSM                     | Compares predicted and observed states; manages rollback, recovery, and replanning  |
| Mentor Agent    | FSM, logs                          | Extracts reusable action/guidance sequences and stores them in memory $K$           |

Workflow includes multi-candidate plan generation and ranking, dynamic FSM augmentation based on live perception and subgoal context, and robust error-handling via explicit state tracking and recovery logic.

## 4. Dynamic FSM Construction Algorithm

At each task step $i$, MAPLE runs an UpdateFSM procedure. The workflow:
1. Initialize $s_0$ for the home screen if $i=0$.
2. Prompt the MLLM with $p_i$, $g_i$ to obtain $d_i$, predicted $d_{i+1}$, $\mathrm{post}^i$, and $\mathrm{pre}^{\,i+1}$.
3. Create or retrieve state nodes $s_i$ and $s_{i+1}$ matching $d_i$, $d_{i+1}$.
4. Add or augment transition $(s_i, a_i, s_{i+1}, \mathrm{pre}^{\,i+1}, \mathrm{post}^i)$ in $T$.
5. Actor Agent selects/executes $a_i$.
6. Return the updated FSM and action.

This procedure enables real-time construction of navigation graphs, integration of state and action semantics, and structured context awareness during task execution.

## 5. Evaluation Methodology and Empirical Performance

MAPLE was evaluated on two challenging benchmarks:
- **Mobile-Eval-E:** 25 tasks (19 cross-app), 15 apps, 364 reference actions.
- **SPA-Bench:** 20 English cross-app tasks, 25 apps, 262 reference actions.

**Measured Metrics:**
1. Success Rate (SR): percentage of tasks fully completed.
2. Satisfaction Score (SS): rubric item completion fraction.
3. Action Accuracy (AA): alignment with human action trajectories.
4. Termination Rate (TR): percentage of prematurely aborted tasks.
5. Recovery Success (RS): fraction of failed subtasks successfully recovered.

**Results compared to Mobile-Agent-E + Evo baseline:**

| Metric      | Mobile-Eval-E | SPA-Bench    |
|-------------|---------------|--------------|
| SS          | 86.15% (+7.18 pp) | 88.64% (+8.33 pp) |
| AA          | 83.24% (+6.59 pp) | 84.35% (+6.49 pp) |
| TR          | 16.00% (–8.00 pp) | 20.00% (–5.00 pp) |
| SR          | 84.00% (+12.00 pp) | 80.00% (+5.00 pp) |
| RS          | 71.88% (+4.53 pp) | 66.67% (+13.81 pp) |

Ablation studies demonstrated that removing any key MAPLE component (Planner Agent, multi-plan selection, pre/post-conditions, Mentor Agent) led to substantial performance degradation (e.g., SR as low as 45–52% on SPA-Bench), confirming the necessity and synergy of all core modules. With respect to language model backbones, GPT-4o yielded the strongest results, but MAPLE maintained superiority over baselines even with weaker models (Claude-3.5, Gemini-1.5-Pro) [2505.23596].

## 6. Role and Impact of Structured FSM Memory

MAPLE’s FSM memory delivers multiple functional advantages:
- **Context Tracking:** By recording visited and predicted states, the agent maintains navigation context over extended app flows, mitigating redundant actions and loops.
- **Error Detection:** Explicit pre- and post-condition annotations provide precise criteria for detecting and diagnosing execution failures, instead of relying solely on perception deltas.
- **Robust Recovery:** The FSM encodes reliable rollback points and recovery transitions; notably, this design increased Recovery Success on SPA-Bench by up to 13.8 percentage points.
- **Cross-task Knowledge Transfer:** Persisting FSM graphs and distilled guidance cues in long-term memory $K$ accelerates subsequent planning and execution for similar app flows.

These mechanisms collectively demonstrate that a lightweight, model-agnostic FSM memory augments MLLM-based GUI agents, yielding improved structured planning, real-time verification, and reliable error recovery. The FSM-centric design is model-agnostic and can serve as a memory layer for future mobile GUI agent architectures [2505.23596].

Source: https://www.emergentmind.com/topics/maple-mobile-gui-task-reasoning