---
title: 'CRAFT-GUI: Curriculum-Reinforced Agent for GUI Tasks'
url: https://www.emergentmind.com/topics/craft-gui
type: topic
---

# CRAFT-GUI: Curriculum-Reinforced Agent for GUI Tasks

CRAFT-GUI, short for **Curriculum-Reinforced Agent For GUI Tasks**, is a curriculum-guided reinforcement learning framework for GUI task automation built on **Group Relative Policy Optimization (GRPO)**. It is designed to address two limitations identified in prior RL-based GUI agents: treating training data as a uniform set despite substantial variation in GUI-task difficulty, and relying on coarse rewards that collapse tool choice, argument correctness, formatting, and semantic adequacy into a single signal. CRAFT-GUI therefore combines stage-wise curriculum learning with fine-grained verified rewards, and is evaluated on **AndroidControl**, **AndroidWorld**, and an internal online benchmark derived from real human interaction logs [2508.11360].

## 1. Research setting and problem formulation

CRAFT-GUI is situated in the literature on autonomous agents that operate graphical user interfaces by perceiving screens and issuing actions such as clicking, typing, swiping, or invoking system controls. Its starting observation is that GUI tasks are not uniformly difficult. Some tasks are short and atomic, while others require long trajectories, cross-app navigation, or visual-semantic interpretation. The framework treats this heterogeneity as a first-class optimization problem rather than as noise in the training distribution.

The paper identifies two principal deficiencies in earlier RL-based GUI-agent training. First, optimization becomes unstable when easy and hard tasks are mixed without structure: difficult trajectories can dominate gradients early, while easy trajectories become insufficiently informative later. Second, a single binary success signal is too coarse for GUI automation, because correctness depends on several distinct subdecisions, including the selected tool, the arguments or coordinates, response format, semantic adequacy for understanding tasks, and response length. CRAFT-GUI’s central thesis is that **curriculum learning plus richer verified rewards** yields more stable and more effective policy optimization for GUI interaction [2508.11360].

The framework is primarily developed in a **mobile GUI** setting, but the problem statement is broader. The paper explicitly suggests future extension to **computer usage scenarios** and to **trial-and-error with rollback mechanisms**, indicating that the proposed training principles are intended to generalize beyond the immediate benchmark suite.

## 2. GRPO-based framework and stage-wise curriculum

At the optimization level, CRAFT-GUI is built on **GRPO** rather than PPO. The stated reason is architectural economy: unlike PPO, GRPO **does not require a separate value function**, which reduces memory and compute overhead for large multimodal GUI agents. This choice is directly aligned with the framework’s emphasis on scalable RL over visually grounded action spaces [2508.11360].

The training loop combines four elements: curriculum learning, GRPO, fine-grained verified rewards, and KL regularization. The model is trained sequentially through curriculum stages. For each prompt in the current stage, the policy samples a **group** of candidate outputs; each candidate is scored by the reward function; the rewards are normalized within the group; and the policy is updated using the GRPO objective while a frozen reference model provides KL regularization.

The paper gives the group-relative advantage estimate as

$$
\hat{A_i} = \frac{r_i - \text{Mean}(\{r_1, r_2, \dots, r_G\})}{\text{Std}(\{r_1, r_2, \dots, r_G\})}.
$$

This formulation means that a sampled trajectory is evaluated relative to peer samples produced for the same prompt, rather than against a globally calibrated scalar baseline. The intended effect is variance reduction and more stable learning.

The framework description also presents a curriculum dataset set

$$
\mathcal{C} = \{\mathcal{D}_{\text{basic}, \mathcal{D}_{\text{intermediate}, \mathcal{D}_{\text{advanced}, \mathcal{D}_{\text{ui}\}
$$

together with the update rule in which the reference policy is set from the current policy and then kept frozen during GRPO optimization. The notation is reproduced as described in the source, which also notes that the implementation follows the original GRPO advantage-estimation strategy rather than introducing a more elaborate critic.

## 3. Difficulty modeling and curriculum design

CRAFT-GUI defines task difficulty primarily by **trajectory length**, namely the number of interaction steps required to complete an instruction. This is the framework’s operational proxy for difficulty: short trajectories are treated as simpler, while longer trajectories imply more planning, stronger state dependence, and more opportunities for error. The paper also treats several visual understanding tasks as inherently harder than direct operation tasks because they require semantic interpretation rather than action execution.

| Tier | Criterion | Typical content |
|---|---|---|
| Basic | $\le 3$ interaction steps | Simple operation tasks |
| Intermediate | $4$ to $8$ steps | Typical multi-step workflows |
| Advanced | $> 8$ steps | Long trajectories plus VQA, information extraction, and element localization |

Training proceeds sequentially through **Stage 1: Basic**, **Stage 2: Intermediate**, and **Stage 3: Advanced** [2508.11360]. The framework presents this progression as analogous to human learning: basic UI manipulation is acquired first, followed by multi-step operations, and only then more abstract perception and reasoning tasks. In the paper’s own framing, this is a form of **difficulty-aware decoupling** of the training distribution.

The choice of trajectory length is explicitly acknowledged as practical rather than perfect. A short task may still be hard, and a long task may be routine. Even so, the paper treats length as a workable proxy that can be implemented consistently across large GUI-task corpora.

## 4. Reward decomposition and verified supervision

A major contribution of CRAFT-GUI is its hybrid reward design, which combines **rule-based verification** with **model-judged evaluation**. The objective is to replace a blunt success/failure signal with a structured reward that exposes where an action failed.

For **mobile operation tasks**, the overall reward is

$$
R_{operation} =\ w_{tool} * R_{tool} + w_{args} * R_{args} + w_{f} * R_{format} + w_{l} * P_{length}.
$$

Here, $R_{tool}$ checks tool selection, $R_{args}$ checks parameter correctness, $R_{format}$ enforces structured output, and $P_{length}$ penalizes overlong responses. The tool reward is defined as

$$
R_{tool} = \begin{cases} 1 & \text{if } o_{tool} \text{ matches } gt_{tool}, \\ 0 & \text{else.} \end{cases}
$$

and the argument reward is defined as

$$
R_{paras} = \begin{cases} 1 & \text{if } o_{paras} \text{ matches } gt_{paras}, \\ 0 & \text{else.} \end{cases}
$$

with task-specific matching logic. For **click** and **long press**, the predicted point must fall within the same widget as ground truth. For **swipe**, the angle between predicted and ground-truth vectors must be within $30^\circ$. For **type** and **open app**, correctness is determined by semantic similarity plus character-level matching. For **system button**, matching is strict string equality. For **wait** and **terminate**, there is no argument reward.

The format reward is

$$
R_{format} = \begin{cases} 1 & \text{if } o_i \text{ matches response pattern}, \\ 0 & \text{else.} \end{cases}
$$

and the paper states that the response schema uses HTML-style tags such as `<think>`, `<tool_call>`, and `<task_progress>`. This makes formatting itself part of policy optimization rather than a downstream post-processing concern.

The length term is introduced to suppress “runaway reasoning-token growth.” The source describes a piecewise penalty intended to preserve useful reasoning while discouraging verbosity, though it notes formatting errors in the printed expression. The conceptual role of the term is therefore clearer than its exact typographic rendering.

For **visual understanding tasks**, the reward becomes

$$
R_{understanding} = w_{s} * R_{sem} + w_{f} * R_{format} + w_{l} * P_{length},
$$

where the semantic component $R_{sem}$ is supplied by an **LLM-as-judge** mechanism rather than symbolic verification [2508.11360]. This distinction reflects the different verification regime of understanding tasks: open-ended answers often cannot be checked with exact rules.

The reward design is thus intentionally heterogeneous. It uses hard verification where exact correctness is available and model-judged feedback where the task is semantically open-ended. A plausible implication is that CRAFT-GUI treats GUI interaction not as a monolithic action problem, but as a bundle of partially separable subskills.

## 5. Training pipeline, benchmarks, and empirical results

CRAFT-GUI uses **Qwen2.5-VL** backbones, specifically **7B** and **32B** variants. The training data are split into **basic operation data**, **intermediate operation data**, **advanced operation data**, and **understanding data**. Policy optimization then proceeds stage by stage, with group sampling, reward computation, group-relative normalization, and a GRPO update against a frozen reference model [2508.11360].

The public evaluation is conducted on **AndroidControl** and **AndroidWorld**. AndroidControl reports **Type**, **Grounding**, and **Success Rate (SR)**, and the paper defines the main metric as

$$
SR = \frac{N_{\text{success}}}{N_{\text{total}}} \times 100\%.
$$

The experiments are evaluated under **zero-shot generalization** on unseen task distributions and **pass@1** single-attempt execution.

The internal online benchmark is constructed from **real human interaction logs** and spans six application categories: **food delivery**, **in-store dining**, **medical services**, **financial services**, **insurance services**, and **gaming**. Each category contains **120–200 unique test cases**, and the benchmark is intended to measure page comprehension, contextual reasoning, and decision-making.

| Setting | Result |
|---|---|
| Public benchmarks | **+5.6%** improvement |
| Internal online benchmarks | **+10.3%** improvement |
| Internal benchmark, CRAFT-GUI-32B | **75.7% average SR** |

The abstract’s headline claim is that CRAFT-GUI outperforms previous state-of-the-art approaches by **5.6% on public benchmarks Android Control** and **10.3% on internal online benchmarks** [2508.11360]. On the internal benchmark, **CRAFT-GUI-32B** reaches **75.7% average SR**, compared with **65.4%** for **UI-TARS-72B**, **64.8%** for **Qwen2.5-VL-72B**, **35.9%** for **Claude-3.7-Sonnet**, and **27.7%** for **GPT-4.1**.

On AndroidControl and AndroidWorld, **CRAFT-GUI-32B-stage3** reports **92.7%** SR on **AndroidControl-Low**, **80.3%** SR on **AndroidControl-High**, and **51.7%** SR on **AndroidWorld**. The paper compares these with strong baselines including **UI-TARS-72B** at **91.3 / 74.7 / 46.6** and **Qwen2.5-VL-72B** at **93.7 / 67.4 / 35.0**. The pattern emphasized in the paper is that CRAFT-GUI is competitive on lower-difficulty AndroidControl settings while stronger on harder settings and on AndroidWorld.

The training-method ablation is central to the paper’s argument. On the internal benchmark, **SFT** yields **60.8%**, **Vanilla GRPO** yields **71.9%**, and **Curriculum GRPO** yields **75.7%**. The paper therefore reports **+11.1% over SFT** for vanilla GRPO, **+14.9% over SFT** for curriculum GRPO, and **+3.8% over vanilla GRPO**. A separate data-mixture ablation shows **73.2%** for **Operation only** and **75.7%** for **Operation + Understanding**, a **+2.5%** gain indicating that understanding data improves operation performance.

## 6. Interpretation, limitations, and relation to adjacent work

The paper’s interpretation of its own results is that easy-to-hard progression stabilizes optimization, richer rewards provide denser and more informative feedback, and understanding tasks strengthen visual-semantic representations that transfer back to GUI operation [2508.11360]. In this sense, CRAFT-GUI reframes GUI-agent training from uniform RL over heterogeneous tasks into a staged regime with explicit difficulty structure and decomposed supervision.

Several limitations are also explicit. **Trajectory length as difficulty proxy** is practical but imperfect. The curriculum itself is **hand-designed** rather than adaptively learned. Rule-based reward terms depend on task structure and therefore work best for structured GUI operations. The **LLM-as-judge** component may introduce noise or bias. The primary empirical setting remains **mobile GUI**, and the **length penalty** is acknowledged as heuristic, with the risk of suppressing beneficial reasoning if tuned poorly.

Within adjacent GUI-agent research, CRAFT-GUI occupies a distinct niche. **GUICrafter** addresses GUI-agent weakness primarily as a **data-scarcity** problem and uses a two-stage weakly supervised curriculum built from unannotated screenshots and webpages, followed by GRPO-based calibration on a smaller high-quality annotated set [2606.29705]. **GUI-World** addresses a different gap again, focusing on **dynamic GUI videos**, **multi-window interaction**, and **temporal reasoning**, and concludes that current multimodal models still struggle with dynamic GUI content and sparse GUI-video signals [2406.10819]. Relative to these works, CRAFT-GUI’s contribution is not benchmark construction or weak supervision from raw GUI evidence, but the integration of **difficulty-aware curriculum learning** and **fine-grained hybrid rewards** into RL-based GUI interaction.

It should also be distinguished from **CGui**, a **declarative, constraint-based language and toolchain** for highly customized cross-platform GUI development, which concerns GUI construction rather than GUI-agent policy learning [2302.03739]. The similarity of names can obscure the fact that the two systems belong to different problem classes: one generates or automates user interfaces, while the other trains agents to act within them.

Taken together, these distinctions place CRAFT-GUI as a reinforcement-learning framework for GUI-task execution whose primary technical claims concern curriculum structure, verified reward granularity, and empirical gains on operation-heavy benchmarks. Its broader significance lies in showing that, for GUI agents, **how tasks are ordered** and **how correctness is decomposed** can matter as much as model scale or raw data volume.

Source: https://www.emergentmind.com/topics/craft-gui