---
title: 'GUI-Libra: Post-Training for Native GUI Agents'
url: https://www.emergentmind.com/topics/gui-libra
type: topic
---

# GUI-Libra: Post-Training for Native GUI Agents

Searching arXiv for the primary paper and the potentially confusable earlier “Libra” toolkit paper.
GUI-Libra is a post-training framework for **native GUI agents**: single end-to-end vision-language-action models that map a user instruction, GUI observation, and interaction history directly to executable actions in web and mobile environments. It is designed for long-horizon navigation, where open-source native GUI agents have lagged behind stronger closed-source or multi-module systems, and it combines a curated **81K-step / 9K-trajectory** GUI reasoning dataset, **action-aware supervised fine-tuning**, and **conservative RL** for partially verifiable step-wise rewards [2602.22190]. The name should not be conflated with the earlier **Libra Toolkit for Probabilistic Models**, which is a command-line toolkit for discrete probabilistic modeling rather than a graphical user interface system [1504.00110].

## 1. Scope, problem setting, and nomenclature

In the terminology of the 2026 paper, native GUI agents are **single end-to-end models** that directly produce executable GUI actions from instruction and observation, instead of relying on an external planner or a separate grounding module. The policy is formulated as
$$
\pi_\theta(a_t \mid \ell, h_t, o_t),
$$
where $\ell$ is the task instruction, $h_t$ the interaction history, and $o_t$ the current observation. The target use case is **high-level multi-step GUI navigation** across Android and web interfaces, rather than isolated grounding or one-step clicking [2602.22190].

The work is motivated by two failures in generic post-training pipelines when transferred to GUI agents. First, **standard SFT with CoT reasoning often hurts grounding**. Second, **step-wise RLVR-style training faces partial verifiability**, because multiple actions can be correct at a given state while offline supervision usually verifies only a single demonstrated action. The paper treats these as structural properties of GUI training rather than incidental implementation defects. A plausible implication is that GUI-Libra is less an architectural proposal than a training recipe specialized to the joint requirements of reasoning, grounding, and sequential control.

A recurrent source of confusion is the word “Libra.” The 2015 **Libra Toolkit** is a collection of command-line programs and shared libraries for learning and inference with discrete probabilistic models, including Bayesian networks, Markov networks, dependency networks, sum-product networks, arithmetic circuits, and mixtures of trees. It is explicitly not presented as a GUI, visualization dashboard, or interactive graphical front-end; it is described as a toolkit of command-line executables plus shared OCaml libraries [1504.00110]. In that sense, “GUI-Libra” is not a GUI wrapper around the 2015 Libra software, but a separate 2026 line of work on GUI agents.

## 2. Overall training recipe and system formulation

GUI-Libra is organized as a three-part recipe: a curated reasoning dataset, **Action-Aware SFT (ASFT)**, and **conservative RL**. The dataset is intended to compensate for the shortage of high-quality, action-aligned reasoning traces. ASFT is intended to preserve explicit reasoning without allowing long chain-of-thought to dominate grounding learning. Conservative RL is intended to stabilize policy optimization when step rewards are only partially verifiable [2602.22190].

The model input contains a system prompt listing available actions, a user instruction, interaction history, and the current screenshot. The model output uses a structured format with `<think> ... </think>` reasoning and `<answer> ... </answer>` action JSON. The unified action schema contains `action_type`, `action_description`, `action_target`, `value`, and `point_2d`. The action space comprises **13 action types**: **Click**, **Write**, **Terminate**, **Swipe**, **Scroll**, **NavigateHome**, **Answer**, **Wait**, **OpenAPP**, **NavigateBack**, **KeyboardPress**, **LongPress**, and **Select** [2602.22190].

The formulation distinguishes reasoning tokens, action tokens, and grounding tokens. This partition is central because the paper’s diagnosis is that autoregressive training on long CoT responses causes optimization to overemphasize semantic reasoning relative to executable action specification. The resulting degradation is not merely theoretical: the paper reports that longer responses correlate with lower grounding accuracy on ScreenSpot-v2. This suggests that, in GUI settings, token-level supervision must be explicitly structured around execution-critical output rather than treated as a homogeneous language modeling target.

## 3. GUI-Libra-81K: dataset construction and filtering

The released dataset, **GUI-Libra-81K**, is constructed from public GUI trajectory corpora: **GUI-Odyssey**, **AMEX**, **AndroidControl**, **AitZ**, **AitW**, **GUIAct**, and **MM-Mind2Web**. Relative to AGUVIS’s collection, the dataset additionally includes the **Chinese subset from GUIAct**. Initial cleaning removes incomplete trajectories, trajectories shorter than **3 steps**, trajectories longer than **50 steps**, and steps with compound actions outside the action space. After this stage, the source pool contains **19K trajectories** and **170K steps** [2602.22190].

Most source datasets do not contain rich reasoning traces, so GUI-Libra synthesizes them with prompting. The paper compares **GPT-4o**, **o4-mini**, and **GPT-4.1**, and uses **GPT-4.1** because it produces richer visible rationales. The prompt includes observation description, reflection on instruction and history, planning, action-related requirements, and strict output formatting. Importantly, the generator is not forced to repeat the original dataset action exactly; the original annotation is treated as a reference, and a different action may be produced if justified. Coordinates, however, are reused from the original dataset [2602.22190].

Filtering is aggressive and two-stage. First, **Qwen3-VL-8B-Instruct** is run for **10 stochastic runs** on each input, and a step is discarded if action re-prediction accuracy is below **0.3**. Second, coordinate alignment is verified by prompting **Qwen3-VL-32B-Instruct** to predict a bounding box from screenshot and target description; a sample is kept only if the original point lies inside the predicted box. After filtering, the final SFT set contains **81K steps** and **9K trajectories**. The RL subset is further balanced by downsampling early steps and mobile-heavy data, producing **40K steps** [2602.22190].

The dataset is predominantly mobile: only **14.3%** of the SFT data is web-domain data. The average reasoning length is **210 thought tokens per step**, compared with **11** for AndroidControl, **37** for GUI-Net-1M, **56** for AGUVIS Stage 2 L2, and **85** for AGUVIS Stage 2 L3. The action distribution is highly skewed: **Click** accounts for about **60%** of steps, followed by **Write**, **Terminate**, and **Swipe**, while **LongPress** and **Select** are rare. This imbalance is one of the paper’s explicit motivations for RL.

| Component | Value | Role |
|---|---:|---|
| Source pool after cleaning | 19K trajectories / 170K steps | Pre-filter corpus |
| Final SFT set | 81K steps / 9K trajectories | GUI-Libra-81K |
| Final RL subset | 40K steps | Balanced RL data |

## 4. Action-aware supervised fine-tuning

ASFT modifies standard SFT in two ways. It mixes **reasoning-then-action** samples with **direct-action** samples produced by removing the CoT segment, and it reweights token groups to emphasize action specification and especially grounding coordinates. The loss is
$$
\mathcal{L}_{\text{ASFT}}(\theta) = - \mathbb{E}_{(x_t, c_t, a_t, g_t) \sim D_{\rm mix}}
\frac{
\log \pi_\theta(c_t \mid x_t) + \alpha_a \log \pi_\theta(a_t \mid x_t, c_t) + \alpha_g \log \pi_\theta(g_t \mid x_t, c_t, a_t)
}{
|c_t| + \alpha_a |a_t| + \alpha_g |g_t|
},
$$
where $c_t$ denotes reasoning tokens, $a_t$ action tokens excluding `point_2d`, and $g_t$ the grounding tokens associated with `point_2d` [2602.22190].

Default weights are $\alpha_a = 2$ and $\alpha_g = 4$, except for GUI-Libra-4B where $\alpha_a=\alpha_g=1$. In effect, coordinates receive the strongest supervision. The paper also notes useful special cases: $\alpha_a=\alpha_g=1$ recovers standard SFT; $\alpha_a=\alpha_g\gg1$ approximates CoT-free SFT; and $\alpha_g\gg\alpha_a$, $\alpha_g\gg1$ approximates grounding-only SFT. This framing makes ASFT a controlled interpolation among standard instruction tuning, direct-action tuning, and grounding-focused tuning.

Ablations show that mixed direct-action data already yields large gains, particularly on online AndroidWorld, and that action-aware weighting adds further improvements. With **Qwen2.5-VL-3B**, the progression on **MM-Mind2Web-v2 Pass@1 / AndroidControl-v2 High Pass@1 / AndroidWorld** is **23.4 / 36.4 / 3.5** for the base model, **28.5 / 45.7 / 5.2** for SFT, **30.2 / 45.5 / 11.3** for SFT plus mixed data, and **32.0 / 44.5 / 13.0** for ASFT [2602.22190].

The grounding analysis is especially significant. Under reasoning mode on ScreenSpot-style evaluation, **SFT-3B** scores **73.4**, **SFT 3B + mixed data** scores **73.8**, **ASFT 3B** scores **76.2**, and **GUI-Libra-3B** reaches **83.4**. At **7B**, the corresponding reasoning-mode scores rise from **79.0** for SFT-7B to **81.4** for mixed-data SFT, **83.4** for ASFT, and **89.3** for GUI-Libra-7B. The paper interprets this as evidence that RL nearly removes the remaining reasoning-grounding gap after ASFT.

## 5. Conservative RL under partial verifiability

The paper defines GUI RL rewards as **partially verifiable**. For a state $s$, the dataset provides one demonstrated action $\tilde a(s)\in\mathcal{A}^*(s)$, and the verifier reward is
$$
\tilde r(s,a) \triangleq \mathbf{1}\{a=\tilde a(s)\}.
$$
This is partially verifiable because
$$
\tilde r(s,a)=1 \Rightarrow a\in \mathcal{A}^*(s)
\quad\text{but}\quad
\tilde r(s,a)=0 \not\Rightarrow a\notin \mathcal{A}^*(s).
$$
A zero reward may therefore correspond either to a truly incorrect action or to a valid alternative that the offline annotation did not record [2602.22190].

RL uses **GRPO** with KL regularization:
$$
\mathcal{L}_{\mathrm{GRPO}}(\theta) = - \mathbb{E}_{x \sim D, \{a_k\}_{k=1}^{G}\sim \pi_{\theta_{\rm old}}(\cdot\mid x)}
\left[
\frac{1}{G}\sum_{k=1}^{G}
\min\!\Big( \rho_k \hat{A}_k,\; \mathrm{clip}(\rho_k, 1-\epsilon, 1+\epsilon)\hat{A}_k \Big)
- \beta \cdot \mathrm{KL}\!\left(\pi_\theta(\cdot\mid x)\,\|\,\pi_{\mathrm{ref}}(\cdot\mid x)\right)
\right],
$$
with $\rho_k=\pi_\theta(a_k\mid x)/\pi_{\theta_{\mathrm{old}}}(a_k\mid x)$ and group-normalized advantage $\hat A_k=(r_k-\mu_r)/(\sigma_r+\delta)$. Reward shaping combines a small format term and a mostly action-accuracy term:
$$
\tilde r(s,a) = w_{\text{fmt}}\, r_{\text{fmt}} + (1 - w_{\text{fmt}})\, r_{\text{acc}},
\qquad
w_{\text{fmt}}=0.1,
$$
with
$$
r_{\text{acc}} = r_{\text{act}} \cdot r_{\text{val}} \cdot r_{\text{g}}.
$$
Here $r_{\text{act}}$ verifies action type, $r_{\text{val}}=1$ if word-level F1 for `value` exceeds **0.5**, and $r_{\text{g}}=\mathbf{1}\{\mathbf{u}\in b^\star\}$ verifies that the predicted point lies inside the demonstrated bounding box [2602.22190].

A central theoretical claim is that offline step metrics predict online success only when occupancy mismatch and off-demonstration valid-action mass are controlled. The paper defines
$$
M_{\text{off}}(\pi) \triangleq \mathbb{E}_{s\sim d_\mu}\big[\pi(\tilde a(s)\mid s)\big],
$$
$$
C(\pi)\triangleq \max_{t\in[H]}\sup_{s:\,d_\mu(s)>0}\frac{d_{\pi,t}(s)}{d_\mu(s)},
$$
and
$$
\eta_\pi(s)\triangleq \pi\!\left(\mathcal{A}^*(s)\setminus\{\tilde a(s)\}\mid s\right), \qquad
\bar\eta_\pi\triangleq \mathbb{E}_{s\sim d_\mu}[\eta_\pi(s)].
$$
The main lower bound is
$$
J(\pi)\ \ge\ 1\;-\;H\cdot C(\pi)\cdot\Big(1-M_{\mathrm{off}}(\pi)-\bar\eta_\pi\Big).
$$
This suggests that improving offline step matching is insufficient unless the policy remains close enough to the reference distribution and does not move excessive probability mass onto uncredited but valid alternatives.

The RL recipe therefore emphasizes a **KL trust region**. Empirically, moderate KL is optimal: for **3B** on **AndroidWorld**, **no KL** gives **21.7**, **KL = 0.001** gives **25.2**, **KL = 0.01** gives **21.7**, and **KL = 0.05** gives **20.0**. Checkpoint correlations between offline and online metrics strengthen markedly when KL is present: overall Pearson correlation is **0.76**; with **KL \(>0\)** it rises to **0.89** with Spearman **0.83**; without KL it drops to Pearson **0.63** and Spearman **0.53** [2602.22190]. The paper treats this as evidence that KL is not merely a generic stabilizer but a mechanism for preserving offline-to-online predictability under partial verifiability.

GUI-Libra also introduces **success-adaptive negative gradient scaling**. Let
$$
\hat p_g(s) \triangleq \frac{1}{G}\sum_{k=1}^{G} \tilde r_k,
$$
and
$$
\lambda_g(s) \triangleq \min \left( \lambda_0 + \kappa\,\hat p_g(s),\; 1 \right).
$$
Negative advantages are rescaled as
$$
\tilde A_k \triangleq
\begin{cases}
A_k, & A_k \ge 0,\\[2pt]
\lambda_g(s)\,A_k, & A_k < 0.
\end{cases}
$$
This downweights ambiguous negative updates when group success is low. On **GUI-Libra-4B**, the method raises **AndroidWorld** from **39.1** to **42.6** and **WebArena-Lite-v2** from **22.2** to **24.4** [2602.22190].

## 6. Models, evaluation, and significance

GUI-Libra is trained from **Qwen2.5-VL-3B-Instruct**, **Qwen2.5-VL-7B-Instruct**, **Qwen3-VL-4B-Instruct**, and **Qwen3-VL-8B-Instruct**, yielding released models from **3B to 8B**. The observation modality is primarily the screenshot plus textual context. **Qwen2.5-VL-based** models use **absolute pixel coordinates**, whereas **Qwen3-VL-based** models use **normalized coordinates in $[0,1000]$**. SFT uses full-parameter fine-tuning with learning rate **$1\times10^{-5}$** and effective batch size **256**. RL uses learning rate **$1\times10^{-6}$**, **300** training iterations, rollout batch size **256**, global batch size **128**, group size **8**, clip ratio **$0.2$**, and KL coefficient **0.005 for 7B** and **0.001 for others** by default [2602.22190].

Evaluation spans grounding, offline navigation, and online navigation. Grounding uses **ScreenSpot-v2** and **ScreenSpot-Pro**. Offline navigation uses **AndroidControl-v2** and **MM-Mind2Web-v2**. Online navigation uses **AndroidWorld**, **WebArena-Lite-v2**, and **Online-Mind2Web**. The offline datasets were refined because MM-Mind2Web contains symbolic or non-natural action histories and AndroidControl contains substantial label errors. The online suites cover **115 usable AndroidWorld tasks**, **154 WebArena-Lite-v2 tasks**, and **300 Online-Mind2Web tasks over 136 websites** [2602.22190].

The main quantitative results show consistent gains in both step-wise accuracy and end-to-end success. On **AndroidControl-v2 High Pass@1**, GUI-Libra improves from **36.4** to **57.3** at **3B**, from **46.5** to **59.3** at **7B**, from **49.3** to **62.3** at **4B**, and from **54.8** to **64.3** at **8B**. On **MM-Mind2Web-v2 average Pass@1**, the improvements are **23.4 → 42.7** at **3B**, **32.5 → 46.5** at **7B**, **41.2 → 50.0** at **4B**, and **43.8 → 50.5** at **8B** [2602.22190].

Online gains are larger still. On **AndroidWorld**, the progression is **3.5 → 25.2** at **3B**, **7.8 → 29.6** at **7B**, **27.0 → 42.6** at **4B**, and **30.4 → 42.6** at **8B**. On **WebArena-Lite-v2 average**, the scores become **0.8 → 16.7** at **3B**, **4.9 → 22.6** at **7B**, **11.9 → 24.4** at **4B**, and **15.3 → 26.6** at **8B**. On **Online-Mind2Web average overall**, the changes are **4.8 → 21.3**, **15.8 → 25.5**, **21.7 → 25.7**, and **19.3 → 28.0** respectively [2602.22190].

These results support three broader claims. First, long-horizon GUI performance can be improved substantially without costly online data collection, provided data curation and post-training are specialized to GUI interaction. Second, explicit reasoning is useful, but only when balanced against executable action supervision; removing CoT from training hurts sharply, especially online. Third, RL for GUI agents is not well described by fully verifiable RLVR assumptions. GUI-Libra’s central conceptual contribution is therefore the reframing of GUI post-training around **action-aligned reasoning**, **grounding-sensitive supervision**, and **partial verifiability**.

The paper also states clear limitations. The training data are limited to existing open-source corpora, web coverage is still small relative to mobile coverage, fully online RL is not studied, and direct grounding supervision can trade off against reasoning and navigation performance. This suggests that GUI-Libra should be understood as a data-efficient post-training framework rather than a complete solution to open GUI-agent learning [2602.22190].

Source: https://www.emergentmind.com/topics/gui-libra