UItron: Open-Source GUI Agent

• UItron is an open-source foundational model that integrates multimodal perception, grounded captioning, and chain-of-thought action planning for automated GUI operations.
• It employs a systematic data engineering pipeline, including multi-task unification and targeted manual annotation for Chinese app scenarios, to overcome UI trajectory challenges.
• The model utilizes a multi-stage training paradigm with curriculum reinforcement learning and backtracking, ensuring robust performance across mobile and PC environments.

UItron is an open-source foundational model designed as an automatic graphical user interface (GUI) agent, specifically targeting automated operations on both mobile and PC devices. Developed in response to persistent challenges in GUI agent construction—including the scarcity of high-quality operation trajectories, integration of robust perception and planning, and the need for realistic interactive testbeds—UItron unifies advanced multimodal perception, grounding, and action planning within a single architecture. Its notable contributions include systematic data engineering, curriculum reinforcement learning for complex reasoning and exploration, and dedicated infrastructure for seamless agent evaluation and deployment on real devices, with a pronounced emphasis on capability in Chinese app scenarios (Zeng et al., 29 Aug 2025).

1. Unified GUI Perception and Planning

UItron’s architecture centers on the ability to robustly perceive GUI environments and plan appropriate action sequences based on real task instructions and prior actions. Its perception capabilities are built on supervised fine-tuning in grounding, captioning, visual question answering (VQA), and optical character recognition (OCR), producing a model that can accurately interpret interface screenshots and detect UI components.

The planning formulation follows:

$$a_n = M_{\theta} (T, \{a_1, a_2, ..., a_{n-1}\}, o_n)$$

where $a_n$ is the action at step $n$, $T$ the task instruction, $o_n$ the current GUI observation, and $\{a_1, ..., a_{n-1}\}$ the action history. UItron augments forward planning with a backtracking mechanism, enabling it to reconstruct prior actions and reason about the trajectory:

$$a_{n-1}, a_{n} = M_{\theta}(T, \{a_1, ..., a_{n-2}\}, o_{n-1}, o_{n})$$

This multi-level (L1–L3) inference, including explicit chain-of-thought steps and reflective planning, equips UItron to both predict and rationalize action sequences, addressing error recovery and consistent task execution.

During reinforcement learning, UItron applies group relative policy optimization (GRPO), using a loss formulated over groups of candidate responses and normalized advantages, thus facilitating dense reward propagation and convergence in both offline and online training regimes.

2. Data Engineering Pipeline

A core pillar of UItron’s effectiveness is its systemic data engineering strategy, comprising:

• Multi-Turn Conversation Synthesis: Aggregation of several instruction–response pairs into multi-round dialogues for richer context per screenshot.
• Multi-Task Unification: Simultaneous inclusion of traditional GUI trajectories, OCR, VQA, and descriptive captioning datasets, blending varied sources into a converged training set.
• Layered Planning Data: Decomposition into hierarchical reasoning levels; for instance, L2 planning includes explicit intermediate “reasoning” between state and action, and backtracking enforces the link between recent actions and resulting states.
• Automated Trajectory Distillation: Automated task recovery, seeding, and iterative candidate trajectory generation. Candidate quality is adjudicated via vision-LLM (VLM)-based voting, ensuring selection of high-fidelity learning targets.
• Manual Annotation for Chinese Scenarios: Recognizing limited coverage of Chinese UIs in previous datasets, over one million operation steps were manually collected and annotated from the top 100 Chinese mobile apps. This enables robust performance in domains underrepresented in prior research.

These strategies address data sparseness, diversity of UI interactions, and ensure representative sampling across application genres and linguistic domains.

3. Interactive Infrastructure Across Mobile and PC

UItron implements a dual infrastructure for agent deployment and evaluation:

• Android Cloud Environment: Utilizes Scrcpy for real-time screen streaming, a Phone-Server component translating browser interaction into touch events, and a Device-Agent managing device state and exposing HTTP APIs for operations and APK installations.
• PC Environment (OSWorld): Provides real-computer emulation (across Windows, macOS, Ubuntu) with full keyboard and mouse control integration.

This infrastructure is geared for both automatic data collection (with screenshots and pointer coordinates auto-captured for every step) and online reinforcement learning. It ensures that experimental evaluations are conducted in environments representative of real end-user scenarios, minimizing the simulation–reality gap.

4. Multi-Stage Training Paradigm

Training UItron proceeds through three major sequential stages:

1. Perception Supervised Fine-Tuning: Focused on grounding, VQA, OCR, and captioning, this stage calibrates the visual backbone and ensures high-fidelity representation of GUI states.
2. Action Planning Fine-Tuning: Involves both next-action prediction (forward planning) and recovery/backtracking (reconstructing previous steps based on observation sequences). This dual training enhances both robustness and reasoning depth.
3. Curriculum Reinforcement Learning (RL): Employs a curriculum-based approach where task complexity is gradually increased. Initial offline RL leverages dense, step-level rewards for rapid convergence; subsequent online RL incorporates whole-trajectory evaluation using advanced VLM-based scorers, with multi-model consensus filtering to ensure scoring reliability.

In RL, the GRPO algorithm is central, optimizing over relative advantages within candidate groups and thus facilitating both intra-batch and inter-batch consistency in learning.

5. Performance Evaluation and Benchmarks

Benchmarking involves a comprehensive battery of both perception and planning tasks:

Benchmark	Role in Evaluation	Domains Covered
VisualWebBench	Perception/grounding	Web GUIs, element location
RefExp	Referring expression comprehension	GUI element referencing
WidgetCap	Widget captioning	Visual description tasks
WebSRC	Visual question answering	Role reasoning, OCR
AndroidControl	Planning (Low/High-level)	Multi-step mobile navigation
GUI-Odyssey	Planning, action precision	Cross-app mobile workflows
OSWorld	Real-computer (PC) planning	Full operating system tasks

UItron exhibits high accuracy and step/task success rates on these tasks. In studies involving the top Chinese mobile apps, UItron shows marked gains in both offline and online scenarios, attributed directly to targeted data engineering and annotation. RL-enhanced UItron (UItron-RL) demonstrates improved recovery from exploration errors and consistency in long-horizon action planning.

6. Specialization for Chinese App Scenarios

UItron’s design specifically addresses a functional gap in Chinese application environments, as state-of-the-art models have historically underperformed due to minimal Chinese data in training. The manual collection and annotation of >1M steps from the top Chinese mobile applications result in significant improvements in grounding, action precision, and overall task success rates for UI scenarios involving complex iconography, diverse page layouts, and localized language elements. This capability is validated through both controlled offline tests and online agent rollout experiments.

7. Scientific Significance and Implications

UItron sets a new baseline in foundational GUI agent development by integrating multimodal perception, advanced planning (with backtracking and chain-of-thought reasoning), and comprehensive data-centric engineering. Its contributions address crucial challenges including generalization across device types, robustness in non-English (notably Chinese) UI contexts, and the bridging of simulation and deployment environments.

Experimental results demonstrate that UItron achieves or surpasses leading models (e.g., UI-TARS) on key perception and planning benchmarks, with parameter scaling (e.g., 7B vs. 72B variants) further boosting performance. The use of curriculum RL and principled evaluation demonstrates reliable ability for trajectory recovery and action sequence consistency, critical for real-world interactive deployment.

A plausible implication is that UItron’s systematic fusion of data, infrastructure, and learning methodology is adaptable for future research in autonomous interface agents, workflow automation, and interactive system design, especially in culturally and linguistically diverse software ecosystems.