UI-TARS: Native GUI Task-Aware Reasoning

• UI-TARS is a unified native agent model that interacts with GUIs through screenshot-level perception and a consolidated action space.
• It integrates advanced reasoning, enhanced perception, and multi-turn reinforcement learning to achieve state-of-the-art performance on diverse GUI tasks.
• The system employs a continuous loop of perception, reasoning, and action, supporting hybrid environments for robust real-world deployment and scalable automation.

UI-TARS (Universal Interface Task-Aware Reasoning System) refers to a family of native GUI agent models specialized for autonomous interaction with graphical user interfaces through end-to-end learning. Unlike modular frameworks reliant on external models and prompt engineering, UI-TARS agents consume only screenshot-level perceptual input and emit actions in a unified space grounded in human interaction modalities (mouse, keyboard, gestures). Progressing from the original UI-TARS to UI-TARS-2, the approach systematically integrates enhanced perception, unified action modeling, explicit System-2 reasoning, scalable training with reflective trace bootstrapping, and, in its latest iteration, stabilized multi-turn reinforcement learning within hybrid GUI-tool environments. These models achieve state-of-the-art performance on diverse GUI and game tasks and exhibit scalability, generalization, and robust real-world deployment characteristics (Qin et al., 21 Jan 2025, Wang et al., 2 Sep 2025).

1. Core System Architecture and Operation

UI-TARS agents implement a native pipeline for GUI-centric agentic reasoning, perception, and control. Each agent cycle consists of:

• Perceptual Processing: Ingests current GUI screenshots (and, optionally, associated DOM/A11y data) to extract structured element descriptions, dense captions, and state deltas, using a large vision encoder (e.g., 532M parameters) interfaced with multimodal LLMs such as Qwen-2-VL.
• Unified Reasoning–Action Loop: Maintains an explicit thought stream $t_t$ before every action $a_t$, interleaving instructions, observations, working memory $W_t$ (sliding window of recent steps), and compressed episodic memory $E_t$ (summaries of past episodes).
• Action Output: Predicts GUI actions—including clicks, drags, typing, scrolling—or issues predefined SDK functions (e.g., shell commands, file operations) when enabled by hybrid environments.
• Observation Update: Receives new observations after executing each action, iterating the perception–reasoning–action–observation loop until task completion.

This continuous loop is formalized as: $$P(t_{t+1}, a_{t+1} \mid \text{instruction}, W_t, o_{t+1}, E_t)$$

Reasoning leverages System-2 methodologies: task decomposition, milestone recognition, reflection for error recovery, and explicit thought generation prior to acting (Qin et al., 21 Jan 2025, Wang et al., 2 Sep 2025).

2. Enhanced Perception and Unified Action Modeling

The perception module is trained on large-scale multimodal GUI datasets (15M elements, 6M tutorials, 7.5M captions; 50B+ tokens). Five principal subtasks are supported: element description, dense captioning, state-transition captioning, visual QA, and set-of-mark prompting.

Unified Action Modeling abstracts GUI manipulations and tool invocations into a shared, cross-platform space. This comprises:

• Atomic Actions: $\mathrm{Click}(x, y)$, $\mathrm{Type}(\text{content})$, $\mathrm{Scroll}(x, y, dir)$, $\mathrm{Drag}(x_1, y_1, x_2, y_2)$, etc.
• Platform-Specific Actions: Hotkeys, long presses, Android navigation commands.
• Parameter Grounding: Actions are associated with normalized screen coordinates and parameter embeddings.

Behavior cloning is performed on multi-step human interaction traces, employing cross-entropy for action types and mean squared error for grounding coordinates: $$\mathcal{L}_{\mathrm{cap}} = -\sum_{t=1}^T \log P(y_t\mid y_{<t},\mathbf{I}), \qquad \mathcal{L}_{\mathrm{bbox}} = \frac{1}{N}\sum_{i=1}^N \left\|(\hat x_i, \hat y_i) - (x_i, y_i)\right\|_2^2$$ (Qin et al., 21 Jan 2025).

3. Iterative Training and Data Flywheel Strategies

UI-TARS adopts a multi-stage iterative training pipeline that addresses data scarcity and enables continual improvement:

• Continual Pretraining (CT): Large-scale corpus pretraining on perception, grounding, and reasoning data.
• Supervised Fine-Tuning (SFT): Selective training on high-quality, filtered trajectories obtained via interactive online annotation, rule-based heuristics, and human review.
• Reinforcement Learning (RL) (UI-TARS-2): Multi-turn asynchronous rollouts with environment context preserved across steps; PPO with reward shaping based on task verifiers or ORM when deterministic evaluation is unavailable.
• Direct Preference Optimization (DPO): Optimization on error-corrected vs. errorful pairs with a Bradley–Terry likelihood objective and analytical RL policy for improved preference learning.

The data flywheel in UI-TARS-2 operates in three stages: CT, SFT, and RL cycles, with a validator routing verifiable high-quality trajectories into SFT. Rejection sampling and self-judged filtering continually improve both data and agent quality (Qin et al., 21 Jan 2025, Wang et al., 2 Sep 2025).

4. Hybrid GUI Environments and Sandbox Implementation

UI-TARS-2 extends beyond the GUI-only paradigm by integrating a hybrid environment:

• Cloud VM GUI Sandbox: Automated and scalable testing across Windows, Ubuntu, and Android, leveraging PyAutoGUI and ADB with a unified SDK for session allocation, resource cleanup, and remote debugging.
• Shared File System: Agents can chain GUI (e.g., download via browser) and system-level operations (e.g., shell commands) in a seamless workflow.
• Hardware-Accelerated Browser Sandbox: Enables high-throughput reinforcement learning in HTML5/WebGL mini-games, with headless browser instances, GPU-accelerated observations, and full Gym-like APIs.
• Concurrency and Scaling: Infrastructure supports thousands of parallel sessions for large-scale rollout and evaluation, with checkpointing and replay for reproducibility (Wang et al., 2 Sep 2025).

5. Mathematical and Algorithmic Foundations

UI-TARS-2's training is underpinned by multi-turn RL objectives with several stabilizing enhancements:

• Main RL Objective:

$a_t$0

with policy gradients:

$a_t$1

• PPO Clipped Objective:

$a_t$2

with reward shaping, decoupled and length-adaptive GAE, Monte Carlo-based value pretraining, and asymmetric clipping bounds.

• Auxiliary Losses: MSE for value fitting, cross-entropy for policy cloning.

Empirical tuning—e.g., decoupling λ_policy and λ_critic, length adaptation, "Clip Higher" exploration—achieves stable training on long-horizon, multi-step environments susceptible to value drift and critic bias (Wang et al., 2 Sep 2025).

6. Empirical Performance and Generalization

UI-TARS and UI-TARS-2 demonstrate strong empirical results on a range of standard and custom benchmarks. Notable metrics include:

Benchmark	UI-TARS (Max)	UI-TARS-2 (Max)	Claude	OpenAI o3
OSWorld	24.6%	47.5%	22.0%	—
WindowsAgentArena	—	50.6%	—	—
AndroidWorld	46.6%	73.3%	—	34.5%
Online-Mind2Web	—	88.2%	—	—
LMGame-Bench (mean norm)	—	59.8%	—	—

UI-TARS-2 outperforms UI-TARS-1.5, Claude-4, and OpenAI agents on both computer and mobile GUI tasks and remains competitive with proprietary baselines in novel game settings. Enabling SDK extensions improves browser multi-hop tasks and information-seeking abilities (e.g., BrowseComp-en from 7.0% to 29.6%) and supports software engineering workflows (TerminalBench 45.3%, SWE-Bench 68.7%) (Qin et al., 21 Jan 2025, Wang et al., 2 Sep 2025).

7. Training Dynamics, Ablations, and Future Prospects

In-depth analysis highlights:

• Convergence and Exploration: Reward curves exhibit smooth monotonic growth; entropy increases with policy exploration during RL.
• Reasoning Efficiency: Average "think" tokens per step decrease over training in GUI domains, oscillate in games, and environment rounds required per task shrink, reflecting increasing agent efficiency.
• Training Enhancements: Value pretraining delivers higher final rewards and accuracy; ablations confirm that PPO, decoupled/length-adaptive GAE, asymmetric clipping, and hybrid RL each improve robustness and final performance.
• Resource Efficiency: W4A8 quantization boosts throughput (29.6 to 47 tokens/s) with minor accuracy tradeoff.
• Parameter Merging: Interpolating across specialty-trained vertical agents yields merged models with near-specialist proficiency across domains.

This suggests the method is extensible to settings demanding rapid adaptation, active lifelong learning, and multi-agent collaboration. Research directions include scaling perception to more diverse GUI forms, fine-grained affordance reasoning, and new safety/trust benchmarks for open-ended interaction (Qin et al., 21 Jan 2025, Wang et al., 2 Sep 2025).