---
title: ByteDance UI-TARS System
url: https://www.emergentmind.com/topics/bytedance-ui-tars-system
type: topic
---

# ByteDance UI-TARS System

ByteDance’s UI-TARS System is an end-to-end native agent framework designed for autonomous interaction with graphical user interfaces (GUIs), unifying visual perception, reasoning, action, and memory within a single policy. The system, comprising UI-TARS and its successor UI-TARS-2, represents a major milestone in scalable, multi-turn reinforcement learning (RL) for GUI agents. It departs from prompt-centric, commercial model wrappers, instead relying on a Mixture-of-Experts Transformer backbone and a vision-language model that directly processes screenshots and GUI traces. The technical architecture leverages a unified action space, advanced System-2 reasoning, and iterative data generation through reflection and reinforcement, targeting robust performance across desktop, mobile, and web platforms [2509.02544][2501.12326].

## 1. Architectural Foundations and Unified Policy Loop

UI-TARS-2 deploys a native agent formulation built on a Mixture-of-Experts Transformer that incorporates a 532M-parameter vision encoder. The policy $\pi_\theta$ operates over multimodal observations, memory, and task instructions:
\[
\pi_\theta(r_t,a_t \mid \mathrm{instr},\mathcal W_t,o_t,\mathcal E_t)
\]
where each timestep $t$ includes an internal reasoning token $r_t$, action $a_t$ (GUI or SDK operation), and new observation $o_{t+1}$, with joint trajectory
\[
\tau = \{(r_0,a_0,o_1), (r_1,a_1,o_2), \ldots, (r_T,a_T,o_{T+1})\}
\]
Memory consists of high-fidelity working memory $\mathcal W_t$ (last $N$ steps) and episodic memory $\mathcal E_t$ (compressed summaries of past episodes), enabling persistent context across long-horizon interactions. This architecture obviates modular action-perception pipelines, supporting unified end-to-end learning for perception, action grounding, and system-level reasoning [2509.02544].

## 2. Data Flywheel and Iterative Trace Bootstrapping

UI-TARS introduces an iterative data flywheel, comprising continual pre-training (CT) on diverse corpora, supervised fine-tuning (SFT) from high-quality human and LLM-annotated traces, and reinforcement learning (RL) on verifiable tasks. After each training cycle $t$, new trajectories are validated:
- High-quality samples ($V(s) = 1$) enter $D_{SFT}^{(t+1)}$
- Lower-quality samples are appended to $D_{CT}^{(t+1)}$

This mechanism ensures that
\[
P(V(s) = 1 \mid M^{(t)}) > P(V(s) = 1 \mid M^{(t-1)})
\]
enabling accelerated capability gains. UI-TARS further employs reflective online trace bootstrapping by deploying agents on large virtual device pools, multi-stage filtering (heuristics, VLM scoring, human review), and error correction through direct preference optimization (DPO), where both positive and negative action traces are used for fine-tuned RL [2501.12326][2509.02544].

## 3. System-2 Reasoning and Thought-Augmented Planning

Both UI-TARS and UI-TARS-2 embed explicit System-2 reasoning mechanisms to support complex multi-step decision making. Intermediate "thought" tokens $t_i$ are generated to encode reasoning patterns such as task decomposition, milestone recognition, reflection, and trial-and-error. For UI-TARS:
\[
P(t_n, a_n \mid \mathrm{instr}, (o_{n-N}, t_{n-N}, a_{n-N}), \ldots, o_n)
\]
The ActRe annotation protocol generates thoughts causally aligned to actions:
\[
t_n = \mathrm{VLM}(\mathrm{instr}, (o_1, t_1, a_1), \ldots, o_n, a_n)
\]
During thought bootstrapping, candidate thought–action pairs are sampled from an early checkpoint, selecting those matching the gold action:
\[
\{(\hat t_{n,i}, \hat a_{n,i})\}_{i=1}^K = \mathrm{UI\text{-}TARS}_{\mathrm{early}}(\ldots),\quad \hat t_n = \hat t_{n,i^*} \text{ s.t. } \hat a_{n,i^*} = a_n
\]
This explicit reasoning is foundational for long-horizon generalization, error correction, and adaptive planning [2501.12326][2509.02544].

## 4. Reinforcement Learning and Stability Mechanisms

UI-TARS-2 incorporates a stabilized multi-turn RL framework, centered on Proximal Policy Optimization (PPO) with several enhancements:
- Decoupled-GAE: independent $\lambda$ for policy and value functions to manage bias/variance
- Length-Adaptive GAE: $\lambda_{\mathrm{policy}} = 1 - \frac{1}{\alpha \ell}$, where $\alpha = 0.05$
- Value Pretraining: offline convergence with $\lambda = 1.0$ using Monte Carlo targets
- Clip Higher: distinct upper/lower $\varepsilon$ for PPO clipping to encourage exploration
- Reward shaping and length/format penalties

Asynchronous rollouts, streaming training pools, and stateful environments further stabilize RL in long-horizon scenarios. Analysis demonstrates steady reward and entropy ascent, reduction in reasoning tokens for GUI tasks, efficient interaction rounds, and effective value pretraining. Quantization (W4A8) achieves a $\sim$40% latency reduction (from 4.0s to 2.5s) with minimal accuracy loss [2509.02544].

## 5. Hybrid GUI-SDK Environment and Sandbox Infrastructure

UI-TARS-2’s environment design extends beyond pure GUI interaction by integrating a hybrid GUI-SDK backend. Agents can:
- Perform on-screen operations
- Access file systems via shell commands
- Invoke SDK tools (e.g., REST API, MCP tools)
- Utilize embedded development environments (terminals, VS Code/Jupyter previews, proxy URLs)

This enables workflows such as immediate post-download file processing and broadens applicability to software engineering and system administration tasks. The sandbox infrastructure uses:
- Cloud VM Sandbox: orchestrating thousands of Windows/Ubuntu/Android VMs
- Browser Sandbox: deterministic, GPU-accelerated containers for WebGL/HTML5 mini-games

Provisioning supports millions of rollouts with real-time monitoring, auto-recovery, and consistent tool APIs [2509.02544].

## 6. Empirical Performance across Benchmarks

UI-TARS-2 achieves state-of-the-art performance on diverse GUI and game benchmarks, consistently outperforming both prior open-source models and proprietary baselines:

| Model        | OSWorld | WAA  | TB† | SB† | AndroidWorld | Online-Mind2Web | BC-zh        | BC-en        |
|--------------|---------|------|-----|-----|--------------|-----------------|--------------|--------------|
| UI-TARS-1.5  | 42.5    | 42.1 | ✗   | ✗   | 64.2         | 75.8            | ✗            | ✗            |
| UI-TARS-2    | 47.5    | 50.6 |45.3†|68.7†| 73.3         | 88.2            | 32.1 (50.5†) | 7.0 (29.6†)  |
| OpenAI o3    | 42.9    |  —   | ✗   | ✗   | 52.5         | 71.0            | —            | —            |
| Claude-4     | 43.9    |  —   |39.2 |72.7 | —            | —               |22.5          | 14.7         |

† Indicates enhanced action space with GUI-SDK. BrowseComp scores improve significantly (e.g., BC-en 7.0→29.6; BC-zh 32.1→50.5) with GUI-SDK extensions.

In the 15-game suite, UI-TARS-2 RL attains mean normalized scores of 59.8 (human = 100), surpassing OpenAI CUA (24.7) and Claude CU (21.6), with several games exceeding 90% human-level. On LMGame-Bench, UI-TARS-2 remains competitive with OpenAI o3 and Gemini 2.5 Pro [2509.02544][2501.12326].

## 7. Generalization Potential and Evolutionary Roadmap

UI-TARS-2 demonstrates robust generalization to long-horizon, open-ended tasks, notably information-seeking (web browsing) and software engineering (shell, code bench tasks). RL fine-tuning elevates Online-Mind2Web accuracy from 83.7% (SFT) to 88.2% (RL); similar uplift observed in OSWorld and AndroidWorld. The system’s hybrid environment and thought-augmented policy contribute to multi-hop, API-free web search and multi-modal system administration capacities.

The UI-TARS roadmap outlines evolution from rule-based and modular architectures to fully native end-to-end agents. Stage 4 research focuses on lifelong adaptation, autonomous task proposal, robust self-supervision for novel UI patterns, and sample-efficient System-2 reasoning. This trajectory suggests continued advancement toward active agents capable of dynamic self-evaluation and adaptation in live computing environments [2501.12326][2509.02544].

Source: https://www.emergentmind.com/topics/bytedance-ui-tars-system