UI-Venus-1.5: Unified GUI Agent for Web and Mobile

• UI-Venus-1.5 is a unified GUI agent that integrates grounding, web, and mobile navigation into a single model using a diverse reinforcement learning framework.
• It employs a four-stage training curriculum—mid-training, offline RL, online RL, and model merging—to achieve state-of-the-art performance on multiple interface benchmarks.
• The system uses a unified XML-like action schema and normalized coordinate representation, ensuring robust, cross-device deployment in dynamic digital environments.

Searching arXiv for the cited UI-Venus papers and closely related work. arxiv_search(query="UI-Venus-1.5 Technical Report", max_results=5) UI-Venus-1.5 is a unified, end-to-end GUI agent family for screenshot-based interaction with digital interfaces, introduced as a successor to the earlier UI-Venus line and designed for both grounding and long-horizon navigation across mobile and web environments. The system is built on the Qwen3-VL family, released in dense 2B and 8B variants and a 30B-A3B mixture-of-experts variant, and combines mid-training on large-scale GUI corpora, offline reinforcement learning, online reinforcement learning with full-trajectory rollouts, and cross-domain model merging into a single checkpoint (Veuns-Team et al., 9 Feb 2026).

1. Origin, scope, and relation to earlier UI-Venus models

UI-Venus-1.5 was introduced as a unified GUI agent intended for robust real-world applications, in contrast to the earlier UI-Venus release, which emphasized separate series models for grounding and navigation built on Qwen2.5-VL and trained via reinforcement finetuning (Gu et al., 14 Aug 2025). The 1.5 report identifies three central additions relative to the previous version: a Mid-Training stage using 10 billion tokens across 30+ datasets, Online Reinforcement Learning with full-trajectory rollouts, and a single unified GUI agent produced through Model Merging (Veuns-Team et al., 9 Feb 2026).

The term “unified” is technically important. In the earlier UI-Venus system, the report explicitly separated UI-Venus-Ground and UI-Venus-Navi, partly because mixed reward structures destabilized training and because grounding benefited from “no-think” inference while navigation benefited from explicit reasoning traces (Gu et al., 14 Aug 2025). UI-Venus-1.5 instead merges grounding, web, and mobile domain-specialized policies into one model family. This suggests a shift from task-specific specialization toward a consolidated checkpoint intended to preserve domain competence while improving deployment simplicity.

The scope of the 1.5 report is broader than benchmark performance alone. It covers architecture, data construction, training curricula, online environment design, merging methodology, and real-world execution on Chinese mobile apps. At the same time, several implementation details remain undisclosed, including optimizer schedules, hardware compute budgets, and the precise internal MoE routing configuration.

2. Model family, input representation, and action interface

The model family comprises three variants:

Variant	Type	Backbone lineage
UI-Venus-1.5-2B	Dense	Qwen3-VL family
UI-Venus-1.5-8B	Dense	Qwen3-VL family
UI-Venus-1.5-30B-A3B	MoE	Qwen3-VL series

All variants inherit the vision encoder and language backbone from Qwen3-VL, with screenshots encoded by the visual tower and fused with tokenized text in a multimodal transformer (Veuns-Team et al., 9 Feb 2026). Spatial targets are represented in normalized coordinates over $[0,1000]\times[0,1000]$, which provides a resolution-independent coordinate convention across devices and screen geometries.

The agent is visually grounded rather than DOM-driven. The report states that it operates with screenshots, text instruction, and prior action history, and that for AndroidLab it used only screenshots and no XML while still outperforming several XML-augmented baselines (Veuns-Team et al., 9 Feb 2026). This places UI-Venus-1.5 in the class of native screenshot-only GUI agents rather than tool-augmented planners relying on accessibility trees or document structures.

Action generation is performed by a single generative decoder using a strict XML-like schema: $[x,y]$6 The schema is enforced in training through format rewards and at inference through parsing. The unified action space spans both mobile and web usage, including Click, DoubleClick, Hover, LongPress, Drag, Scroll, Type, Hotkey, PressBack, PressHome, PressEnter, PressRecent, Launch, Wait, Finished, and CallUser (Veuns-Team et al., 9 Feb 2026). Mobile scrolling requires explicit start and end coordinates, whereas web scrolling is direction-only. Grounding outputs are represented as $[x,y]$ or [-1, -1] when refusal is appropriate.

For the MoE variant, the report adopts a generic top-$K$ mixture form,

$$y=\sum_{k\in \mathrm{TopK}(G(x))} p_k f_k(x), \qquad p=\mathrm{softmax}(G(x)),$$

but does not disclose the number of experts, capacity factor, or detailed routing policy (Veuns-Team et al., 9 Feb 2026). A plausible implication is that the report treats the Qwen3-VL A3B MoE implementation largely as inherited infrastructure rather than as a novel contribution of UI-Venus-1.5 itself.

3. Training curriculum and optimization objectives

The training pipeline is organized as a four-stage curriculum: Mid-Training, Offline RL, Online RL, and Model Merging (Veuns-Team et al., 9 Feb 2026). This stage-wise design is central to the system’s identity.

Mid-Training is a supervised phase over 10 billion tokens from 30+ datasets, including Mind2Web, ShowUI, and AITW. The corpus is functionally stratified: semantic perception contributes approximately 20.8%, GUI-VQA 22.1%, and grounding 24.8%, with additional hybrid navigation and reasoning data. The objective is standard supervised cross-entropy,

$$L_{\mathrm{sup}}=-\sum_{(x,y)} \log p_\theta(y\mid x).$$

The report attributes to this phase the acquisition of foundational GUI semantics and simple navigation skill. Representationally, post-Mid-Training GUI clusters become more separable, with Silhouette Score increasing from 0.235 to 0.315, a reported gain of 34.0% (Veuns-Team et al., 9 Feb 2026).

Data refinement is itself a substantial methodological component. Qwen3-VL-235B-A22B is used to score traces from 0 to 10; traces scored $\ge 7$ are treated as gold, those scored 4–6 are rewritten, and those scored 0–3 are discarded or reconstructed. Manual audits show the high-fidelity proportion improving from 69.7% to 89.7%. In a real-device data generation loop, success rates reportedly rise from 17.9% to more than 70%, ultimately yielding more than 30,000 verified interaction traces (Veuns-Team et al., 9 Feb 2026).

Offline RL then specializes the model into grounding, mobile navigation, and web navigation policies. For grounding, the reward is

$R = w_1 R_{\mathrm{format}} + w_2 R_{\mathrm{point\mbox{-}box}},$

and refusal behavior is explicitly trained through the [-1, -1] target when the referred element is absent (Veuns-Team et al., 9 Feb 2026). For navigation, the reward combines format and action terms,

$$R = w_1 R_{\mathrm{format}} + w_2 R_{\mathrm{action}},$$

where $R_{\mathrm{action}}$ includes action-type correctness, token-level F1 for text entry, and coordinate tolerances for click and drag actions.

A key diagnosis of the report is the “step–trace mismatch”: step-level rewards continue rising while full-trace success peaks and then degrades under Offline RL (Veuns-Team et al., 9 Feb 2026). This observation motivates the transition to trajectory-level online optimization.

4. Online reinforcement learning and model merging

Online RL is conducted in a Device-as-a-Service environment with a Group Control Gateway and a Unified Client SDK integrating Android via ADB, Chrome via CDP, and Linux via SSH (Veuns-Team et al., 9 Feb 2026). The report describes thousands of heterogeneous devices, millisecond-level allocation, zero-copy I/O, support for hundreds to thousands of concurrent devices, and millions of daily operations. This infrastructure is not ancillary; it is the substrate enabling full-trajectory rollouts in dynamic environments.

Task generation uses a mixed pool built from static LLM-derived libraries over apps and websites and dynamic screenshot-to-goal synthesis. Novel tasks are retained under a deduplication criterion

$$T_{\mathrm{new}}=\{q' \mid \psi(q',T_{\mathrm{pool}})<\epsilon\},$$

and training uses a step-count curriculum with Easy, Medium, and Hard buckets (Veuns-Team et al., 9 Feb 2026).

The trajectory reward is defined as

$[x,y]$0

where the length-decay factor penalizes detours relative to the shortest rollout in a sampled group and unparseable actions incur a penalty $[x,y]$1 (Veuns-Team et al., 9 Feb 2026). Success is verified either by rules, such as URL or system API checks, or by an MLLM-as-a-judge using the final screenshot and the goal.

Optimization uses Group Relative Policy Optimization. The report gives the trajectory-level GRPO objective

$[x,y]$2

with

$[x,y]$3

Adaptive KL regularization to a reference policy and annealed entropy are added for stability (Veuns-Team et al., 9 Feb 2026). Unlike token-local supervision, the advantage is constant within a trajectory, which the report argues is suitable for sparse and delayed rewards in long-horizon GUI tasks.

After task-specific fine-tuning, the specialized policies are merged. The report considers linear merge,

$[x,y]$4

and TIES-Merge, which computes task vectors relative to a base model, prunes small-magnitude updates, resolves sign conflicts, and aggregates aligned parameter changes. Empirically, TIES-Merge is favored. In the 30B-A3B ablation, pre-merge performance is 71.0% on ScreenSpot-Pro and 75.5% on AndroidWorld, whereas post-TIES-Merge performance becomes 69.6% and 77.6%, respectively; linear merge produces larger losses on both metrics (Veuns-Team et al., 9 Feb 2026).

5. Benchmark performance and empirical scaling

UI-Venus-1.5 reports state-of-the-art or near-state-of-the-art results on a range of grounding and navigation benchmarks (Veuns-Team et al., 9 Feb 2026).

For grounding, the 30B-A3B model reports 75.0% on VenusBench-GD, 69.6% on ScreenSpot-Pro, 96.2% on ScreenSpot-V2, 88.6% on MMBench-GUI L2, 76.4% on OSWorld-G-Refine, 70.6% on OSWorld-G, and 54.7% on UI-Vision. The report states that ScreenSpot-Pro, VenusBench-GD, OSWorld-G-Refine, OSWorld-G, and UI-Vision are state-of-the-art, while ScreenSpot-V2 is second best, 0.3 percentage points behind MAI-UI-32B. Scaling on ScreenSpot-Pro is monotonic across sizes, from 57.7% at 2B to 68.4% at 8B and 69.6% at 30B-A3B.

For navigation, the 30B-A3B variant reports 77.6% on AndroidWorld, 52.9% official and 68.1% human-verified on AndroidLab, 21.5% on VenusBench-Mobile, and 76.0% on WebVoyager. The AndroidWorld result exceeds MAI-UI-32B by 4.3 points, according to the report. The 8B model reaches 73.7% on AndroidWorld and 55.1% official on AndroidLab, while the 2B model reaches 55.6% and 36.2%, respectively (Veuns-Team et al., 9 Feb 2026).

The ablation study ties these outcomes directly to the training pipeline. Mid-Training followed by Offline RL yields gains of approximately 6–7% on ScreenSpot-Pro and up to 6.5% on AndroidWorld for the 8B model. The transition from Offline RL to Online RL adds 14.5% on AndroidWorld for the 2B model, 9.2% for the 8B model, and 7.5% for the 30B-A3B model. This pattern supports the report’s claim that trajectory-wise Online RL addresses the degradation associated with step-level optimization (Veuns-Team et al., 9 Feb 2026).

One evaluation caveat is explicitly noted for AndroidLab: the report attributes discrepancies between official scores and human-verified scores to issues in the official evaluation scripts. No confidence intervals are reported.

6. Deployment characteristics, limitations, and significance

Post-merge, UI-Venus-1.5 is deployed as a single checkpoint taking a screenshot, a user instruction, and a brief action history, then generating structured actions for execution (Veuns-Team et al., 9 Feb 2026). The coordinate normalization to $[x,y]$5 simplifies device interoperability, and the refusal mechanism in grounding reduces hallucinated element selections when targets are absent. The report also emphasizes that no external OCR or parser dependency is required at inference.

The system is reported to operate across 40+ Chinese mobile apps in real settings. The paper illustrates this with a train-ticket purchasing scenario involving Launch, coordinate-based Click, Type, Scroll, and Finished actions. This example is not a new benchmark; rather, it demonstrates the intended real-world operating mode of the unified agent.

Several limitations are identified. Fine-grained misclicks on dense interfaces remain a failure mode, as do over-scrolling and occasional divergence on very long horizons. Long-horizon failures remain present in the hardest mobile and web tasks. More fundamentally, the report does not disclose optimizer schedules, compute-days, or detailed MoE internals, which constrains full reproducibility (Veuns-Team et al., 9 Feb 2026).

In the lineage of screenshot-based GUI agents, UI-Venus-1.5 is significant less for introducing a new action formalism than for integrating four ingredients into a single system: large-scale GUI mid-training, trace-level online RL on real devices, refusal-aware grounding, and cross-domain model merging. Compared with the earlier UI-Venus architecture, which separated grounding and navigation and relied on Qwen2.5-VL with GRPO-style reinforcement finetuning, UI-Venus-1.5 advances toward a more consolidated and deployment-oriented agent paradigm (Gu et al., 14 Aug 2025). A plausible implication is that subsequent work will focus less on isolated task specialization and more on unified policy construction, richer memory mechanisms, and more explicit control of long-horizon planning risk.