Uni-GUI: Unified Interface Framework

• Uni-GUI is a unification design pattern that standardizes diverse GUI implementations across robotics and agent learning systems.
• It leverages declarative configuration and metadata-driven specifications to enable interface reuse and cross-platform consistency.
• Uni-GUI supports applications from humanoid teleoperation to GUI-agent learning, improving usability and simplifying maintenance.

Uni-GUI is a recurrent unification idea in the GUI literature, but not a single canonical artifact. In recent arXiv work, the name is used for at least three distinct objects: a web-based teleoperation interface for humanoid robots, an open and reconfigurable web interface for complex ROS-based robotic systems, and a high-quality cross-platform GUI interaction dataset for GUI-agent learning. Around these explicit usages, closely related work pursues the same underlying objective: replacing bespoke, per-tool interfaces or fragmented supervision with a common interaction layer, common representation, or common action substrate across heterogeneous software and devices (Barret et al., 15 Oct 2025, Fresnillo et al., 2024, Lian et al., 5 Jul 2026).

1. Meanings and scope

In the cited literature, “Uni-GUI” refers to explicit systems in robotics and to a dataset for multimodal GUI agents, while older and adjacent work instantiates the same unification principle under different names.

Usage	Domain	Definition
Uni-GUI	Humanoid teleoperation	A web-based GUI for non-expert operation of a kid-size humanoid robot in the FIRA HuroCup obstacle-run setting (Barret et al., 15 Oct 2025)
Uni-GUI	ROS systems	An open and reconfigurable web UI for controlling, monitoring, and configuring complex ROS-based robotic systems (Fresnillo et al., 2024)
Uni-GUI	GUI-agent learning	A cleaned, normalized dataset of executable desktop and mobile GUI trajectories used in UI-MOPD (Lian et al., 5 Jul 2026)

This multiplicity is consistent with earlier work on generic GUI frameworks. GuiLiner defines a generic, configurable front-end for scientific command-line programs driven by XML metadata; EasyInterface provides a common web and Eclipse-oriented frontend for research prototype tools through declarative wrapping and an XML output language; X3DUI proposes a reusable widget library for X3D/Web3D, replacing ad hoc construction from primitives with a standardized GUI layer (0806.0314, Doménech et al., 2017, Sopin et al., 2019).

A plausible implication is that “Uni-GUI” is best understood not as a single software lineage, but as a design pattern centered on interface reuse, declarative configuration, and cross-application consistency. The later GUI-agent literature extends that pattern from human-facing frontends to unified data, grounding, and action spaces.

2. Precursor frameworks for reusable and declarative GUIs

A major antecedent of the Uni-GUI idea is the generic wrapper around existing tools. GuiLiner targets scientific analysis and simulation software that already expose a command-line interface. It reads an XML configuration file containing program information, options, documentation, and display details, then builds a Java-based GUI with a color-coded option tree, an option pane, integrated documentation, a Preview function, a RUN PROGRAM button, and separate stdout and stderr panels. Its central purpose is to preserve CLI strengths such as scripting, batch use, portability, and preservation of execution settings while presenting a point-and-click interface for option specification and execution (0806.0314).

EasyInterface pushes the same principle toward research prototypes and multi-client deployment. A tool executable from the command line and writing to standard output can be exposed rapidly through a web interface or within Eclipse. The server side uses declarative XML configuration files specifying execution templates, parameters, examples, and output handling; the client side includes a web interface, an Eclipse IDE plug-in, and a remote command-line shell. Its XML-based output language supports richer behaviors than plain console text, including syntax highlighting, markers, dialog boxes, click-triggered actions, downloadable outputs, and streaming/background output. This is a stronger form of unification than mere wrapping, because the same tool descriptions and output conventions are intended to drive multiple frontends consistently (Doménech et al., 2017).

X3DUI generalizes unification into a domain-specific widget library. Rather than wrapping external executables, it standardizes GUI construction inside X3D/Web3D through 27 prototypes in four categories—System, Visual, Group, and Layout. Architectural elements such as the singleton Display, the global Settings prototype, and the LayoutManager formalize windowing, focus management, look-and-feel control, and adaptive layout. The library emphasizes a 2D or 2.5D HUD-like layer over 3D scenes, conventional widgets, cross-compatibility, and reusable containers such as Panel, TabPanel, Frame, and TaskBar. In Uni-GUI terms, this is a library-level rather than wrapper-level unification strategy (Sopin et al., 2019).

Taken together, these works establish three persistent properties of unified GUI systems: declarative interface generation, reuse across multiple tools or scenes, and a stable interaction vocabulary that lowers the cost of maintenance.

3. Uni-GUI as a human-facing robotics interface

One explicit use of the name appears in humanoid teleoperation. In that setting, Uni-GUI is a browser-based frontend intended for non-expert operators controlling a kid-size humanoid robot through a FIRA-regulated obstacle course. The motivation is the inadequacy of a legacy interface that was cluttered, weakly labeled, and incomplete in motion support. The new interface is designed to be simple, intuitive, scalable, responsive, and competition-ready, with the live camera feed moved to the forefront and separated from controls. It supports forward/backward walking, turning, lateral shifting, body shifting, and later-added actions such as crawl, get-up or self-stand, reset, and start position. Familiar keyboard bindings—WASD for robot movement and arrow keys for camera movement—reduce learning burden, while adjustable motion coefficients allow walking step length and turn angle to be changed on the fly and persisted in browser session storage until a command is issued. The stack is web-based, with HTML, CSS, JavaScript, optional jQuery, and mjpegcanvas.js on the frontend, and ROSbridge, roslib.js, and JSON messages connecting the browser to ROS1-based robot control on Ubuntu Mint 16.04 with Python 2.7 and C++ backends (Barret et al., 15 Oct 2025).

A second explicit use of the name appears in industrial ROS systems. Here Uni-GUI is an open and reconfigurable Single-Page Application implemented with HTML, CSS, JavaScript, and Vue.js, communicating with ROS through ROSBridge and roslibjs. Its structural shell comprises a side menu, a top bar, and a page content area, but the main emphasis is feature modularity and reconfiguration. The initial platform includes multirole security clearance, dynamic module launching and monitoring, sensor data visualization, robot manual motion control, robot automatic control and monitoring, video stream visualization, system and process configuration, trajectory/routine recording and testing, alarms visualization, and databases update. Manual motion control exposes predefined configurations, relative Cartesian motion, and absolute Cartesian motion; sensor visualization uses Chart.js; automatic control relies on an action server/action client backend split; alarms expose bidirectional access to safety status; and deployment was demonstrated across four industrial use cases involving different robot combinations, vendors, and UI devices (Fresnillo et al., 2024).

The two robotic Uni-GUI systems share several architectural motifs despite different scopes. Both are web-based, browser-accessible, and layered over ROS rather than replacing it. Both prioritize reduced cognitive load through interface organization, and both retain a strong backend dependency: the humanoid teleoperation system depends on robot-specific motion and camera streams, while the industrial ROS system requires matching topics, services, and auxiliary nodes. Neither paper reports a formal controlled user study; evaluation is qualitative in the humanoid case and deployment-based with positive feedback in the ROS case.

4. Uni-GUI as a cross-platform GUI-agent dataset

In UI-MOPD, Uni-GUI is not a frontend at all, but a data substrate for continual GUI-agent learning. It is described as a high-quality cross-platform graphical user interface interaction dataset providing a data foundation for learning executable trajectories across desktop and mobile environments. The dataset is built to address sparse high-quality cross-platform trajectories, platform coverage limits, behavioral pattern mixing, catastrophic forgetting, and noise in raw trajectories. Its construction pipeline has four stages: query generation, trajectory collection, trajectory cleaning, and post-processing. Desktop query generation is assisted by Kimi-K2.6 in OSWorld-style environments, while mobile query generation is assisted by Gemini-3.1-Pro in MobileWorld and AndroidWorld environments. Collected trajectories record observations, actions, intermediate reasoning, and task states, but actions remain in their native platform form during collection (Lian et al., 5 Jul 2026).

The cleaning stage removes malformed step structures, trajectories whose actions cannot be mapped to the student model’s action space, trajectories longer than 40 steps, and trajectories whose queries are inconsistent with the environment or unsupported by the extracted functional points. Gemini-3.1-Pro is then used as an automatic judge to retain only successful trajectories by decomposing a task into ordered sub-tasks and checking completion step by step. Post-processing rewrites reasoning traces into a structured chain-of-thought format, re-annotates grounding bounding boxes, and normalizes trajectories into consistent tool-call/action representations compatible with Qwen-VL-based policies. The appendix reports approximate composition as ~95K steps, ~7K trajectories for desktop self-collected data, ~13K steps, ~0.8K trajectories for cleaned OpenCUA, ~17K steps, ~1K trajectories for mobile self-collected data, and ~35K steps, ~2.7K trajectories for cleaned OpenMobile, totaling ~160K interaction steps and ~11.5K trajectories (Lian et al., 5 Jul 2026).

A central technical point is that unification does not collapse platform semantics. Desktop uses a computer_use interface with actions such as key, type, mouse_move, left_click, left_click_drag, right_click, middle_click, double_click, triple_click, scroll, wait, and terminate, whereas mobile uses a mobile_use interface with click, long_press, swipe, type, answer, system_button, wait, ask_user, and terminate. This preserves platform-specific conventions while still enabling common logging, cleaning, and downstream training. In Stage 1, Uni-GUI supports SFT of platform-specific teachers from Qwen3-VL-32B-Thinking; in Stage 2, a Qwen3-VL-8B-Thinking student is trained with reinforcement learning and platform-conditioned multi-teacher on-policy distillation. Reported interactive results are 38.2% on OSWorld and 12.0% on MobileWorld, compared with a base model at 33.9% / 7.7%, and the paper explicitly frames these gains as improved cross-platform retention and adaptation rather than mere specialization (Lian et al., 5 Jul 2026).

A common misconception is that a unified GUI dataset should force a single identical action space across all environments. Uni-GUI in UI-MOPD does the opposite during collection: it preserves native desktop and mobile action forms, then uses platform-specific teachers as stable behavioral anchors. The paper’s own unified-training analysis also shows that joint optimization remains difficult in heterogeneous GUI environments.

5. Unified GUI search, representation, grounding, and action

Several adjacent systems supply the perception and representation layers that a broader Uni-GUI agenda would require. Gallery D.C. constructs a large-scale searchable gallery of GUI components mined from real Android apps and app-store introduction screenshots. Its pipeline combines runtime screenshots annotated through automated GUI exploration and UI Automator with 469k app introduction screenshots from 126k Android apps, then applies screenshot augmentation and a Faster R-CNN detector to recover component type, position, width, and height. Indexed attributes include component class/type, application category, height, width, primary color, and text content, enabling search and higher-order “design knowledge discovery.” The detector achieved 0.62 recall, 0.76 precision, and 0.51 mAP on runtime screenshots and 0.53 recall, 0.84 precision, and 0.62 mAP on manually annotated app-introduction screenshots, with screenshot augmentation improving decomposition on app-introduction screenshots by 43.2% in recall, 5% in precision, and 44.2% in mAP (Feng et al., 2022).

Screen2Vec addresses a different layer: unified semantic representation. It learns embeddings for GUI components and full screens through a two-layer architecture inspired by Word2Vec, combining component text, component class, visual layout, app-store description, and interaction-trace context. Component embeddings are 768-dimensional; the final screen representation in the described configuration is 1536-dimensional. The method is self-supervised and evaluated through screen prediction, nearest-neighbor retrieval, composability, and task representation. In a Mechanical Turk nearest-neighbor study, average similarity ratings were 3.295 for Screen2Vec, 3.014 for TextOnly, and 2.410 for LayoutOnly; for task representation across 10 common smartphone tasks with two app variants each, it retrieved the matching task variant with 18/20 = 90% accuracy, compared with 70% for a TextOnly representation (Li et al., 2021).

Grounding-focused GUI agents then operationalize unification at interaction time. SeeClick is a screenshot-only visual GUI agent built by continuing pre-training on Qwen-VL-Chat with GUI-centric data, centered on the claim that GUI grounding is the foundation of visual GUI agents. It introduces the ScreenSpot benchmark spanning mobile, desktop, and web with over 600 screenshots and 1200+ instructions, and reaches 53.4% average click accuracy on ScreenSpot, above CogAgent’s 47.4%, while showing that grounding improvements correlate with better performance on MiniWob, AITW, and Mind2Web (Cheng et al., 2024). UIPro extends this trajectory by combining a 20.6 million sample GUI understanding dataset, a unified action space for mobile, web, and desktop, and continued fine-tuning on merged multi-source trajectories; it reports 70.4 overall Step SR on AITW for UIPro-Qwen2VL and strong scores across GUIAct, Multimodal-Mind2Web, ScreenSpot, ScreenSpot-v2, MOTIF, RefExp, and VisualWebBench (Li et al., 22 Sep 2025). POINTS-GUI-G, although not named Uni-GUI, pushes unified grounding through coordinate normalization to [0,1], dataset-format unification, difficulty grading via Layout Entropy, and RL with verifiable rewards, reaching 59.9 on ScreenSpot-Pro, 66.0 on OSWorld-G, 95.7 on ScreenSpot-v2, and 49.9 on UI-Vision (Zhao et al., 6 Feb 2026).

These works suggest that a full Uni-GUI stack has at least four layers: searchable component corpora, unified screen/component embeddings, robust screenshot grounding, and executable cross-platform action prediction.

6. Architectural patterns, misconceptions, and limitations

Across the literature, unification is achieved through a small number of recurring mechanisms. One is declarative or metadata-driven specification: GuiLiner uses XML to describe options and documentation; EasyInterface uses XML for execution templates and client-side GUI commands; X3DUI centralizes appearance and layout through Settings and LayoutManager; UI-MOPD normalizes trajectories into consistent tool-call/action representations; UIPro serializes actions as JSON in a unified action space; and POINTS-GUI-G normalizes coordinates and task formats (0806.0314, Doménech et al., 2017, Sopin et al., 2019, Lian et al., 5 Jul 2026, Li et al., 22 Sep 2025, Zhao et al., 6 Feb 2026).

A second pattern is that unification usually coexists with backend heterogeneity rather than eliminating it. The industrial ROS Uni-GUI is explicitly a frontend only and requires corresponding ROS topics, services, and backend nodes; the humanoid teleoperation Uni-GUI relies on ROSbridge, roslib.js, and robot-specific execution nodes; SeeClick’s unified screenshot interface still struggles when jointly trained across heterogeneous environments; and UI-MOPD explicitly preserves platform-specific action semantics and teacher policies rather than averaging them away (Fresnillo et al., 2024, Barret et al., 15 Oct 2025, Cheng et al., 2024, Lian et al., 5 Jul 2026).

Several limitations recur. GuiLiner is not designed for interactive display of program output, focusing instead on option setting and execution (0806.0314). X3DUI reports “no actual assessment” and depends on proprietary nodes/functions supported by a particular X3D player (Sopin et al., 2019). The humanoid teleoperation Uni-GUI reports no formal human-subject evaluation and depends heavily on camera and internal logs because external sensors are unavailable (Barret et al., 15 Oct 2025). The industrial ROS Uni-GUI reports positive deployment feedback but no task-timing study or quantitative usability score (Fresnillo et al., 2024). In GUI-agent work, both SeeClick and UI-MOPD report that unified training across heterogeneous environments remains weaker than some separately trained variants, indicating that “unified” does not yet mean fully solved cross-platform generalization (Cheng et al., 2024, Lian et al., 5 Jul 2026).

The most precise synthesis is therefore narrow but robust: Uni-GUI names specific systems and datasets, yet also denotes a broader methodological orientation toward reusable interface layers, normalized representations, and common interaction substrates. In human-facing settings, it lowers the barrier to operating scientific or robotic systems without discarding existing command-line or ROS infrastructures. In agentic settings, it supplies the cleaned trajectories, grounding pre-training, and action harmonization needed for cross-platform screenshot-based interaction. The literature consistently treats these as complementary, not interchangeable, interpretations of GUI unification.