ShowUI-$π$: Flow-based Generative Models as GUI Dexterous Hands

Abstract: Building intelligent agents capable of dexterous manipulation is essential for achieving human-like automation in both robotics and digital environments. However, existing GUI agents rely on discrete click predictions (x,y), which prohibits free-form, closed-loop trajectories (e.g. dragging a progress bar) that require continuous, on-the-fly perception and adjustment. In this work, we develop ShowUI-$π$, the first flow-based generative model as GUI dexterous hand, featuring the following designs: (i) Unified Discrete-Continuous Actions, integrating discrete clicks and continuous drags within a shared model, enabling flexible adaptation across diverse interaction modes; (ii) Flow-based Action Generation for drag modeling, which predicts incremental cursor adjustments from continuous visual observations via a lightweight action expert, ensuring smooth and stable trajectories; (iii) Drag Training data and Benchmark, where we manually collect and synthesize 20K drag trajectories across five domains (e.g. PowerPoint, Adobe Premiere Pro), and introduce ScreenDrag, a benchmark with comprehensive online and offline evaluation protocols for assessing GUI agents' drag capabilities. Our experiments show that proprietary GUI agents still struggle on ScreenDrag (e.g. Operator scores 13.27, and the best Gemini-2.5-CUA reaches 22.18). In contrast, ShowUI-$π$ achieves 26.98 with only 450M parameters, underscoring both the difficulty of the task and the effectiveness of our approach. We hope this work advances GUI agents toward human-like dexterous control in digital world. The code is available at https://github.com/showlab/showui-pi.

• The paper presents a novel flow-based generative model that unifies discrete clicks and continuous drags for enhanced GUI manipulation.
• It introduces ScreenDrag, a benchmark with 20K detailed drag trajectories across five domains to rigorously evaluate digital agents.
• Empirical results show that ShowUI-π outperforms token-based baselines in trajectory fidelity and task success rates through flow-matching and unified modeling.

Flow-based Dexterous GUI Manipulation: ShowUI-$π$

Motivation and Problem Setting

The proliferation of multimodal agents capable of perceiving and interacting with digital interfaces has highlighted the limitations of existing models for GUI automation. Predominantly, GUI agents operate with discrete click-based action representations, constraining them to simple atomic interactions and impeding their ability to execute high degree-of-freedom, visually grounded manipulations such as continuous dragging, free-form drawing, or interactive Captcha solving. This bottleneck precludes closed-loop reasoning and precise, on-the-fly adjustments essential for advanced digital automation.

ShowUI-$π$ advances the state of GUI automation by proposing the first flow-based generative model designed to unify discrete and continuous GUI actions, providing a lightweight (450M parameter) Vision-Language-Action policy capable of generating fine-grained, smooth cursor trajectories from streaming visual observations and complex user instructions (2512.24965). Distinct from token-based models, ShowUI-$π$ supports both click and drag modalities within a single modeling head and precisely orchestrates cursor movements using flow-based trajectory generation.

ScreenDrag Benchmark and Data Pipeline

A major technical contribution is ScreenDrag, a novel benchmark targeting fine-grained, trajectory-centric GUI manipulation. ScreenDrag comprises 20K manually collected and synthetic drag trajectories and an evaluation suite of 505 tasks distributed across five representative domains—PowerPoint, OS Desktop/File Manager, Handwriting on canvas, Adobe Premiere Pro, and Captcha workflows—each necessitating complex spatial control and real-time perceptual feedback.

ScreenDrag's data pipeline integrates UI parsing, automatic instruction generation via LLMs, and trajectory synthesis verified in live software environments. The protocol ensures dense, high-resolution trajectory recording and UI state logging, making it scalable for both supervised and evaluation scenarios.

Figure 1: ScreenDrag automated data collection pipeline, synthesizing dense GUI manipulation data across domains via UI element parsing, LLM-driven task proposal, and executable trajectory synthesis.

A key property of ScreenDrag is its balanced coverage of GUI manipulation modalities (Figure 2), supporting the co-training and benchmarking of agents for both professional and everyday contexts.

Figure 2: The domain and category breakdown of ScreenDrag data distribution, enabling systematic evaluation across varied manipulation tasks.

Evaluation Protocols: Offline and Closed-loop Online Metrics

ShowUI-$π$ is evaluated with complementary offline and online protocols. Offline evaluation quantifies trajectory fidelity by measuring average trajectory error (ATE) and endpoint accuracy across ground-truth trajectories. Online evaluation adopts a closed-loop regime where agents interact with a dynamic environment constructed from screen recordings and state logs, measuring success rates in task completion under realistic, sequential observation-action feedback.

Figure 3: Comparison between stepwise offline and sequential online evaluation pipelines in ScreenDrag, enabling comprehensive agent benchmarking.

Model Architecture

ShowUI-$π$ extends the SmolVLA backbone with critical architectural modifications for real-time GUI control. Notably:

• Unified Discrete-Continuous Head: All actions (clicks and drags) are encoded as sequences of $(x, y, m)$ triplets, where $m$ denotes mouse button state, allowing seamless task adaptation and multi-dataset training.
• Flow-based Action Generation: The cursor trajectory is generated incrementally via a conditional velocity field $\mathbf{v}_\theta$ over a continuous parameter $s$, trained using flow-matching objectives with temporal reweighting for endpoints and directional regularization to enforce robust geometric alignment.

  Figure 4: Model overview illustrating streaming perception, interleaved VLM/action-expert processing, and closed-loop incremental trajectory generation.

  

  Figure 5: ShowUI-$π$ architecture detail: unified LLM head with cross-attention to transformer-based action expert for joint click and drag chunk prediction.

Quantitative Results and Ablations

Empirical evaluation demonstrates that ShowUI-$π$ achieves superior drag manipulation capabilities relative to mainstream proprietary and open-source baselines that use discrete-action tokenization. In ScreenDrag's closed-loop online evaluation, ShowUI-$π$ attains an overall success rate of 26.98%, outperforming Gemini-2.5-CUA (22.18%) and Operator (13.27%) with a fraction of the parameter count. Offline metrics confirm the model's ability to minimize trajectory error and maximize endpoint accuracy.

The ablation analysis reveals:

• Flow-matching outperforms diffusion policy and language tokenization for drag trajectory modeling.
• Temporal reweighting and directional regularization dramatically enhance both geometric fidelity and task completion, with optimal weighting schemes presenting a marked increase in success rate.
• Unified modeling head is not only more practical but also matches or surpasses separate heads in efficacy and efficiency.

Data-driven Closed-loop Online Evaluation

To enable reproducible online evaluation, the authors devise a data-driven closed-loop protocol that dynamically selects the next screen state based on the proximity of the predicted action to pre-collected trajectory waypoints.

Figure 6: Visualization of the closed-loop evaluation pipeline, supporting on-the-fly continuous actions with reproducible environment snapshots.

Qualitative Analysis and Failure Modes of Baseline Models

Systematic case studies demonstrate that discrete-token baselines frequently:

• Fail to enact non-linear or arc-shaped drags (Figure 7).
• Stall or prematurely terminate continuous interactions due to lack of trajectory modeling (Figure 8, Figure 9).
• Misclassify benign UI manipulation tasks as safety violations (Figure 10).
• Misinterpret semantic intent in UI layouts (Figure 11).
• Substitute required continuous actions with series of clicks (Figure 12).
• Exhibit poor geometric control resulting in significant overshoot or drift (Figure 13).
• Excessively request clarification or halt progress due to their dialog-oriented RLHF tuning (Figure 14).

In contrast, ShowUI-$π$ circumvents these deficiencies via on-the-fly, observation-conditioned action rollouts informed by dense ground-truth trajectories and unified continuous-discrete modeling.

Figure 7: Baseline model knows execution plan but lacks trajectory tool support, failing non-linear rotations.

Figure 8: Baseline transitions from intent to execution fail due to insufficient continuous trajectory modeling, resulting in stranded icons.

Figure 10: Safety misclassification halts correct Captcha drag execution prematurely.

Figure 11: Visual instruction misunderstanding results in incorrect cursor placement.

Figure 12: Selection of discrete clicks over required continuous drag primitivizes and fails slider action.

Figure 13: Correct direction but substantial overshoot; lacks fine-grained geometric control.

Figure 14: Unnecessary requests for clarification interrupt embodied autonomous control.

Figure 9: Early episode termination due to erroneous "Instruction Unclear" state.

Implications, Limitations, and Future Directions

ShowUI-$π$ sets a new benchmark for GUI agent dexterity by leveraging lightweight flow-based generative policies trained on dense demonstration trajectories. The methodology demonstrates the importance of closed-loop perception-action integration and continuous-time modeling for digital manipulation tasks that mirror the complexity of human cursor control.

Limitations include constrained model size and training data scale, as well as absence of advanced text-centric planning integration. Prospective extensions involve model scaling, co-training with larger and more varied trajectory datasets, hybrid integration with dialog/planning architectures, and adaptation to mobile and cross-platform scenarios.

Conclusion

In summary, ShowUI-$π$ addresses crucial limitations of conventional discrete-action GUI agents through unified flow-based trajectory modeling and comprehensive benchmarking in ScreenDrag. Its capacity for smooth, observation-conditioned action generation establishes a strong foundation for embodied digital agents approaching human-like manipulation fluency, with substantial future scope for scaling up performance and generalization (2512.24965).