Zero-Click Predictive GUI Interaction (PAD Paradigm)

• Zero-click predictive GUI interaction is a paradigm that foresees user intent and automates GUI actions using the PAD (Preview, Accept, or Discard) framework.
• It employs sophisticated feature extraction and ranking models to minimize pointer travel and enhance task efficiency, significantly reducing error rates.
• Agent-driven PAD systems further extend the concept to enable fully autonomous, multi-step workflows that improve task outcomes in complex interfaces.

Zero-click predictive GUI interaction, as instantiated by the PAD (Preview, Accept or Discard) paradigm, refers to a class of human-computer interfaces and automated agents that minimize or eliminate direct clicking by foretelling user intent, offering ranked predictions, and executing actions either autonomously or with minimal input. This paradigm spans interaction styles including real-time suggestion, multi-step agent-initiated workflows, and fully autonomous execution, depending on the underlying system autonomy. Zero-click mechanisms underpin a new generation of GUI agents and input strategies that seek to reduce fine-motor workload, accelerate task completion, and provide more ergonomic or efficient alternatives to manual GUI manipulation (Berengueres, 13 Nov 2025, Feng et al., 12 Feb 2026, Zheng et al., 10 Feb 2026, Huang et al., 1 May 2025).

1. Conceptual Foundations and PAD Interaction Models

Zero-click predictive GUI interaction represents a shift from deliberate, manual GUI interaction to agent- or ML-driven prediction and execution. The PAD paradigm formalizes this shift through three core phases: Prediction (anticipating likely targets or actions), Action (executing or preparing selected commands), and Decision (either by user acceptance, agent arbitration, or both). Two major instantiations exist:

• Preview–Accept–Discard (Classical PAD): The system computes a ranked list of predicted GUI targets, presents previews, and enables the user to accept the top choice, cycle alternatives, or discard—all without pointer-based clicking. Interaction proceeds by key sequences and release timing (Berengueres, 13 Nov 2025).
• Agent-Driven PAD: For high-autonomy GUI agents, PAD is absorbed into multi-step decision-making: the agent perceives the GUI, predicts likely actions, executes or simulates ahead, and only falls back on user approval in edge cases or exceptions (Feng et al., 12 Feb 2026, Huang et al., 1 May 2025).

2. System Architectures and Formalism

PAD-based systems exhibit a characteristic architecture with predictive modeling tightly coupled to interface state extraction:

• Feature Extraction: At each screen update, all actionable elements (e.g., DOM nodes, screen regions) are encoded with features including spatial coordinates, style, label embeddings, historical usage, and context (Berengueres, 13 Nov 2025).
• Scoring and Ranking: A learned model (linear or nonlinear) computes a score $R(x) = w\cdot\phi(x)$ for each element $x$, or more generally, $R(x) = P(x | \text{context})$. The top-k candidates are maintained as ordered sets for user previewing or agent consideration.
• User Arbitration (Classical PAD): Interaction is defined by a state machine: key press enters preview, key tap cycles candidates, and release timing ($\Delta t \leq \tau$) triggers acceptance or discard.
• Agent-Autonomy PAD: Modern GUI agents run loops of perception (state capture), anticipation (action prediction and simulation using world models), and execution, with zero-click operation except on fallbacks (Feng et al., 12 Feb 2026, Zheng et al., 10 Feb 2026, Huang et al., 1 May 2025).

Key mathematical elements in agent-driven PAD include: $$a_t,\, \hat{\text{history}}_{<t} = \mathcal{M}_\theta\Bigl(\text{task}, (o_1, a_1),\ldots,(o_{t-1}, a_{t-1}), o_t\Bigr)$$ where actions $a_t$ and their predicted context history are generated from the state and prior sequence (Huang et al., 1 May 2025).

3. GUI Agent Autonomy Taxonomies and Zero-Click Criteria

The GUI Agent Autonomy Levels (GAL) framework (Feng et al., 12 Feb 2026) provides an explicit six-tier taxonomy:

Level	Name	Zero-Click Predictivity	Example Systems
0	No Automation	None	Manual user operation
1	Minimal Assistance	Inline predictions, no action	Smart Compose, tooltips
2	Basic Automation	Executes user-selected atomic predictions	UI Automator, AppleScript
3	Conditional	Plans tasks; confirms exceptions	UiPath, WebAgent, AppAgent
4	High Automation	Fully zero-click except rare fallbacks	ChatGPT Atlas, Claude Use Agent
5	Full Automation	Universal zero-click, full task autonomy	Future true universal GUI agents

"True zero-click"—i.e., once a high-level goal is issued, the agent proceeds without intervention—arises in levels 4 (high) and 5 (full automation).

PAD’s Prediction, Action, and Decision phases map onto these autonomy levels: Prediction dominates at levels 1–2, Action at levels 2–3, and multi-step Decision at 3–5. Metrics for such mapping include suggestion accuracy, completion rates, and user intervention frequency (Feng et al., 12 Feb 2026).

4. Model Training, Data Construction, and Evaluation

Both classical and agent-driven PAD architectures demand carefully constructed predictive models:

• Supervised and Reinforcement Learning: Approaches such as Code2World (Zheng et al., 10 Feb 2026) employ supervised fine-tuning (HTML code generation from screenshots and action context) followed by render-aware RL, with rewards combining semantic fidelity and action consistency. ScaleTrack (Huang et al., 1 May 2025) blends grounding (point regression for actionable elements) with planning and back-tracking.
• Synthetic and Unified Data: Datasets are synthesized by extracting labeled GUI action traces across millions of screenshots, merging sources (e.g., Uground, Aria-UI, Aguvis) into unified pools of point annotations and multi-step task traces (Huang et al., 1 May 2025, Zheng et al., 10 Feb 2026).
• Objective Functions: Core losses include: $$L_g = \frac{1}{N}\sum_{i=1}^N \bigl\|\hat b_i - b_i^*\bigr\|^2$$ for grounding and

$$L_p = -\sum_{t=1}^T \log P(a_t | G_t, h_{t-1}) + \lambda\sum_{k=1}^{t-1} \mathcal{D}(h_k, \hat{h}_k)$$

for planning with back-tracking (Huang et al., 1 May 2025).

• Evaluation Metrics: Quantitative metrics include action adherence (Sad), action identifiability (Sid), visual alignment (Sele, Stay), semantic similarity (SigLIP), next-action accuracy, step-wise and end-to-end task success rates (SR) (Zheng et al., 10 Feb 2026, Huang et al., 1 May 2025). User-facing measures include pointer travel reduction, throughput (bits/s), strokes per task, and error rates (Berengueres, 13 Nov 2025).

5. Empirical Results and Practical Outcomes

Empirical studies demonstrate the feasibility and effectiveness of PAD-based, zero-click predictive GUI interactions:

• Interaction Efficiency: Classical PAD eliminates nearly all pointer travel (≈600 px per click, ≈85–90% reduction), holding task time steady against trackpad controls when predictive accuracy exceeds 90% (Berengueres, 13 Nov 2025).
• Error Rates and User Experience: PAD with high-accuracy models yields lower error rates (1% vs. 9% for traditional pointer), though cycling through mispredicted options raises decision costs. Users acquire proficiency rapidly and report strong agency.
• Agent-Based Autonomy: Code2World-8B matches or exceeds closed-source VLMs such as GPT-5 on in-domain and out-of-distribution next-UI prediction, with downstream agent navigation success rates improving by 1–9% when predictive world models are plugged in (Zheng et al., 10 Feb 2026). ScaleTrack achieves top-1 click accuracy of 86.8% and raises end-to-end success on AndroidWorld to 44% (vs. 37% with earlier systems) (Huang et al., 1 May 2025).
• Task Automation: True zero-click operation is realized in business workflows such as data extraction, analysis, and reporting, with agents executing full multi-app sequences autonomously except in rare edge cases (Feng et al., 12 Feb 2026).

6. Design Guidelines, Limitations, and Future Directions

Key design imperatives for PAD-based, zero-click predictive interfaces include (Berengueres, 13 Nov 2025, Feng et al., 12 Feb 2026):

• Limit candidate set k to 2–6 to manage cognitive decision costs.
• Use visually clear previews that match the UI’s visual design, supporting rapid preview and selection without pointer movement.
• Fine-tune acceptance thresholds (release timing τ) per user to maximize comfort and precision.
• Progressive onboarding: Reveal advanced PAD functionalities incrementally, starting with preview and cycling before introducing keyed acceptance/discard.
• Agent-centric guidelines: Guarantee security (defined permission boundaries, transparent logs), privacy (on-device processing, data minimization), and personalization (contextual adaptation to user routines).
• Scalability and Robustness: Unifying multiple data sources and employing massive synthetic datasets supports robust transfer across GUI platforms.

Limitations include sensitivity to model misprediction rates, limited clinical evaluation of RSI mitigation, dependency on high-fidelity UI simulation for planning agents, and challenges in adapting to complex widget types or novel layouts (Berengueres, 13 Nov 2025, Zheng et al., 10 Feb 2026, Huang et al., 1 May 2025). Prospective extensions involve hybrid input (e.g., voice, gaze), integration with closed-loop RL for robust adaptation, and longitudinal ergonomic studies.

7. Connections to Broader Research and Applied Perspectives

Zero-click predictive GUI interaction operates at the confluence of human–computer interaction, vision-language modeling, and software automation. PAD-style frameworks intersect with work on predictive text input, visual grounding in VLMs, and hierarchical agent planning. Embedding the PAD paradigm within the GUI Agent Autonomy Levels supplies a rigorous measurement framework for benchmarking progress toward autonomous, trustworthy software interaction (Feng et al., 12 Feb 2026). Applied utility spans accessibility (reduced dependence on fine-motor skills), enhanced productivity in business and developer settings, and foundational infrastructure for general-purpose software agents.