GhostUI: Unveiling Hidden Interactions in Mobile UI

Abstract: Modern mobile applications rely on hidden interactions--gestures without visual cues like long presses and swipes--to provide functionality without cluttering interfaces. While experienced users may discover these interactions through prior use or onboarding tutorials, their implicit nature makes them difficult for most users to uncover. Similarly, mobile agents--systems designed to automate tasks on mobile user interfaces, powered by vision LLMs (VLMs)--struggle to detect veiled interactions or determine actions for completing tasks. To address this challenge, we present GhostUI, a new dataset designed to enable the detection of hidden interactions in mobile applications. GhostUI provides before-and-after screenshots, simplified view hierarchies, gesture metadata, and task descriptions, allowing VLMs to better recognize concealed gestures and anticipate post-interaction states. Quantitative evaluations with VLMs show that models fine-tuned on GhostUI outperform baseline VLMs, particularly in predicting hidden interactions and inferring post-interaction screens, underscoring GhostUI's potential as a foundation for advancing mobile task automation.

• The paper investigates hidden interactions in mobile UIs that lack visual affordances and presents GhostUI, a dataset of 1,970 validated gestures across 81 Android apps, capturing gestures like tap, double-tap, and long press
• The GhostUI dataset showed that hidden interactions account for a significant portion of mobile UI functionalities, particularly affecting automated systems and user experience in areas without visual cues
• The integration of simplified XML hierarchies into a visual-binding model led to a substantial improvement in predicting hidden interactions
• The authors demonstrated that the fine-tuning of models like GPT-4o significantly enhanced their ability to predict hidden interactions, achieving a 65.6% accuracy and 42.5% Io
• The study highlights the importance of including simplified interactive boundaries in models to accurately predict hidden interactions, addressing a key limitation in current vision-only models.

Motivation and problem statement

Modern mobile applications increasingly rely on hidden interactions—gestures such as long presses, double taps, and swipes that trigger functionality without any visual affordance. These interactions serve two purposes: providing shortcuts to frequently used functionality and revealing UI elements that would otherwise consume scarce screen space. While human users can discover such gestures through exploration, onboarding, or prior experience, VLM-based mobile agents can only perceive what is visually present in a screenshot. The problem is compounded by a many-to-many mapping between elements and gestures: on the YouTube video player, tap reveals controls, double tap skips, and long press enables 2x playback—all on the same region; conversely, the same double tap likes a post in Instagram but switches accounts when applied to the profile icon.

The authors' survey of mobile agent action spaces substantiates this gap: none of the surveyed agents or interactive environments support double tap or pinch, and only five implement long press. Existing datasets (RICO, AITW, MobileViews, UIBert) predominantly capture visually apparent interactions with simple gestures. This dataset bias directly constrains system capability—agents cannot learn interactions absent from training data.

Dataset construction

GhostUI contains 1,970 validated hidden interaction instances from 81 popular Android applications, drawn from an initial pool of 8,312 automatically probed interactions—a 23.7% yield that itself indicates how non-trivially distributed hidden functionality is. Collection proceeded in three phases:

1. Automated probing: An open-source Appium-based tool executes six gesture types (tap, double tap, long press, swipe, scroll, pinch) on all interactive elements identified via XML view hierarchy parsing, using path-based element tracking for dynamic content and consistent gesture parameters for reproducibility. For long press and pinch, an additional "during" state is captured to detect transient effects. State change detection combines view-hierarchy comparison with pixel-level screenshot differencing (5% average RGB threshold) for purely visual changes.
2. Manual validation: Five annotators assessed interaction validity, labeled visual elements within bounding boxes (border, text, icon, media, whitespace), and judged hidden nature against structured guidelines. Inter-annotator agreement was strong: Fleiss' $\kappa$ = 0.89 for validity/hidden nature and mean $\kappa$ = 0.76 for element labeling.
3. Task contextualization: GPT-4o generated natural language task descriptions expressing user intent, manually reviewed for accuracy, providing supervision for intent-to-action prediction.

Each instance includes before/after screenshots (plus "during" screenshots where applicable), both raw XML and simplified HTML-like view hierarchies, gesture metadata with coordinates, task descriptions, and app metadata. Data was split at the application level (56 train / 25 test apps) to force generalization rather than app-specific memorization.

Empirical characterization of hidden interactions

The dataset analysis yields several notable findings. Tap accounts for 30.3% of hidden interactions, followed by swipe (26.0%) and long press (19.3%); specialized gestures—long press, double tap, pinch—collectively constitute 37.7% of hidden functionality while being precisely the gestures least supported by existing agents. LDA topic modeling of task descriptions revealed both diversity and redundancy: distinct gestures often produce similar outcomes (double tap and pinch both zoom in maps), while the same gesture serves different roles across contexts.

The visual-context analysis is arguably the most consequential finding: 210 instances involve isolated whitespace as the sole labeled element, demonstrating that completely unmarked regions are legitimate interaction targets. Border elements show the weakest association with hidden interactions across all gestures, whereas media strongly supports hidden long press and double tap, and whitespace supports hidden pinch and scroll. Container-based architectures further create ambiguous targets where visually distinct regions trigger identical responses. These patterns establish that hidden interactions are systematic design practice rather than edge cases, quantifying a discoverability deficit affecting both users and automated systems.

VLM experiments

Two tasks were evaluated: Hidden Interaction Prediction (predict gesture type and bounding box given a screenshot and task description) and UI Transition Prediction (describe post-gesture screen changes). Models were Qwen2.5-VL (7B) and GPT-4o, evaluated zero-shot and fine-tuned (LoRA and OpenAI's Vision Fine-tuning API respectively) across five input configurations.

Key results:

Model	Setting	Accuracy (%)	IoU (%)
GPT-4o	Zero-shot	51.1	36.0
GPT-4o	Fine-tuned	65.6	42.5
Qwen2.5-VL	Zero-shot	33.3	19.5
Qwen2.5-VL	Fine-tuned	40.5	22.8

Ablations produced two structurally important findings. First, removing the simplified view hierarchy collapsed IoU dramatically (−32.6 points for zero-shot GPT-4o; −37.6 fine-tuned) while leaving classification accuracy nearly intact. The authors attribute this to a mismatch between visual and interactive boundaries: VLMs localize visually salient objects, whereas touch targets often encompass entire containers including padding and whitespace. This is a direct challenge to vision-only grounding approaches such as SeeClick and OmniParser, which are trained exclusively on visible elements—hidden interactions lack visual affordances by definition, so their interactive boundaries are invisible to vision-only methods. Second, removing gesture usage patterns caused Qwen2.5-VL's accuracy to fall from 33.3% to 12.0%, indicating smaller models depend heavily on explicit gesture priors; app metadata contributed little.

Confusion matrix analysis showed severe zero-shot bias toward tap: over 91% of double taps were misclassified as tap, and long press misclassification ranged 56–85%. Fine-tuning substantially rebalanced predictions (double tap recognition 0%→36%, long press 40%→75%, swipe 22%→72%), though double tap remained hardest (36%) and pinch confused with double tap (30%) due to overlapping visual contexts.

For UI Transition Prediction, measured by cosine similarity between predicted and ground-truth after-state descriptions, fine-tuning yielded statistically significant improvements for five of six gesture types (Wilcoxon signed-rank tests, mostly $p<0.001$). Double tap was the exception ($p>0.05$): zero-shot models exhibited a strong pre-training bias toward predicting zoom outcomes, and double tap's high outcome diversity resisted correction through fine-tuning.

Potential applications

The authors sketch two mock-up applications grounded in their results. An interactive guidance system would surface hidden gestures contextually—for example, overlaying instructions to long-press video edges for 2x playback—preserving clean interfaces under progressive disclosure principles. A design recommendation tool would audit implementations against empirical cross-app conventions, e.g., recommending scroll-up for revealing ephemeral controls on whitespace while warning that tap on whitespace typically dismisses keyboards. These applications follow directly from the demonstrated ability to predict gesture type, target region, and resulting state changes.

Limitations and open questions

Several limitations are acknowledged explicitly. Coverage is Android-only, excluding iOS's distinct gesture conventions and other form factors; wearables, drag-and-drop, multi-finger gestures, and sensor-based interactions (shake, tilt) remain unexplored. The key-screen collection strategy limited exploration of nested interfaces, and the cold-start problem meant features emerging only after sustained usage history were largely inaccessible—the authors report that current mobile automation frameworks were insufficient for reliable autonomous exploration as a remedy. Evaluation is also constrained in two ways the authors flag: the test set contains only screens where hidden interactions exist, omitting mixed conditions and infeasible tasks, and exact gesture matching penalizes functionally equivalent alternatives (e.g., double tap vs. pinch for zooming). Finally, the dataset quantifies prevalence but not human cognition; whether hidden interactions function as convenient shortcuts or sole pathways varies by app—their own sampling found features like emoji reactions consistently lacking visible triggers, and cases where even alternative pathways were themselves hidden—which motivates dedicated user studies on discoverability and learnability.

Conclusion

GhostUI provides the first systematic dataset of hidden mobile interactions, a semi-automated collection pipeline released as open source, and quantitative evidence that fine-tuning on this data improves VLM gesture classification, spatial localization, and transition prediction on unseen applications. Its ablations carry a clear design implication for mobile agents: vision-only grounding is insufficient for interactions without visual affordances, and structural information capturing interactive boundaries independent of appearance is necessary. The remaining open questions—platform extension, richer action spaces, functionally equivalent evaluation, and human-factors validation of discoverability—define the immediate research agenda this work sets up.