World Mouse: Exploring Interactions with a Cross-Reality Cursor

Abstract: As Extended Reality (XR) systems increasingly map and understand the physical world, interacting with these blended representations remains challenging. The current push for "natural" inputs has its trade-offs: touch is limited by human reach and fatigue, while gaze often lacks the precision for fine interaction. To bridge this gap, we introduce World Mouse, a cross-reality cursor that reinterprets the familiar 2D desktop mouse for complex 3D scenes. The system is driven by two core mechanisms: within-object interaction, which uses surface normals for precise cursor placement, and between-object navigation, which leverages interpolation to traverse empty space. Unlike previous virtual-only approaches, World Mouse leverages semantic segmentation and mesh reconstruction to treat physical objects as interactive surfaces. Through a series of prototypes, including object manipulation and screen-to-world transitions, we illustrate how cross-reality cursors may enable seamless interactions across real and virtual environments.

• The paper introduces a novel cross-reality cursor framework that decouples 2D input precision from physical reach through seamless 2D to 3D transitions.
• The paper details innovative techniques such as within-object traversal using surface normals and between-object interpolation via an 'invisible mesh' for smooth navigation.
• The paper demonstrates practical impacts in spatial authoring, IoT control, and collaborative XR workflows by reducing user fatigue and enhancing accuracy.

World Mouse: Comprehensive Cross-Reality Cursor Interaction Framework

Motivation and Overview

The paper "World Mouse: Exploring Interactions with a Cross-Reality Cursor" (2603.10984) addresses persistent challenges in XR interaction paradigms, particularly the fatigue and imprecision inherent in direct touch, mid-air gestures, and gaze-based interfaces when attempting fine control or extended usage. Contrary to prevailing views in spatial computing—which often emphasize embodied interaction—the authors propose a reconsideration of desktop mouse input. They introduce the World Mouse, a spatially-aware, cross-reality cursor system, designed to decouple 2D input precision from physical reach and enable seamless manipulation across blended 3D environments that combine virtual and physical interactables.

Technical Design and Core Mechanisms

The World Mouse framework is underpinned by two principal cursor behaviors:

• Within-Object Navigation: Enables the cursor to move along object surfaces by dynamically aligning the cursor orientation with local surface normals and adapting depth to the 3D mesh topology, ensuring precise spatial selection and manipulation.
• Between-Objects Navigation: Employs interpolation over an "invisible mesh" connecting disjoint objects, leveraging semantic segmentation and 3D mesh reconstruction (using APIs such as Meta's Scene API and Android XR Scene Meshing) to facilitate continuous cursor traversal across empty space, rather than relying solely on raycasting.

Additionally, the system supports context-aware 2D to 3D transitions, allowing for fluid movement between conventional 2D applications or panels and 3D spatial environments, mirroring desktop cursor metaphors in immersive settings.

Figure 1: Cursor behaviors explored through the World Mouse, including surface traversal, interpolation between objects, and seamless 2D/3D transitions.

Implementation and Application Scenarios

The World Mouse is implemented as a blended scene graph mapping 2D mouse deltas onto spherical angular displacements relative to the user's viewpoint—following extensions of prior depth-adaptive cursor research. The environment is continuously reconstructed using convex hulls and semantic labels, yielding a stable mesh that supports both static and dynamic elements without excessive recomputation.

The authors demonstrate its versatility across several spatial computing workflows:

• Spatial Authoring: Enables high-precision 3D manipulation, including spline editing and vertex-snapping—operations traditionally difficult with mid-air gestures. Interactive gizmos afford granular transform control, supporting CAD-like editing and annotation of real-world meshes.
• Cross-Device and IoT Control: Facilitates the migration of digital content from physical screens into spatial contexts; IoT devices are controlled through semantically-aware proxies and passthrough filters, unifying physical and virtual device interaction.
• Social XR and Collaborative Workflows: Supports multi-user engagement, allowing for shared manipulation of virtual assets and compatibility with conventional laptop or desktop displays by providing a 2D projection of the scene graph.

  Figure 2: World Mouse application scenarios, including spatial authoring, IoT device control, and collaborative social XR workflows.

Further, by integrating cross-device toolkits (e.g., XDTK), the system exploits familiar touchscreens—such as smartphones and smartwatches—as high-fidelity input proxies, surpassing hand-tracking and raycasting in precision, particularly for multi-user and distributed productivity scenarios.

Figure 3: Use of mobile touchscreens and wearables as high-precision World Mouse controllers via XDTK.

Empirical and Conceptual Contributions

The work counters prevailing assumptions that spatial computing should abandon mouse-based interaction. It empirically demonstrates that cursor-based models substantially reduce fatigue and improve accuracy in extended workflows, especially for tasks requiring fine spatial manipulation and intent disambiguation. Unlike prior efforts limited to single modality (VR or AR), this approach traverses the full spectrum of blended environments, integrating both real and synthetic interactables.

A salient conceptual contribution is the notion of continuity in cross-reality interaction: the World Mouse bridges 2D and 3D, digital and analog, local and remote environments. It supports established desktop metaphors (point-and-click, right-click contextual menus, scroll for depth/zoom) and adapts them for spatial computing, maintaining user familiarity and dexterity.

Implications and Future Directions

Practically, the World Mouse provides a robust input framework for spatial authoring, collaborative XR, and device control, addressing the limitations of mid-air and gaze-based interactions in high-precision use cases. Theoretically, the model underscores the importance of continuity and intent granularity in future spatial UI/UX design.

With ongoing advances in semantic segmentation and depth mapping, the system is positioned to serve as a precise anchor for AI-driven workflows, enabling deictic reference and manipulation tasks that require unambiguous object targeting—a critical requirement for generative and embodied AI agents operating in spatial environments.

Future developments may explore incorporation into unconstrained freehand tasks (e.g., sculpting, drawing), optimization for large-scale collaborative spaces, and further integration with multimodal AI for enhanced scene understanding and intent expression in XR. The cursor-based paradigm may also serve as a bridge for cross-device interoperability, supporting distributed interaction among heterogeneous hardware ecosystems.

Conclusion

The World Mouse articulates an authoritative framework for cursor-driven interaction across blended XR environments, leveraging geometric and semantic scene representations to surpass the limitations of touch and ray-based input. It enriches spatial computing workflows by preserving precision, reducing fatigue, and facilitating fluid, cross-reality engagement—while providing backward compatibility with traditional desktop metaphors. As spatial and generative AI systems proliferate, the World Mouse offers a viable model for intent-rich, low-effort interaction, supporting both practical and conceptual advancements in human-computer interaction.