World Mouse: Precision in Cross-Reality

• World Mouse is an XR and cross-reality interaction paradigm that repurposes a conventional 2D mouse as a depth-aware 3D cursor for precision and reduced fatigue in spatial tasks.
• It utilizes indirect input, converting mouse motion into stable cursor motion with surface-normal alignment and Voronoi-based depth interpolation to traverse both virtual and physical objects.
• The system supports seamless modality switching between precise desktop control and hand-tracked manipulation, demonstrating lower docking error and ergonomic strain compared to ray-based methods.

Searching arXiv for the named papers and closely related background work so the article can cite current sources directly. {"query":"World Mouse cross-reality cursor arXiv (Tütüncü et al., 11 Mar 2026)", "max_results": 5} World Mouse denotes an XR and cross-reality interaction paradigm that reinterprets a familiar 2D mouse as a cursor for complex 3D scenes, with the explicit goal of combining precision, reach, and low-fatigue operation. In current arXiv literature, the term covers closely related systems: a cross-reality cursor that moves continuously across physical and virtual objects, and a hybrid XR workflow in which a desk-anchored mouse drives a depth-aware 3D cursor for selection before control is handed to tracked hands for manipulation. Across both formulations, the central idea is to decouple precision from physical reach, avoid sustained arm extension, and preserve continuity of control as interaction moves among 2D panels, 3D geometry, and blended physical–virtual environments (Tütüncü et al., 11 Mar 2026, Tütüncü et al., 31 Jul 2025).

1. Conceptual framing and motivation

World Mouse emerges from a critique of the dominant XR emphasis on embodied input. The 2026 formulation states that touch is limited by human reach and fatigue, gaze often lacks the precision required for fine interaction, and combined techniques such as gaze+pinch still demand bimanual effort and mid-air gesturing. It further notes that consumers rely heavily on raycasting, with about 60% of VR titles using controller-based rays for selection, and that XR has traditionally emphasized near-field interactions at roughly 0.5–3.5 meters, constraining interaction range and workflow flexibility (Tütüncü et al., 11 Mar 2026).

Within this problem framing, World Mouse treats precision and long-session comfort as central requirements for XR tasks such as 3D modeling, annotation, and spatial authoring. The proposed solution is indirect input: a desktop-style cursor preserves the mouse’s well-known fine control while avoiding extended arm postures. This suggests that World Mouse is not a rejection of embodied interaction, but a repositioning of it within a broader input ecology in which precision, reach, and endurance are explicitly optimized.

The HandOver system operationalizes this idea in XR by assigning distinct roles to two modalities. The mouse supplies a depth-aware 3D cursor for precise and restful targeting, while hand tracking supplies expressive 3D manipulation once selection has been completed. Its stated contribution is a unified workflow in which the system detects when to hand over control, so users do not manually switch modes or choose among separate tools (Tütüncü et al., 31 Jul 2025).

2. Cursor mechanics in 3D and cross-reality space

A defining property of World Mouse is the conversion of 2D mouse motion into stable cursor motion in 3D. In HandOver, mouse-controlled screen coordinates are denoted by $s = (u, v)$ in pixels. Using normalized device coordinates, the mapping is

• $x_{ndc} = 2u/W - 1$
• $y_{ndc} = 1 - 2v/H$

A ray $r(t)$ in camera space is obtained by unprojecting $(x_{ndc}, y_{ndc}, 1)$ through the inverse projection to produce a normalized direction $d_c$, yielding

• $r_c(t) = o_c + t d_c$, with $o_c = (0, 0, 0)$

The ray is transformed to world space with camera extrinsics $(R, t)$:

• $d_w = R d_c$
• $x_{ndc} = 2u/W - 1$0
• $x_{ndc} = 2u/W - 1$1

If camera intrinsics $x_{ndc} = 2u/W - 1$2 are used, the camera-space point at depth $x_{ndc} = 2u/W - 1$3 is

• $x_{ndc} = 2u/W - 1$4
• $x_{ndc} = 2u/W - 1$5
• $x_{ndc} = 2u/W - 1$6

and the world point is $x_{ndc} = 2u/W - 1$7 (Tütüncü et al., 31 Jul 2025).

Depth handling is the critical technical issue. HandOver places the cursor at the nearest raycast hit $x_{ndc} = 2u/W - 1$8 when a collision occurs. If no collision occurs, it estimates depth using a Voronoi diagram over nearby colliders to interpolate a plausible depth behind the current screen-space position. Depth and pose are then smoothed to avoid jitter, for example with the low-pass update

$x_{ndc} = 2u/W - 1$9

and the cursor is distance-rescaled to maintain consistent apparent size and legibility over 2–6 m. The stated effect is a cursor that feels anchored to the scene, resists tremor amplification, and supports small-target selection and disambiguation in clutter (Tütüncü et al., 31 Jul 2025).

The 2026 World Mouse generalizes this cursor model beyond virtual-only scenes. It distinguishes two interaction mechanisms. “Within-object interaction” constrains the cursor to the surface of the intersected object, with cursor orientation aligned to the local surface normal. “Between-object navigation” interpolates across empty space using a blended scene mesh and a “dynamic bridge,” allowing the cursor to traverse disjoint objects without abrupt depth jumps. The paper does not provide explicit mathematical formulations, but its algorithmic description emphasizes cast-from-viewpoint intersection, surface-normal alignment, and interpolation toward adjacent geometry (Tütüncü et al., 11 Mar 2026).

3. Scene representation, semantics, and cross-reality continuity

The cross-reality version of World Mouse depends on scene understanding. Its environment model is obtained from platform scene-reconstruction pipelines such as Meta’s Scene API or Android XR’s Scene Meshing, which provide geometry anchored to the headset’s tracking coordinate frame. On top of this geometric substrate, semantic segmentation identifies interactive surfaces and 2D-relevant regions such as monitors, panels, text, and screens. The paper cites recent advances such as Segment Anything and diffusion-based approaches, but does not commit to a single model or report training or performance numbers (Tütüncü et al., 11 Mar 2026).

A key representational device is the construction of a continuous “invisible mesh” from convex hulls of detected objects, linked relative to the camera view. This interaction surface is intended to stabilize cursor motion across static and dynamic real-world elements and to provide continuity across voids. Physical objects become first-class interactive surfaces rather than background geometry, and virtual objects are registered similarly in a blended scene graph. The result is a cursor model in which real and virtual surfaces are treated uniformly.

This continuity extends to 2D panels and screen-to-world transitions. The cursor can slide out of a browser-like panel into 3D space, including physical surroundings, while preserving continuity of control and visual feedback. A plausible implication is that World Mouse is designed less as a specialized 3D pointing trick than as a generalization of desktop interaction primitives—point, hover, click, drag, context menu, clipboard—into semantically structured spatial environments (Tütüncü et al., 11 Mar 2026).

4. HandOver as an XR realization of the World Mouse

HandOver provides the most explicit realization of the “World Mouse” idea in XR by combining mouse-driven targeting with hand-tracked manipulation in a mutual-exclusive state machine. The mouse controls the depth-aware 3D cursor. When the user intends to manipulate, the system detects hover above the mouse and transitions to a virtual “hand clone” located at the cursor position. The user then performs pinch and grasp gestures directly on the selected object (Tütüncü et al., 31 Jul 2025).

The gating logic is specified in detail. An invisible bounding box is placed approximately 8 cm above the user’s real-hand anchor point, accounting for mouse height and rested hand. When the hand enters this zone with a vertical offset of about 0.05 m above the anchor, and the cursor has been stationary below a 1 mm threshold for multiple frames, intent to manipulate is inferred. The 3D cursor fades out over 0.2 s and a hand clone fades in at the cursor’s last position; during clone activity, mouse input is ignored. Hand-clone displacement is amplified by a configurable gain for comfortable far-field manipulation, and the displacement between the real hand and the clone is clamped to 4.0 m to prevent runaway drift (Tütüncü et al., 31 Jul 2025).

Object attachment is also thresholded. Within a 5 cm radius around the clone, if index/thumb pinch strength exceeds 0.5, the object is attached to the clone’s transform and remains grabbed until the pinch ends. “Closed” hand states can be thresholded at 0.3–0.5 across multiple fingers, and an “open palm” requires all pinch strengths below 0.05 and a palm-facing-camera check via a dot product less than $y_{ndc} = 1 - 2v/H$0. When the user releases the pinch and returns the hand within approximately 2 cm of the original anchor, the hand clone fades out, the cursor reappears at the prior point, and mouse control resumes (Tütüncü et al., 31 Jul 2025).

Continuity of manipulation is maintained with hand-relative deltas. If $y_{ndc} = 1 - 2v/H$1 is the current hand clone transform, $y_{ndc} = 1 - 2v/H$2 the hand clone at grab onset, and $y_{ndc} = 1 - 2v/H$3 the object transform at grab onset, then

$y_{ndc} = 1 - 2v/H$4

so that the object follows the hand’s incremental motion relative to its initial condition. Position and orientation gains may be applied per-axis, and damping may be introduced by filtering $y_{ndc} = 1 - 2v/H$5 or by a spring-damper model if higher stability is required. In this implementation, the World Mouse is therefore not merely a cursor but a modality-aware control architecture linking precise world-space selection to expressive manipulation (Tütüncü et al., 31 Jul 2025).

5. Empirical evidence and comparative performance

HandOver is accompanied by a formal user study, whereas the 2026 World Mouse paper does not report a formal user study or quantitative evaluation. No time, error, throughput, latency, or reconstruction or segmentation accuracy metrics are provided for the latter, which leaves its contribution primarily at the level of system design, prototypes, and interaction mechanisms (Tütüncü et al., 11 Mar 2026).

The HandOver study used a 3D docking task structured around Fitts’ Law principles. Targets were arranged on a circular layout around the participant with a radius of approximately 1 m; the circle was divided into 11 equal angles of approximately $y_{ndc} = 1 - 2v/H$6, and objects were spawned with up to $y_{ndc} = 1 - 2v/H$7 m offset from corresponding square target zones. Distances were Near (2 m), Mid (4 m), and Far (6 m). Apparatus consisted of Unity 2022.3.39f1, Oculus Quest 3 with the Oculus hand-tracking API, and a portable laptop. Seventeen participants took part: 9 male and 8 female, with mean age 28.94 years ($y_{ndc} = 1 - 2v/H$8). VR familiarity was self-reported as $y_{ndc} = 1 - 2v/H$9, $r(t)$0, and VR development experience as $r(t)$1, $r(t)$2, both on 7-point scales (Tütüncü et al., 31 Jul 2025).

Three techniques were compared: HandOver (HO), Ray+Hand (R+H), and Ray (R). Aggregated across all distances, targeting time was HO 5.3 ($r(t)$3), R+H 4.1 ($r(t)$4), and R 6.4 ($r(t)$5) seconds, with technique $r(t)$6, $r(t)$7; distance $r(t)$8, $r(t)$9; interaction $(x_{ndc}, y_{ndc}, 1)$0. Docking time was HO 5.1 ($(x_{ndc}, y_{ndc}, 1)$1), R+H 4.2 ($(x_{ndc}, y_{ndc}, 1)$2), and R 6.9 ($(x_{ndc}, y_{ndc}, 1)$3) seconds, with technique $(x_{ndc}, y_{ndc}, 1)$4, $(x_{ndc}, y_{ndc}, 1)$5; distance $(x_{ndc}, y_{ndc}, 1)$6, $(x_{ndc}, y_{ndc}, 1)$7; interaction $(x_{ndc}, y_{ndc}, 1)$8, $(x_{ndc}, y_{ndc}, 1)$9. Docking error was HO 2.3 ($d_c$0), R+H 2.6 ($d_c$1), and R 3.7 ($d_c$2) cm, with technique $d_c$3, $d_c$4; distance $d_c$5, $d_c$6; interaction $d_c$7, $d_c$8. By distance, HandOver achieved the lowest error at all ranges: Near approximately 2.3 cm and Far approximately 2.9 cm, versus 2.6–3.2 cm for Ray+Hand and greater than 3.5 cm for Ray (Tütüncü et al., 31 Jul 2025).

Ergonomic measures were equally central. RULA composite scores were HO 5.8 ($d_c$9), R+H 6.8 ($r_c(t) = o_c + t d_c$0), and R 7.2 ($r_c(t) = o_c + t d_c$1), with technique $r_c(t) = o_c + t d_c$2; HandOver showed significantly lower strain than both baselines. Accumulated movement was HO 956.4 ($r_c(t) = o_c + t d_c$3), R+H 1085.9 ($r_c(t) = o_c + t d_c$4), and R 1585.5 ($r_c(t) = o_c + t d_c$5) cm, again with technique $r_c(t) = o_c + t d_c$6. Regrab counts were highest for Ray, especially at Far at approximately 1.4 per trial, lower for HandOver at 0.7–0.8, and intermediate for Ray+Hand at 0.5–1.0. NASA-TLX subscales showed significant differences in Physical Demand, Performance, and Frustration at approximately $r_c(t) = o_c + t d_c$7–.0065; HandOver was rated lowest in Physical Demand and Frustration with medians of 2.0, while Ray was rated most physically demanding and frustrating with median 6.0. These results support the specific claim that World Mouse-style interaction can reduce docking error, accumulated movement, and ergonomic strain relative to purely ray-based baselines (Tütüncü et al., 31 Jul 2025).

6. Applications, design principles, and open problems

The 2026 World Mouse paper situates the technique in a broad application space. Demonstrated prototypes include desktop metaphors in 3D, spatial clipboard operations, screen-to-world transitions, scroll-based depth and scale adjustment, spatial authoring with 3-axis transform gizmos, spline editing, vertex snapping to real-world meshes, contextual radial menus, and “ghost” object spawning and anchoring. Semantic labels are also used for IoT proxies and context-aware UI, for example exposing different menu options on a wall, a screen, or a device proxy. The same scene representation is presented as a substrate for cross-device control through smartphones or smartwatches via XDTK, for co-located and remote collaboration, and for AI-grounded interaction in which precise cursor pointing reduces ambiguity in deictic references (Tütüncü et al., 11 Mar 2026).

Across the literature, several design principles recur. Depth handling is treated as essential: nearest-surface hits should be combined with a fallback such as Voronoi blending or interpolation, followed by smoothing, to avoid depth popping. Cursor size should be distance-rescaled for a consistent visual feel across 2–6 m. Surface alignment to local normals is used to convey topology and support precise placement. Continuity across modalities is preserved by treating real and virtual objects uniformly and by allowing seamless transitions among surfaces, gaps, and panels. In HandOver, modality switching should be implicit and exclusive, using spatial gating and stationary cues while disabling mouse input during hand-clone activity. Semantic labels are used to drive context-aware UI, and scroll-wheel or related controls are used to disambiguate depth and scale (Tütüncü et al., 31 Jul 2025, Tütüncü et al., 11 Mar 2026).

The limitations are equally explicit. HandOver depends on desk and seated setups; standing or mobile contexts require alternatives such as mid-air pointer emulation or touch surfaces like TriPad to retain precision without hardware. Camera-based hand tracking can degrade with fast motion or suboptimal lighting or view angles. Densely occluded scenes still pose disambiguation challenges even when Voronoi blending stabilizes depth. The bounded manipulation zone in HandOver leaves longer-range placement and richer constraints as future work. In the broader cross-reality formulation, robust scene understanding is a prerequisite: segmentation errors, dynamic scenes, transparent or reflective surfaces, rough or irregular meshes, disconnected geometry, and calibration drift may degrade cursor stability, yet failure-handling specifics are not quantified. The 2026 paper also states that World Mouse is not designed for unconstrained freehand sketching or continuous mid-air sculpting, which continue to favor embodied techniques (Tütüncü et al., 31 Jul 2025, Tütüncü et al., 11 Mar 2026).

Taken together, these systems define World Mouse as a precision-oriented, low-fatigue interaction strategy for blended physical–virtual environments. Its distinctive contribution is to recover the mouse as a high-fidelity control instrument in XR, while extending it beyond the desktop: onto world-space surfaces, across empty space, and, in hybrid workflows, into seamless hand-based 3D manipulation.