Reality Proxy: Bridging Digital and Physical Worlds

• Reality Proxy is a system that uses digital or physical stand-ins to replicate real-world objects for cross-reality interaction.
• It employs precise geometric matching, AI-driven semantic enrichment, and bidirectional transformation to ensure high-fidelity experiences in VR, AR, and MR.
• Applications range from haptic feedback in VR to cognitive state monitoring and adaptive interfaces in MR for enhanced engagement.

A reality proxy is a physical or digital intermediary that stands in for a real-world or virtual object, affording users a mechanism to perceive, interact with, or manipulate elements outside their direct cognitive or physical reach. In immersive technologies—including Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR)—reality proxies bridge perceptual or interactive gaps, decouple input from physical constraints, and provide tangible or abstracted surrogates for seamless cross-reality interaction. The term encompasses high-fidelity haptic proxies, abstracted physical surrogates, AI-constructed digital representations, and bidirectional transformation layers unifying the digital and physical domains (Savino, 2020, Liu et al., 23 Jul 2025, Godden et al., 2024, Datta, 2022, Kahl et al., 2023).

1. Definitions and Theoretical Foundations

A reality proxy, as first formalized in VR literature, is a physical object whose form factor, weight, size, and—where feasible—functional affordances replicate or approximate a virtual object, aiming for a one-to-one mapping between tactile sensation and visual representation (Savino, 2020). While a generic proxy object serves simply as a hand-held intermediary (e.g., a controller-embedded item), a reality proxy is distinguished by domain-specific replication—preserving the somatosensory features of its virtual analogue to maximize haptic fidelity.

The concept generalizes in MR: reality proxies may be purely digital, representing physical world objects with semantically enriched, interactable 3D abstractions that lift interaction beyond real-world constraints (e.g., distance, occlusion) (Liu et al., 23 Jul 2025). The reality proxy thus encompasses:

• Physical Stand-ins: tangible artifacts matched in geometry and affordance to virtual models (Savino, 2020, Kahl et al., 2023)
• Digital Abstractions: proxy objects in MR/AR that encapsulate real objects’ semantics, relationships, and states (Liu et al., 23 Jul 2025)
• Bidirectional Views: synchronization and mediation between physical and software representations, with transformation “hooks” enabling seamless cross-domain interventions (Datta, 2022)

2. Reality Proxies in Physical Haptics and VR

Reality proxies in VR are tangible objects that closely mimic their virtual counterparts with strict attention to geometric, weight, and functional alignment. Savino et al. (Savino, 2020) prototyped two such proxies:

• Paper Map Proxy: An A3 paper map with an HTC Vive controller for 6DoF tracking enabled intuitive grasp, rotation, and translation in Unity3D, ensuring every real movement had a one-to-one virtual correspondence.
• Smartphone Proxy: An Android phone in a rubber case, coupled with a Vive controller, streamed touch gestures (tap, pinch, rotate) to the VR environment for task execution (e.g., zoom, camera shutter).

Empirical observations indicate precise reach and manipulation even without users seeing their hands in VR, provided that the real and virtual objects are dimensionally matched. Savino et al. emphasize that even minor mismatches (on the order of centimeters) can degrade the reliability of reach, grasp, and gestural input, supporting earlier findings (Kahl et al., 2023).

A core design tension emerges: high-fidelity, domain-specific proxies maximize tactile realism at the cost of adaptivity, whereas generalized adaptive haptic devices offer flexibility but with lower fidelity match to any given object (Savino, 2020).

3. Abstracted and Digital Reality Proxies in MR/AR

Digital reality proxies abstract away physical instantiation, serving as AI-enriched representations for MR interaction. In the Reality Proxy system (Liu et al., 23 Jul 2025), proxies are dynamically instantiated abstract objects that overlay the physical scene, enabling users to interact with distant, crowded, or occluded real objects via familiar gestures (pinch, brush, zoom) in an optimized proxy-space. This is achieved through a three-stage pipeline:

1. Activation: Gaze+Pinch triggers scene understanding with hierarchical detection (DINO-X) and attribute extraction (GPT-4o).
2. Proxy Generation: Bounding box centers are raycast to the scene mesh, with constraint optimization preserving relative spatial relationships while placing proxies in an ergonomically accessible layout.
3. Interaction: Gestures manipulate proxies to effect operations such as skimming (attribute preview), multiple selection, semantic filtering, and group navigation.

AI inference for both hierarchical detection and attribute extraction enables semantic enrichment (e.g., “all objects colored red,” “battery=full”). Interaction decouples physical limitations from task logic, expanding the possible manipulation repertoire without requiring new gestures or menus.

4. Fidelity, Abstraction, and Performance Constraints

The degree to which a physical or digital surrogate can diverge from its counterpart without loss of usability or presence has been systematically explored. Kahl and Krüger (Kahl et al., 2023) tested proxies for an AR sofa model at five abstraction levels, from exact 3D-printed replica to planar board:

• Outcomes: Only proxies with coarse shape similarity (up to ≈1.5× linear dimension deviation) retained presence and performance; more abstract shapes (cuboid, plane) yielded decreased AR presence, higher disturbance, and prolonged task time.
• Guidelines: Maintain coarse affordance cues. For furniture, up to 52% height and 67% depth deviation had no significant impact; cuboid or planar proxies are detrimental for tasks requiring nuanced shape cues.

For high-fidelity VR proxies, system-level latencies (VR tracking, touch network, rendering) must remain under 20 ms to sustain the illusion of real haptics (Savino, 2020). In robotic proxy systems, hand-tracking latency of ~28 ms (steady state) and positional errors of ~1.3 cm begin to erode the seamless linkage between user action and perceived feedback, evidencing the technical sensitivities of such integrations (Godden et al., 2024).

Proxy Type	Fidelity Domain	Best-Suited Modalities
Physical replica	High	VR, haptic precision tasks
Physical abstraction	Moderate	AR with generic affordances
Digital abstraction	Flexible	MR with semantic/intelligent control

5. Cross-Reality Transformation and Proxy Frameworks

Beyond local surrogate objects, cross-reality frameworks implement reality proxies as bidirectional transformation layers. The system described in (Datta, 2022) frames reality proxies as mappings between synchronized streams of physical world images and digital interface buffers:

• Formal Model: The state at time t is $S(t) = (I_{\mathrm{phy}}(t), I_{\mathrm{dig}}(t))$, with a set of intervention hooks $\mathcal{H}$ transforming the observed state for the user.
• Intervention Types: Mask hooks (template region occlusion), text hooks (OCR and overlay/censoring), and model hooks (learned classifiers for object or text regions) compose transformations for real-time, interactive alteration of both physical and digital views.
• Applications: Individual and collaborative workflows enable strategic manipulation of perception in either domain, with cognitive walkthroughs, field personas, and scalability trials demonstrating robust generalization.

This bidirectional paradigm extends the reality proxy notion to include not only tangible or digital stand-ins for objects but also the composable mechanisms by which cross-domain perception and intervention are made seamless.

6. Proxies as Behavioral Indicators and System Adaptation

Beyond tangible and semantic mediation, proxies serve as metrics for latent states (e.g., user presence, cognitive load) in immersive settings. Chandio et al. (Chandio et al., 2024) demonstrate that reaction time (RT) to simple probes in MR can act as a live proxy for immersion and presence. Their cognitive distraction model for MR shows:

• Mediation chain: Cognitive load—imposed by incongruent distractions—reduces presence, which in turn slows RT. This mediation is formalized as:

$$P = a \cdot \mathrm{CL} + \epsilon_1, \quad RT = b \cdot P + c' \cdot \mathrm{CL} + \epsilon_2$$

where $a = -0.44$, $b = -1.78$, $c' = 1.68$.

• Practical implications: MR systems can embed lightweight behavioral probes (e.g., rapid virtual tasks) to index immersion continuously and non-intrusively, informing adaptive interventions to minimize breaks in presence.

A plausible implication is that reality proxies, physical or virtual, should be optimized not only for interaction fidelity but also for their capacity to convey real-time user state back to the adaptive MR system.

7. Limitations, Trade-offs, and Future Directions

Reality proxies, in their various forms, face inherent trade-offs and technical limits:

• Adaptivity–Fidelity Trade-off: Domain-specific proxies offer high realism at the cost of versatility; abstracted proxies generalize but risk degraded affordance match (Savino, 2020, Kahl et al., 2023).
• Technical Limitations: Vision-only scene understanding misses fully occluded or sub-resolution objects; robotic proxies suffer from latency and control-loop delay (Liu et al., 23 Jul 2025, Godden et al., 2024).
• Cognitive Overhead: For digital proxies, users must cognitively map proxy-space to real-space, especially as abstraction levels increase (Liu et al., 23 Jul 2025).
• Collaboration and Scalability: Group-sourced annotation and intervention scale the utility of proxy frameworks, but collaborative consistency remains a challenge (Datta, 2022).

Future research directions include dynamic and shape-aware proxy generation, hybrid AI–human correction loops, continuous adaptation for moving/teleoperated objects, integration of physiological sensing for presence estimation, and systematic performance benchmarking of proxy-based versus direct interaction modalities across application verticals (Liu et al., 23 Jul 2025, Chandio et al., 2024).

---

References:

(Savino, 2020) Virtual Smartphone: High Fidelity Interaction with Proxy Objects in Virtual Reality (Liu et al., 23 Jul 2025) Reality Proxy: Fluid Interactions with Real-World Objects in MR via Abstract Representations (Godden et al., 2024) Towards Robotic Haptic Proxies in Virtual Reality (Datta, 2022) Cross-Reality Re-Rendering: Manipulating between Digital and Physical Realities (Chandio et al., 2024) Reaction Time as a Proxy for Presence in Mixed Reality with Distraction (Kahl et al., 2023) Using Abstract Tangible Proxy Objects for Interaction in Optical See-through Augmented Reality