Guided Reality: Adaptive Digital Guidance

• Guided Reality is a paradigm that combines real-time sensor data, adaptive control, and vision-language reasoning to provide interpretable and dynamic digital guidance.
• It leverages closed-loop optimization, markerless spatial registration, and multi-sensor fusion to align user actions with optimal trajectories and outcomes.
• Applications span clinical rehabilitation, surgical planning, AR-assisted training, and safety operations, resulting in measurable improvements in efficiency and accuracy.

Guided Reality is a paradigm and set of computational methodologies that augment real environments, actions, or feedback loops with contextually grounded, adaptive, or vision-anchored digital guidance. Unlike conventional augmented reality (AR) systems that simply overlay static information, Guided Reality integrates real-time models, sensor data, user context, and adaptive feedback to dynamically steer user behavior, task execution, clinical interventions, or collaborative processes toward target outcomes while preserving interpretability, user autonomy, and context relevance. Across domains—including clinical medicine, surgical robotics, asynchronous VR collaborations, AR-assisted training, photogrammetric outdoor safety workflows, and even diffusion-based image generation—Guided Reality frameworks mediate between user intent, environmental constraints, and idealized references by leveraging closed-loop optimization, adaptive control, or vision-language reasoning to embed guidance that is both actionable and interpretable.

1. Conceptual and Formal Definitions

Guided Reality departs from classic AR by operating as a closed-loop system that infuses dynamic, adaptive, and interpretable feedback into real-world or virtual contexts. In clinical VR rehabilitation, Guided Reality is embodied in a human–avatar control loop where the system continuously balances synchronizing with the patient’s movements versus steering toward optimal trajectories such as minimum-jerk profiles, achieved by minimizing a multi-objective cost function through finite-horizon optimal control (Lellis et al., 10 Dec 2025): $$\min_{u(\cdot)} \ \tfrac12\,\alpha_p(\theta(t_{k+1})-\theta_P(t_{k+1}))^2 + \tfrac12 \int_{t_k}^{t_{k+1}} \Big[\alpha_s (\dot\theta(\tau)-\dot\theta_H(\tau))^2 + \eta\,u(\tau)^2\Big]\,d\tau,$$ where $\alpha_p$ and $\alpha_s$ are dynamically adapted weights trading off "following" versus "guidance".

In AR task instruction, Guided Reality denotes fully automated pipelines that use LLM-generated task decompositions and vision models to identify key physical components, types of spatial guidance, and optimal embedding of 3D dynamic cues for each task step (Zhao et al., 5 Aug 2025).

The essential property is that the system interprets states or user actions in context, computes guidance as a function of both real-time and reference data, and adapts visual, haptic, or algorithmic feedback accordingly.

2. Algorithmic and Control-Theoretic Foundations

Fundamental to Guided Reality are algorithmic substrates that interleave state estimation, model fitting, vision-language grounding, and adaptive feedback:

• Control-Theoretic Guidance: In VR-based rehabilitation (Lellis et al., 10 Dec 2025), a virtual avatar is modeled as a damped harmonic oscillator parametrized by inertia ($I$), damping ($B$), and stiffness ($K$). The system computes control torque $u(t)$ under constraints:

$$I\ddot\theta(t) + B\dot\theta(t) + K\theta(t) = u(t), \ \ 0\leq\theta(t)\leq\theta_{\rm ROM},$$

optimizing over tracking error, healthy velocity adherence, and control effort, with real-time gain adaptation based on a data-driven ability index.

• Adaptive and Closed-Loop Feedback: Real-time metrics such as normalized jerk for motion smoothness are used to update the relative weights of "follow" vs. "lead" guidance, enabling interpretable adaptation that can be monitored and adjusted by clinicians via dashboards showing ability indices, weight trends, and kinematic markers (Lellis et al., 10 Dec 2025).
• Markerless Spatial Registration: Systems such as ARport (Han et al., 15 Feb 2026) achieve spatial guidance by reconstructing patient surfaces from RGB–depth–pose fusion, applying segmentation models (e.g., SAM) for mask growth and iteratively aligning preoperative CT-derived meshes using robust point-to-plane ICP:

$$T^* = \arg\min_{T\in SE(3)} \sum_{(m,o)\in \mathcal{C}(T)} \rho(n_o^T(Rm + t - o)),$$

enabling accurate in situ visualization of planned intervention sites.

• Reality-Guided Diffusion: In image super-resolution, the SARGD method (Zheng et al., 2024) defines "reality guidance" as joint artifact detection (via a learned mask), hard patch-wise latent correction based on realistic latent priors, and a self-adaptive update mechanism that uses a per-step "reality score" to refine guidance regions and suppress synthetic features.
• Vision-Language Integration: In AR task guidance (Zhao et al., 5 Aug 2025), LLMs are orchestrated via structured JSON prompts to enumerate stepwise procedures, from which visual guidance types (e.g., highlight region, movement, tool usage) are selected. Vision models (e.g., Gemini, SAM) provide component localization, and 3D geometric reasoning is used for precise spatial anchoring.

3. Domains and Application Scenarios

Guided Reality is instantiated across diverse sectors, unified by the dynamic mediation of user actions, environmental context, and reference trajectories or instructions:

• Motor Rehabilitation: Patients interact with an adaptive virtual avatar for upper-limb rehabilitation, where smoothness metrics and optimal control underpin the avatar’s movement, and gains are adjusted according to a data-driven ability index, yielding interpretable, high-fidelity guidance (Lellis et al., 10 Dec 2025).
• CT- and Ultrasound-Guided Interventions: Real-time, rigid or surface-based registration pipelines overlay essential trajectory information, anatomical models, or target markers onto the patient, significantly reducing procedure time, needle passes, radiation exposure, and error rates, while also standardizing operator performance (Park et al., 2020, Song et al., 23 Mar 2026).
• Markerless Surgical Planning: ARport demonstrates fully markerless port placement by aligning preoperative trocar sites directly on the patient using surface-based pose estimation, eliminating hardware, fiducials, or manual registration steps (Han et al., 15 Feb 2026).
• Collaborative VR/AR Training and Tours: In asynchronous manufacturing inspection or technical training, Guided Reality systems ensure that observer engagement and spatial recall are maximized via interactive mimicry, viewpoint autonomy, and context-sensitive pausing, yielding higher recall and reduced cognitive load versus passive systems (Giovannelli et al., 2 Feb 2025).
• Backcountry Safety and Photogrammetric Mapping: Guide-skier communication is enhanced by enabling the guide to annotate a photogrammetry-derived 3D landscape with hazards and zones, then relay those overlays into skiers’ in situ HUDs—embedding safety protocols directly into the user’s perceptual field (Johns et al., 25 Feb 2026).

4. Visual and Spatial Guidance Typologies

Guided Reality applications frequently taxonomize visual augmentation and feedback mechanisms according to instructional, cognitive, or ergonomic goals:

Visual Guidance Type	Example Application	Implementation
Component Highlight (Type 1)	AR step-by-step instructions (Zhao et al., 5 Aug 2025)	3D bounding box anchor
Movement Indication (Type 2)	Machine part translation/rotation	Animated arrows, motion planes
Hand Gesture Demonstration (Type 3)	"Press here," "twist this"	3D hand pose model overlay
Tool Usage (Type 4)	Wrench, cloth, screwdriver steps	3D tool mesh, oriented on surface
Widget Display (Type 5)	Timers, status indicators	2D UI anchored in 3D

This structured approach enables automated decision logic (e.g., via LLM rule templates) and modular rendering pipelines that combine machine vision outputs with semantic interpretation for task-specific augmentation (Zhao et al., 5 Aug 2025).

5. Technical Assessments and Measured Benefits

Guided Reality frameworks consistently demonstrate improved technical and human factors performance:

• Motor Control: Adaptive VR avatar guidance increases motion smoothness (ability index $I_A$ rising from 0.60 to 0.90 in simulation), adapts challenge level ($\alpha_p$0), and allows clinician interpretable adjustments (Lellis et al., 10 Dec 2025).
• Procedural Efficiency: AR systems guide needle placement for out-of-plane lesions, cutting mean needle passes from 7.4 to 3.4 ($\alpha_p$1, $\alpha_p$2) and DLP (radiation dose) from 538 to 318 mGy·cm ($\alpha_p$3, $\alpha_p$4), while eliminating complications (Park et al., 2020).
• Registration Precision: Marker-based and markerless spatial registration for augmented overlays achieves sub-centimeter accuracy (TRE ≈ 6 mm for navel region (Han et al., 15 Feb 2026); real-time marker registration error $\alpha_p$5 mm (Cao et al., 2019)) and sub-2° mean angular error in implant orientation (Fotouhi et al., 2020).
• Task Guidance Accuracy: Automated AR task instruction pipelines correctly produce visually enriched step guidance in 65% of real-world queries (with step-type, key component, and instructional accuracy at 90–97%) and reduce error rates during tasks from 17.1% to 10.4% (Zhao et al., 5 Aug 2025).
• Subjective and Cognitive Gains: In AR-guided robotic ultrasound, execution time and needle placement error decreased by 49% and 25% respectively. Usability, trust, spatial understanding, and workload rating improvements were all statistically significant [$\alpha_p$6, (Song et al., 23 Mar 2026)].
• Diffusion-Based Image Guidance: Reality-guided diffusion methods for super-resolution suppress artifacts and improve PSNR by up to 3.86 dB over state-of-the-art methods, converging in half the number of diffusion steps ($\alpha_p$7100 steps) (Zheng et al., 2024).

6. Limitations, Challenges, and Future Directions

Current challenges concern markerless or cross-modality registration robustness, handling of occlusions, scalability of vision-language pipelines to out-of-view components, usability in unconstrained outdoor deployments, and the integration of deformable models or haptic feedback.

A plausible implication is that future Guided Reality systems will need to unify multi-sensor fusion (e.g., RGB, depth, IMU, ambient light), adaptive model-based control, and intelligent bi-directional user interaction, while generalizing to more complex, deformable, and dynamically changing environments.

7. Broader Impacts and Generalization

Guided Reality refines the foundational interaction paradigm between human users, complex tools, and real or virtual environments by instrumenting context-aware, adaptive, and interpretable feedback directly into the action-perception loop. Applications range from remote clinical care, image-guided therapy, and collaborative task training to environmental safety and computer vision-guided generation. Its defining features—closed-loop adaptivity, dynamic real-world anchoring, and multi-layered perceptual augmentation—set it apart as a generalizable framework for embedding intelligence and guidance into physical and digital realities, enabling new operational efficiencies and user experiences across technical domains (Lellis et al., 10 Dec 2025, Zhao et al., 5 Aug 2025, Park et al., 2020, Han et al., 15 Feb 2026, Zheng et al., 2024, Giovannelli et al., 2 Feb 2025, Song et al., 23 Mar 2026, Johns et al., 25 Feb 2026, Zhao et al., 2024).