Extended Reality (XR) Overview

• Extended Reality (XR) is a term for immersive technologies that blend physical and virtual worlds, including VR, AR, and MR.
• XR systems integrate advanced hardware, sensor arrays, and AI-driven software to achieve real-time spatial sensing and adaptive interactions.
• XR transforms industries such as healthcare, education, and manufacturing by enabling applications like remote experiments, virtual prototyping, and adaptive user interfaces.

Extended Reality (XR) is a collective term encompassing technologies that merge real and virtual environments, including Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). XR serves as a foundational paradigm for immersive, interactive experiences across entertainment, healthcare, manufacturing, education, and scientific research. XR systems leverage a spectrum of hardware, sensor arrays, network services, and AI-driven software to create adaptive digital experiences that integrate physical, virtual, and social dimensions, thus enabling new forms of human–environment and human–machine interaction.

1. Conceptual Framework and Taxonomies of Extended Reality

XR is defined by an overarching framework that interpolates between fully physical and fully virtual worlds, sometimes referred to as the "reality–virtuality continuum" (Mann et al., 30 Dec 2024, Zeng, 22 Apr 2025, Wang et al., 27 Mar 2025). The “Socio-Cyber-Physical Taxonomy” formalizes XR as occupying, and extrapolating, a three-dimensional space comprising physical (atoms), virtual (bits), and social (genes) domains (Mann et al., 30 Dec 2024). In this taxonomy:

• Physical Reality is limited to the atoms axis.
• Virtual Reality (VR) is located on the bits axis.
• Augmented Reality (AR) lies in the plane between physical and virtual.
• Mixed Reality (MR) serves as a continuum allowing co-presence and real-time interaction of digital and physical objects.

The scope of XR deliberately extends beyond classic AR/VR to encompass Diminished Reality (DR)—where sensory information is attenuated or reduced—as well as Intelligent Reality (IR), with the possibility for AI-driven contextual augmentation of perception, cognition, and agency (Mann et al., 30 Dec 2024).

2. XR System Architectures and Technology Stack

XR systems are built upon sophisticated hardware and layered software stacks (Zeng, 22 Apr 2025, Wang et al., 27 Mar 2025).

• Hardware components include head-mounted displays (HMDs), micro-OLED/LED displays (driving resolution, field-of-view, and refresh rate), sensor arrays (RGB cameras, LiDAR, structured light, inertial measurement units), and auxiliary controllers.
• Sensor fusion integrates IMUs, hand tracking, eye tracking, and environmental depth sensors to deliver real-time spatial sensing and user context acquisition (Gunkel et al., 2022).
• Software stacks include visual algorithms for spatial computing—simultaneous localization and mapping (SLAM), 3D reconstruction, object detection/recognition—and UI/UX layers incorporating rendering engines, spatial/ambient audio, haptic systems, and multi-modal input (gestures, gaze, voice, BCI).
• Network integration leverages low-latency, high-throughput communication (notably 5G/6G) and edge/cloud computing paradigms to offload computationally intensive tasks and enable distributed experiences (Heo et al., 2023, Gapeyenko et al., 2022).

A simplified diagram representing XR's foundational transformation pipeline is: $$\begin{array}{c} \text{Physical World} \ \downarrow \ \text{Sensors (Cameras, LiDAR, IMUs)} \ \downarrow \ \text{Visual Algorithms (SLAM, 3D Reconstruction)} \ \downarrow \ \text{UI/UX (Rendering, Interaction Engines)} \ \downarrow \ \textbf{Digital/XR Experience} \end{array}$$ (Zeng, 22 Apr 2025)

3. Interaction Models, User Experience, and Universal Interfaces

XR interaction paradigms draw on a rich set of modalities: hand controllers, mid-air gestures, voice (NLP), gaze, and auxiliary mobile interfaces (smartphones, smartwatches) (Knierim et al., 2023). A proposed “universal interaction concept” centers on hybrid user interfaces, cohesive multimodal fusion, and adaptive systems that consider environmental constraints (physical space, context), task properties (complexity, duration), and user factors (proficiency, comfort).

Design challenges include managing physical/ergonomic fatigue (e.g., the "gorilla-arm effect" in gesture input), consistent multimodal feedback, cross-device interoperability, and user-centric adaptation (Knierim et al., 2023).

The user experience (UX) in XR can be formally conceptualized as: $\text{UX} = f(E, T, U)$ where $E$ = environment, $T$ = task, $U$ = user (Knierim et al., 2023).

4. Streaming, Traffic, and Network Optimization for XR

XR experiences, especially those relying on streaming (e.g., cloud gaming, remote rendering), generate unique traffic patterns distinct from conventional multimedia (Wang et al., 27 Mar 2025). These are characterized by high data rate, strict latency/jitter constraints, bi-directional real-time transmission (with frequent, bursty video frames and pose/control updates), and heavy uplink/downlink bandwidth requirements. For instance, immersive 360° XR can require bandwidths in the order of terabits per second.

Optimization strategies include:

• Foveated and tile-based streaming—delivering high-resolution content only within the current gaze region or viewport, using predictions via classical (linear regression) or deep learning models (CNN/LSTM/Transformer) (Wang et al., 27 Mar 2025).
• Adaptive reinforcement learning approaches for real-time codec and bitrate adaptation, employing multi-agent models with attention action mechanisms to minimize delay, jitter, and packet loss (Iturria-Rivera et al., 24 May 2024).
• Network-level enhancements—5G/6G New Radio (NR) features such as split rendering, viewport-dependent streaming, PDU set–aware scheduling, application-adaptive buffer status reporting, and two-stage control channel design (Gapeyenko et al., 2022, Esswie et al., 2023, Gapeyenko et al., 1 Dec 2024).

State-of-the-art simulation studies demonstrate that these methods can raise XR network capacity by up to 33% and reduce packet loss ratio by over 50% compared to baseline adaptive methods (Paymard et al., 2022, Iturria-Rivera et al., 24 May 2024, Gapeyenko et al., 1 Dec 2024).

5. Applications Across Research, Healthcare, Industry, and Sustainability

XR underpins a broad range of advanced applications:

Research and HCI: XR systems enable remote experimentation, leveraging built-in sensor suites for data collection (e.g., head/hand/gaze tracking) and encapsulated experiment design for ecological validity (Ratcliffe et al., 2021). Sensor variance introduces a statistical variance component in measurement ($$\sigma^2_{\text{total}} = \sigma^2_{\text{true}} + \sigma^2_{\text{hardware}}$$), necessitating robust experimental controls.

Accessibility and Assistive Technology: XR is deployed for simulation and augmentation in cases of vision loss, enabling both perceptual experiments under controlled impairments and assistive devices for residual/prosthetic vision (Kasowski et al., 2021). Gaze-contingent displays, simulated prosthetic vision, and dynamic task environments are core methodologies.

Manufacturing/Sustainability: XR contributes to sustainable manufacturing via virtual prototyping, layout optimization, remote training, and data monitoring for emissions and resource consumption (Cao et al., 2023). Use cases mapped to NIST environmental indicators demonstrate both direct (quantitative material/resource reduction) and indirect (improved quality, lowered error rates) sustainability benefits.

Healthcare and Cognitive Science: XR supports advanced cognitive assessment and rehabilitation through ecologically valid simulations. Multimodal data streams (EEG, GSR, eye/hand tracking) allow for real-time adaptation and continuous monitoring, with proven benefits over conventional discrete tests—however, challenges remain in hardware cost, usability, and cybersickness (González-Erena et al., 14 Jan 2025).

6. Software Engineering, Datasets, and Testing Methodologies

XR software poses unprecedented testing challenges due to six degrees of freedom (6DOF) interactions, infinite scene variability, and the complexity of scene graphs (Gu et al., 15 Jan 2025). Key findings include:

• Difficulty in formalizing user interaction sequences and automating test input and oracles for XR.
• Machine learning–based, model-based, and agent-based testing techniques are emerging but require specialized evaluation metrics that consider the full space $\mathbb{R}^3 \times SO(3)$.
• The release of XRZoo—a large-scale, cross-platform dataset (12,528 XR applications, 9 app stores, with detailed metadata)—provides a resource for benchmarking, automated test generation, and security/privacy auditing (Li et al., 9 Dec 2024).

A dedicated repository aggregates tools such as Unity Test Framework (UTF), agent-based toolkits (iv4XR), and test-specific datasets for community use (Gu et al., 15 Jan 2025).

7. Future Directions and Integration with AI

AI is essential for the next generation of XR, both in spatial intelligence (real-time environmental understanding) and adaptive interaction (Zeng, 22 Apr 2025). The future vision sees:

• Multi-modal, context-aware AI models leveraging digital twins (IoT-driven) to enable persistent adaptive spaces.
• AI-driven wearable devices (smart eyeglass, XR-integrated sensors) extending human sensory and cognitive capacity.
• New research frontiers including Diminished Reality for sensory filtering/attenuation, and synthetic synesthesia for cross-modal augmentation (Mann et al., 30 Dec 2024).
• Advanced test automation, standardization for universal interface design, and deeper integration of XR environments into AI agent ecosystems through frameworks such as XARP Tools (Caetano et al., 6 Aug 2025).

Ongoing standardization within 3GPP (Release 18/19) targets further enhancements in system capacity, latency reduction, energy efficiency, and application-aware scheduling across both core and radio access network layers (Gapeyenko et al., 1 Dec 2024, Esswie et al., 2023).

---

XR, as both technology and unifying concept, is rapidly evolving to underpin ubiquitous intelligent computing platforms. By integrating hardware, network, software, AI, and user interface advances, XR continues to redefine the intersection of digital and physical realities, with broad implications for both scientific research and industry.