Augmented Virtual Environments (AVEs)

• Augmented Virtual Environments (AVEs) are virtual spaces enriched with real-world geometry, sensor data, and contextual signals to bridge the gap between reality and virtuality.
• AVE architectures use sensor fusion, semantic enrichment, and bidirectional synchronization to empower diverse applications from live-action VR games to smart collaborative workspaces.
• Current research addresses challenges such as registration coherence, selective information rendering, and adaptive control to balance immersion, safety, and usability.

Augmented Virtual Environments (AVEs) are virtual environments enriched by real-world geometry, physical objects, sensor data, contextual signals, or live links to physical systems. In the cited literature, AVEs appear under several closely related formulations, including augmented virtuality, hybrid reality environments, XR-IoT systems, and adaptive virtual spaces. The common structure is that the virtual environment is primary, but it is not isolated: it is constructed from, aligned to, or dynamically coupled with the physical world, and it often supports reciprocal interaction across the virtual–physical boundary (Valente et al., 2016, Praeger et al., 2020, Guan et al., 2023, Beale et al., 17 Sep 2025).

1. Conceptual scope and terminology

Within the reality–virtuality continuum, AVEs correspond most directly to the mixed-reality region in which reality and virtuality are combined, especially on the augmented virtuality side where the virtual world is augmented by real-world information rather than the real world being overlaid with virtual content. The literature summarized here repeatedly distinguishes this from conventional augmented reality: in AR, the user remains primarily situated in the real environment; in AVEs, the user is primarily situated in a virtual environment whose geometry, semantics, state, or interaction logic depends on physical reality (Chen et al., 2018, Valente et al., 2016).

This distinction is explicit in work on live-action virtual reality games, where the “game stage” is classified as augmented virtuality because the physical room becomes the spatial scaffold for the virtual world, physical objects are mapped to virtual elements, and contextual information from the environment participates in gameplay. The player sees only the virtual representation through the HMD, yet the virtual world remains superimposed on the real one through physical movement, tangible interaction, and context-aware sensing (Valente et al., 2016). A closely related framing appears in the Extended Metaverse, where virtual and physical objects are treated as a hybrid construct linked by XR and IoT, with “shared objects” maintaining synchronized state across both domains (Guan et al., 2023).

Other papers broaden the concept further. Adaptive Virtual Neuroarchitecture (AVN) defines a dynamic and reciprocal connection between physical space, virtual space, and the user, emphasizing that the environment may respond not only to real geometry but also to physiological, cognitive, and emotional state (Jain et al., 2022). In a different register, Dataspace presents a room-sized hybrid reality environment in which movable displays, a projected table, AR headsets, VR participation, and a live 3D room model form a unified collaborative workspace (Cavallo et al., 2019). A plausible implication is that AVE is best understood as a systems category rather than a single interface style.

2. Core architectural patterns

A recurring architectural pattern is the maintenance of a virtual representation that is continuously informed by physical-world state. In the mobile-image AVE pipeline, the system fuses mobile phone images, EXIF metadata, OpenStreetMap (OSM) building data, and DTM/DEM terrain data to construct a Unity-based 3D scene. Python reads EXIF data, queries OSM, converts latitude/longitude to UTM, normalizes coordinates relative to an anchor point, and exports JSON for Unity; Unity then instantiates buildings, terrain, and projective image mappings, while UDP-based two-way communication keeps processing and visualization synchronized (Beale et al., 17 Sep 2025). The environment is thus neither a purely synthetic model nor a direct camera passthrough, but a navigable virtual reconstruction with real imagery embedded into spatial context.

A second pattern is state synchronization between virtual and physical objects. In the Extended Metaverse prototypes, a physical IKEA lamp connected through an Orbecco smart plug and Smart Life app is represented by a virtual lamp in Unity and HoloLens 2; virtual manipulation changes the real lamp state, and physical changes update the virtual state accordingly. The ambient-lighting prototype extends this to Philips Hue color bulbs, where collisions between a virtual rocket and virtual planets determine the color of physical room lighting (Guan et al., 2023). The underlying architectural claim is that cohesiveness depends on bidirectional coupling, not merely on displaying physical information inside a virtual scene.

A third pattern is the use of semantic enrichment to make the environment computationally meaningful. In procedurally generated virtual cities, semantics are embedded during city generation through constructs such as entrance("btype"), zone("ztype"), and @Object("otype"). These semantic tags are later consumed by the crowd system to generate agendas for adults, children, and elders, enabling agents to query meaningful destinations such as homes, schools, workplaces, parks, and benches (Rogla et al., 2018). This suggests that AVEs often depend as much on representational structure as on display technology.

3. Embodiment, interaction, and locomotion

AVE interaction research frequently focuses on how users act on virtual content while preserving physical plausibility and situational safety. The Gesture Authoring Space (GAS) addresses this by allowing users to create object-specific grasp gestures directly in VR: the user places the hand near an object, forms the intended grasp, and holds the pose still for 3 seconds while the system captures the hand pose. Recognition uses template matching over 25 hand joints with similarity

$S_g = \sum_{i=1}^{25} \sqrt{(j_c - j_{gt})^2},$

and a gesture is recognized if the combined joint distance does not exceed 5 cm (Schäfer et al., 2022). The broader AVE implication is that object-linked gestures can replace a universal pinch model when naturalness is more important than maximal speed.

Cross-reality legibility raises a different interaction problem: how much of a virtual world should be shown to an observer situated in AR? SelectVisAR studies this in co-located AR/VR settings through conditions such as Everything, Interactive, Proximity, Dollhouse, and later Context and Spotlight. Its central result is that event recognition remained comparable even when large parts of the virtual environment were filtered out, while qualitative feedback showed that spectators still required enough spatial reference to avoid disorientation (Cools et al., 2021). This directly reframes AVE design as an information-filtering problem rather than a default requirement to render the entire virtual scene.

Locomotion techniques show the same concern with balancing embodiment and physical constraints. CWIP-AVR combines Natural Walking (NW) and Walking in Place (WIP), switching among Stationary, NW, and WIP states using the horizontal position of the user’s Chest joint and a threshold of $V_t = 0.80\,\text{m/s}$. The AVR layer augments the VE with virtual indicators, visual markers, and sound alerts representing real-world hazards such as walls, chairs, and obstacles, organized into Normal, Pre-Warning, Warning, and Danger zones (Sousa et al., 2019). Outpace Reality solves a related scaling problem by placing the user inside a short virtual tunnel whose exterior spans the full route while its interior requires only

$\frac{d(p_s,p_e)}{a_g}$

of real walking, with tunnel walls reducing problematic optical flow and window slits preserving some perception of travel (Cmentowski et al., 15 Aug 2025). In workspace-occlusion research, the problem is not travel distance but physical accessibility: when virtual content blocks a real desk, either Redirected Walking or Automatic Teleport Rotation is used to restore visibility and reachability of the physical workspace (Feld et al., 13 May 2025).

The same general principle appears in HRI. HAVEN is a Unity-only virtual robot environment that supports AR-style HRI studies in VR by rendering robot intent and internal state through path projections, turn signals, thought bubbles, and battery indicators over a virtual TurtleBot2-like agent. Although the environment is fully virtual, the visualization logic reproduces the sort of augmentations normally studied with physical robots in AR (Cleaver et al., 2020).

4. Adaptive and autonomous AVEs

Several papers move from static augmentation toward AVEs as adaptive control systems. In Adaptive Virtual Neuroarchitecture, the environment is not merely a rendered space but a dynamic and reciprocal loop among space, objects, user state, and the virtual world. The paper distinguishes Adaptive Spaces, where physical and virtual environments interact structurally, from Adaptive Experiences, where the virtual world is driven by physiological or psychological constructs such as respiration, mood, or anxiety (Jain et al., 2022). Examples include systems that reconstruct the physical room and map it into VR, robotic walls that change enclosure, and biofeedback environments in which breathing modifies virtual motion dynamics.

A more operational version of this logic appears in the optical-tweezers study. A reinforcement-learning policy is trained in a virtual optical-tweezers “gym” and then transferred directly to the physical apparatus without retraining. The key AVE move is the augmented physical environment: virtual obstacles are overlaid on the physical microscope scene, and the trained policy navigates around both real microspheres and virtual maze-like obstacles because it has learned to treat virtual and physical observations equivalently (Praeger et al., 2020). Here the AVE is not an interface for a human observer but a hybrid perception-action space for an RL agent.

Autonomy can also be embedded at the environmental level. In semantically augmented virtual cities, individual inhabitants and households receive agendas through a grammar-driven system that combines person attributes with city semantics. The framework models households as

$H=\sum^{n}_{i=1}\iota_i$

and the total population as

$\Pi=\sum^{m}_{j=1}H_j,$

then assigns routines such as commuting, school accompaniment, park visits, and bench use according to the available semantic structure of the city (Rogla et al., 2018). A plausible implication is that AVEs become more behaviorally coherent when semantics are shared across both geometry generation and agent planning.

The Extended Metaverse literature generalizes this into an Extended Metaverse Agent framework derived from the MiRA mixed reality agents model and the Milgram reality-virtuality continuum, with the dimensions Mixed Reality Embodiment, Extended Interaction, and Level of Agency. This emphasizes that AVEs may contain not only synchronized objects but also autonomous or semi-autonomous entities acting across both domains (Guan et al., 2023).

5. Evaluation, presence, and cost-benefit

AVE evaluation in the cited literature is notably heterogeneous. One strand uses an information-theoretic framework. Cost-benefit Analysis of Visualization in Virtual Environments treats VE use as a conditional trade-off, summarized conceptually as

$$\frac{\text{Benefit}}{\text{Cost}} = \frac{\text{Alphabet Compression} - \text{Potential Distortion}}{\text{Cost}}.$$

Within this account, AVEs are beneficial when overlap between virtual and real environments increases mutual information and reduces distortion, but they are not automatically superior to conventional displays because they can impose economic, computational, interactional, and cognitive costs (Chen et al., 2018). This is one of the clearest statements against the misconception that more immersion or more augmentation is inherently better.

Psychological evaluation introduces a second strand. Presence in AR is defined not as “being there” in a virtual place, but as the impression that virtual objects are present in the real environment. Factor analysis on the questionnaire data yielded two interpretable dimensions, realness and spatial presence, while perceptual stress emerged as a weaker preliminary factor (Regenbrecht et al., 2021). For AVEs more broadly, this implies that successful integration is not exhausted by geometric registration; users must also experience the augmented content as spatially located and coherently integrated.

Task-oriented studies often report more equivocal results. In immersive analytics, AR and VR produced comparable objective performance, but users demonstrated a higher tolerance for cognitive load in VR and some participants traveled farther in AR; the recommendation was therefore to support switching between AR and VR rather than enforcing a single mode (Zhou et al., 2024). In multisensory fire training, adding heat and smell to an audio-visual VE increased belief that the building was on fire and produced more realistic behavior near hazards, yet it did not improve conventional knowledge-test performance over AV-only VE in a later training study (Lawson et al., 2019). In gesture interaction, controllers remained faster and often more accurate than hand-based methods, even though object-specific custom gestures were rated as more natural (Schäfer et al., 2022). These studies collectively show that AVE benefits depend strongly on the dependent variable being measured.

6. Application domains, controversies, and future directions

The application range of AVEs in this literature is unusually broad. In scientific and engineering settings, AVEs support experimental control in optical tweezers (Praeger et al., 2020), seismic hazard and risk visualization in a CAVE environment that links ShakeOut ground motions, fragility functions, and “Drop, Cover, and Hold On” guidance (Nikbakht, 2019), and collaborative information analysis in Dataspace through 15 OLED 4K displays, robotic arms, a projected table, AR headsets, VR participation, and mobile devices (Cavallo et al., 2019). In urban and investigative settings, mobile photographs can be fused with OSM and terrain models to improve spatial interpretation, event ordering, and contextual understanding (Beale et al., 17 Sep 2025). In smart-environment research, XR-IoT prototypes turn lamps and ambient lighting into synchronized shared objects across physical and virtual space (Guan et al., 2023).

The literature also identifies several persistent controversies. One is match or coherence: poor registration between physical and virtual components increases distortion, cognitive load, and in some cases safety risk (Chen et al., 2018, Feld et al., 13 May 2025). Another is information surplus: rendering everything available in the virtual scene may overwhelm an AR observer, making selective visualization preferable (Cools et al., 2021). A third concerns agency and ethics in adaptive environments. AVN explicitly notes that if a system detects a user’s mood or physiological state, the direction of adaptation should often belong to the user rather than the developer, because environment-driven modulation of psychological state can become manipulative (Jain et al., 2022).

Future directions in the cited work converge on deeper coupling rather than greater visual complexity alone. The Extended Metaverse proposes context-driven content, richer agency, and procedural or generative design including L-systems (Guan et al., 2023). The mobile-image AVE work identifies automated extrinsic calibration through AI-based scene recognition or sensor fusion as an open problem (Beale et al., 17 Sep 2025). Safe locomotion and occlusion research indicate that the physical workspace should be treated as a protected resource whose visibility and reachability must be actively maintained (Sousa et al., 2019, Feld et al., 13 May 2025). A plausible synthesis is that the research trajectory points toward AVEs as living, instrumented, semantically structured environments in which perception, action, adaptation, and physical context are jointly designed rather than separately layered.