VR-Drive: Immersive & Robust Driving Systems

• VR-Drive is a dual-use concept that spans immersive VR driving simulation for human-in-the-loop training and viewpoint-robust autonomous driving frameworks.
• The VR simulation aspect emphasizes safety, cost-effective driver training, and controlled evaluation of take-over performance under rare or adverse conditions.
• Its autonomous framework employs feed-forward 3D Gaussian splatting and view synthesis to enhance planning and adapt to diverse camera viewpoints.

VR-Drive denotes two closely related research usages. In one usage, it refers to portable, immersive driving systems built around head-mounted displays, steering interfaces, and simulated traffic environments for studying driver behavior, takeover performance, rare events, and in-vehicle interaction (Sportillo et al., 2018, Thalapanane et al., 2024). In another, more model-centric usage, it names a viewpoint-robust end-to-end autonomous driving framework that couples planning with view synthesis through feed-forward 3D Gaussian Splatting and explicitly targets camera-viewpoint variation (Cho et al., 27 Oct 2025). Taken together, the term spans human-in-the-loop virtual driving, mixed-reality automotive experimentation, and learned driving systems that use synthesized or reconstructed views as a planning substrate.

1. Terminology and research scope

The simulator-centered literature uses VR-Drive as a practical concept for driving in virtual or mixed reality under controlled conditions. Its core motivations are safety, cost, experimental control, and the ability to expose drivers or algorithms to conditions that are difficult, unsafe, or too rare to collect on public roads (Silvera et al., 2022, Cao et al., 2024). Within this literature, systems range from “light” HMD-based training stations for Level 3 take-over requests to open-source VR research platforms, compact motion-based simulators, mixed-reality cabins, and Vehicle-in-Virtual-Environment setups that couple real vehicles to virtual worlds (Sportillo et al., 2018, Ebel et al., 9 Mar 2026, Cao et al., 2024).

A second research strand uses the term directly for an autonomous-driving method. “VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splatting” proposes a framework that addresses viewpoint generalization by jointly learning 3D scene reconstruction as an auxiliary task to enable planning-aware view synthesis, supports online training-time augmentation from sparse views without additional annotations, and introduces a viewpoint-mixed memory bank and a viewpoint-consistent distillation strategy (Cho et al., 27 Oct 2025). This suggests that the label “VR-Drive” no longer refers only to virtual-reality hardware, but also to viewpoint robustness as a systems problem in end-to-end driving.

2. Take-over training in conditionally automated vehicles

A foundational simulator-oriented formulation appears in work on conditionally automated vehicles, where the system performs the dynamic driving task under specific conditions, the driver is allowed to disengage from monitoring, and the driver must regain control within a limited time window after a Take-Over Request (TOR) (Sportillo et al., 2018). The safety problem is the Transfer of Control (ToC): drivers may lose situational awareness, misunderstand the human–machine interface, or overtrust the automation.

The comparison among a User Manual, a Fixed-Base Simulator, and a Light Virtual Reality system established a canonical “light VR” design for take-over training (Sportillo et al., 2018). The User Manual used a 13.3" laptop and 8 slides with text and images. The Fixed-Base Simulator used a real car cockpit with a driving seat, dashboard, force-feedback steering wheel, pedals, a 65" plasma screen at 1.5 m, and a 9.7" tablet in the center console. The Light Virtual Reality system used a Head-Mounted Display, a racing wheel, and headphones, while reusing the same Virtual Learning Environment: a straight, 2-lane road with guardrails and no traffic, a virtual vocal assistant, an acclimatization phase, and step-by-step practice in manual driving, automated driving, and TOR situations (Sportillo et al., 2018).

The experiment used $N = 60$ in a between-subjects design. All participants first watched an introductory video, then received one training method, and finally completed a test drive on a high-end driving simulator with three TOR events: a 10-second TOR due to road narrowing from a stationary car in the right lane, a 10-second TOR due to loss of ground markings, and a 5-second TOR due to sensor failure (Sportillo et al., 2018). The principal objective metric was take-over reaction time $rt$, defined as the elapsed time from the TOR to the moment the driver takes back control. Reported mean reaction times were $\overline{rt}_{UM} = 5.15\,\text{s}$, $\overline{rt}_{FB} = 3.17\,\text{s}$, and $\overline{rt}_{LVR} = 3.16\,\text{s}$, with a significant difference between UM and the two interactive systems, and no significant difference between FB and LVR (Sportillo et al., 2018).

These findings delimit an important misconception. Full cockpit replication is not required for effective basic ToC training: the HMD-based light VR system matched the fixed-base simulator on reaction time while being portable and relatively low cost, and self-reported measures favored the HMD-based system on perceived usefulness, ease of use, pleasantness, and realism (Sportillo et al., 2018).

3. Platform architectures and hardware ecosystems

Several open or semi-open platforms define the current technical ecology of VR-Drive systems.

System	Stack	Distinctive focus
DReyeVR	Unreal Engine 4 + CARLA + HTC Vive Pro Eye + Logitech G29	Open-source VR behavioral and interaction research under \$5000 USD (Silvera et al., 2022)
WestDrive X LoopAR	Unity 2019.3.0f3 + C# + VR HMD + Fanatec CSL Elite	TOR experiments, AR/HUD interaction, eye tracking, force feedback (Nezami et al., 2020)
TRAVERSE	RoadRunner + SUMO + Unity + Meta Quest Pro + Logitech G29	Rare-event traffic simulation and autonomous-driving safety studies (Thalapanane et al., 2024)
DriveSimQuest	Unity + Meta Quest Pro + Logitech G923 + Movement SDK	Standalone multimodal tracking of gaze, face, hands, and body (Chidambaram et al., 14 Aug 2025)
MRDrive	Unity + Varjo XR-3 + real cabin mock-up + vJoy	Mixed-reality automotive HCI with physical cockpit interaction (Ebel et al., 9 Mar 2026)

DReyeVR is explicitly an open-source VR based driving simulator designed with behavioural and interaction research priorities in mind (Silvera et al., 2022). It is based on Unreal Engine and CARLA, includes eye tracking, a functional driving heads-up display, vehicle audio, custom definable routes and traffic scenarios, experimental logging, replay capabilities, and compatibility with ROS, and can be deployed for under \$5000 USD. On the simulator fidelity scale discussed in the paper, it sits in the medium fidelity range at 9/15 points (Silvera et al., 2022).

WestDrive X LoopAR is a Unity-based toolkit focused on TOR and augmented-reality HMI in highly automated vehicles (Nezami et al., 2020). Its environment covers roughly 25 square km, includes a continuous drivable route of about 11 km across mountain road, village, country road, and highway scenes, and ships with a package of 125 animated pedestrians as well as cars, trucks, and motorbikes. Critical traffic events are implemented as prefabs with start, boundary, and end triggers, making TOR logic and AR windshield overlays directly scriptable in the Unity editor (Nezami et al., 2020).

TRAVERSE combines RoadRunner for detailed 3D road networks, SUMO for microscopic traffic simulation, and Unity for 3D VR rendering on Meta Quest Pro, with Logitech G29 steering wheel and pedals and ROS/ROS2 compatibility through Unity’s ROS-TCP bridge (Thalapanane et al., 2024). It targets rare and adverse events, including a practice scenario and NHTSA-inspired pre-crash scenarios such as sudden lane changes, T-bone intersections, sudden stops, red-light runners, deer crossing, ramp merges, roundabout crashes, and jaywalking pedestrians, with each scenario designed to last about $1$–$3$ minutes (Thalapanane et al., 2024).

DriveSimQuest moves further toward a compact, standalone VR-Drive configuration (Chidambaram et al., 14 Aug 2025). Built on the Meta Quest Pro and Unity, it uses inside-out tracking, Logitech G923 wheel and pedals, Meta XR All-In-One SDK, and the Movement SDK for Unity. The system exposes eye-gaze rays, hand and full-body joint transforms, and approximately 70 facial blendshape weights in real time, and streams all tracked behavior and driving input signals over UDP to external visualization or analysis tools (Chidambaram et al., 14 Aug 2025).

MRDrive occupies a mixed-reality position rather than a purely virtual one (Ebel et al., 9 Mar 2026). It couples a Varjo XR-3 with a real vehicle cabin or cabin mock-up, SteamVR tracking, real steering wheel and pedals, an instrument cluster, and a 17-inch center stack touchscreen. In its reference setup, the outside world is virtual while the immediate cockpit region is shown via passthrough, allowing participants to interact with real cabin components while remaining immersed in a virtual traffic environment (Ebel et al., 9 Mar 2026).

4. Behavioral instrumentation and human-factors evidence

A defining feature of VR-Drive research is dense multimodal instrumentation. DReyeVR packages eye-tracking data from the Vive Pro Eye into a custom CARLA sensor, including gaze direction vectors, pupil diameter and position, eye openness, head pose, steering angle, brake and throttle values, button events, and world state, with asynchronous eye-tracking sampling up to 120 Hz (Silvera et al., 2022). WestDrive X LoopAR similarly integrates Tobii XR SDK and SRanipal, uses a dedicated validation scene, and accepts validation only when error angles do not exceed $1^\circ$ and head motion does not exceed $2^\circ$ from the fixation cross (Nezami et al., 2020).

TRAVERSE extends instrumentation toward rare-event behavioral analysis. It records timestamps, car coordinates, steering angle, brake usage, gas pedal usage, vehicle acceleration and speed, eye gaze position, eye openness, blink frequency, and head pose (Thalapanane et al., 2024). This design supports derivation of reaction time to hazard, braking profile, trajectory deviation, and, where desired, metrics such as $\text{TTC}(t) = \frac{d(t)}{v_{\text{rel}(t)}}$ and $rt$0, both explicitly discussed in the paper as common safety indicators (Thalapanane et al., 2024).

Human-factors evidence shows that portable VR systems can be competitive with or preferable to alternative setups. In TRAVERSE’s within-subject comparison with DReyeVR, involving 31 participants and 5-minute sessions per simulator, total sickness, oculomotor, and disorientation scores were lower for TRAVERSE, while nausea levels were comparable (Thalapanane et al., 2024). The same study reports significant advantages for TRAVERSE on sense of being in VR ($rt$1, $rt$2), ease of adjustment ($rt$3, $rt$4), scenario realism ($rt$5, $rt$6), controls responsiveness ($rt$7, $rt$8), audio immersiveness ($rt$9, $\overline{rt}_{UM} = 5.15\,\text{s}$0), head tracking ($\overline{rt}_{UM} = 5.15\,\text{s}$1, $\overline{rt}_{UM} = 5.15\,\text{s}$2), and overall experience ($\overline{rt}_{UM} = 5.15\,\text{s}$3, $\overline{rt}_{UM} = 5.15\,\text{s}$4), while traffic simulation realism was not significantly different ($\overline{rt}_{UM} = 5.15\,\text{s}$5, $\overline{rt}_{UM} = 5.15\,\text{s}$6) (Thalapanane et al., 2024).

DriveSimQuest broadens the behavioral state space beyond gaze-centric instrumentation. By exposing facial expressions, hand motion, full-body posture, and steering velocity $\overline{rt}_{UM} = 5.15\,\text{s}$7, it supports analysis of affective state and embodied interaction rather than only attentional allocation (Chidambaram et al., 14 Aug 2025). MRDrive, in turn, demonstrates how eye tracking, pupil diameter, and touch interaction can be synchronized with automated-driving scenarios and explanation interfaces in a mixed-reality cabin (Ebel et al., 9 Mar 2026).

5. Simulation back-ends, rare events, and world modeling

The simulation substrate of VR-Drive has expanded from handcrafted training scenes to learned world models and 4D representations. TRAVERSE already frames rare-event simulation as a remedy for the bias of physical-world AV datasets toward safe, uneventful driving, and intentionally over-samples adverse situations to support data collection for learning-enabled systems (Thalapanane et al., 2024). Vehicle-in-Virtual-Environment extends this idea by letting a real vehicle move in an empty physical area while localization and perception sensor data are fed from a configurable virtual environment, with explicit coordinate transformations linking physical and virtual pose (Cao et al., 2024).

On the rendering side, DrivePhysica is a latent video diffusion model specialized to driving that encodes motion reference systems, temporal consistency of object attributes and motion, and spatial relationships and occlusion hierarchies through a Coordinate System Aligner, Instance Flow Guidance, and Box Coordinate Guidance (Yang et al., 2024). On nuScenes it reports FID $\overline{rt}_{UM} = 5.15\,\text{s}$8 and FVD $\overline{rt}_{UM} = 5.15\,\text{s}$9, and its generated data supports downstream perception training and evaluation (Yang et al., 2024). This suggests a VR-Drive back-end in which photorealistic multi-view driving videos are generated from structured conditions rather than from manually authored assets alone.

DriveDreamer4D pushes closer to a directly explorable virtual environment by combining a controllable world model with 4D Gaussian Splatting (Zhao et al., 2024). Its Novel Trajectory Generation Module modifies ego trajectories under explicit safety constraints $\overline{rt}_{FB} = 3.17\,\text{s}$0 and $\overline{rt}_{FB} = 3.17\,\text{s}$1, then uses a video world model to generate “cousin” trajectory videos for 4DGS training (Zhao et al., 2024). Relative to PVG, S$\overline{rt}_{FB} = 3.17\,\text{s}$2Gaussian, and Deformable-GS, it reports FID improvements of $\overline{rt}_{FB} = 3.17\,\text{s}$3, $\overline{rt}_{FB} = 3.17\,\text{s}$4, and $\overline{rt}_{FB} = 3.17\,\text{s}$5, and NTA-IoU improvements of $\overline{rt}_{FB} = 3.17\,\text{s}$6, $\overline{rt}_{FB} = 3.17\,\text{s}$7, and $\overline{rt}_{FB} = 3.17\,\text{s}$8, indicating better viewpoint robustness and agent coherence under lane changes and speed changes (Zhao et al., 2024).

HorizonDrive addresses another core requirement for VR-Drive-like systems: long-horizon, action-conditioned rollout (Zhang et al., 12 May 2026). Its Scheduled Rollout Recovery trains the base model to reconstruct ground-truth future clips from prediction-corrupted histories, yielding a rollout-capable teacher, and its teacher rollout DMD distills that teacher into a more efficient student (Zhang et al., 12 May 2026). On nuScenes, HorizonDrive reduces FID by $\overline{rt}_{FB} = 3.17\,\text{s}$9, FVD by $\overline{rt}_{LVR} = 3.16\,\text{s}$0, and lowers ARE and DTW by $\overline{rt}_{LVR} = 3.16\,\text{s}$1 and $\overline{rt}_{LVR} = 3.16\,\text{s}$2 relative to the strongest long-horizon streaming baselines, while supporting minute-scale autoregressive rollout under bounded memory (Zhang et al., 12 May 2026).

These systems shift VR-Drive from “simulator as interface” toward “simulator as learned world.” The progression is from portable HMD-based task environments, through traffic-responsive rare-event simulation, to physically informed video generation and long-horizon autoregressive scene evolution (Sportillo et al., 2018, Thalapanane et al., 2024, Yang et al., 2024, Zhang et al., 12 May 2026).

6. Viewpoint-robust end-to-end driving and the evolving meaning of VR-Drive

The 2025 paper titled “VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splatting” gives the term a distinct algorithmic meaning (Cho et al., 27 Oct 2025). Its central problem is viewpoint generalization in end-to-end autonomous driving, especially under varying camera viewpoints caused by diverse vehicle configurations. The method jointly learns 3D scene reconstruction as an auxiliary task to enable planning-aware view synthesis, adopts a feed-forward inference strategy, supports online training-time augmentation from sparse views without additional annotations, and introduces both a viewpoint-mixed memory bank for temporal interaction across multiple viewpoints and a viewpoint-consistent distillation strategy that transfers knowledge from original to synthesized views (Cho et al., 27 Oct 2025).

This usage aligns naturally with nearby work on view synthesis, 4D reconstruction, and viewpoint-robust planning, even though those papers use different names. DriveDreamer4D uses world-model-generated novel trajectory videos to improve 4D driving scene representation under unseen ego trajectories (Zhao et al., 2024). DrivoR compresses multi-camera observations into camera-aware register tokens and evaluates trajectories in pseudo-simulation and photorealistic closed-loop simulation, which is directly relevant to viewpoint and compute efficiency in virtual driving (Kirby et al., 8 Jan 2026). DriveStack-VLA injects BEV representation into a VLM decoder and aligns perceptual focus between real images and rasterized images, again emphasizing spatial grounding across views (Wang et al., 23 Jun 2026).

A plausible implication is that “VR-Drive” now spans two technical directions that were previously treated separately: immersive human-facing simulation and viewpoint-robust machine driving. In the former, the central variables are user embodiment, HMI, and take-over performance; in the latter, they are view synthesis, multi-view consistency, and end-to-end policy robustness (Sportillo et al., 2018, Cho et al., 27 Oct 2025). The supplied literature indicates that these directions are converging around a common set of enabling ideas: sparse-view augmentation, explicit geometry, multimodal sensing, rare-event generation, and learned world models.