Streamed Foveated Path Tracing for Volumetric VR Rendering

• The paper introduces a novel streamed foveated path tracing pipeline that combines high-fidelity Monte Carlo path tracing for the foveal view with efficient Gaussian splatting for peripheral regions.
• It employs streaming architectures, temporal and spatial denoising, and adaptive model retraining to mitigate latency and enhance interactive performance in immersive VR environments.
• This hybrid approach optimizes computational resources by focusing on gaze-adaptive rendering while maintaining perceptual scene coherence across the volumetric data.

Streamed foveated path tracing is a hybrid rendering methodology for immersive volumetric visualization, particularly suited for anatomical data in medical imaging. It combines high-fidelity, gaze-adaptive path tracing for the foveal region with lightweight, continuously updated Gaussian Splatting approximations in the periphery. This division optimizes computational resources by concentrating rendering effort where user attention is focused, while maintaining perceptual scene coherence and enabling real-time interactive performance in VR environments. The approach leverages streaming architectures, temporal and spatial denoising, and novel model retraining strategies to bridge latency and quality gaps inherent in remote volumetric rendering (Kleinbeck et al., 29 Jan 2026).

1. System Architecture and Pipeline Overview

The streaming pipeline is partitioned into three loosely coupled modules:

• Foveal Path Tracer: Deployed on a high-performance GPU server, this module receives eye-gaze and head pose data to render high-spp (samples per pixel) volumetric images within a circular foveation radius ($R\approx20^\circ$).
• Gaussian Splatting Peripheral Model Trainer: Hosted on a separate GPU server, it generates and optimizes a low-cost 3D Gaussian cloud to approximate peripheral scene regions, leveraging the Mini-Splatting2 framework for accelerated batched rasterization and optimization.
• Lightweight Real-Time Viewer: Operating locally on a VR headset (HMD) or desktop, the viewer handles pose acquisition, compositing, and asynchronous depth-guided reprojection to mask network and rendering latency.

During initialization ("preparation mode"), $N$ denoised images are path traced from different camera poses ($8\leq\mathrm{spp}\leq32$). First-hit points are extracted for each ray and colored to form a point cloud, which is then converted to $\sim20\,000$ Gaussians using Mini-Splatting2 in approximately 1 second. In interactive streaming mode, the viewer transmits gaze and pose, receives foveated images plus linear depth buffers—reprojects the foveal mesh asynchronously—renders peripheral Gaussians, composites results via alpha blending, and queues new view data for Gaussian refinement at intervals or upon thresholded scene novelty.

2. Foveated Path Tracing

High-quality volumetric shading is achieved via ray-marching Monte Carlo path tracing with standard absorption-scattering, supporting up to 4 bounces and environment lighting. The gaze-adaptive foveation function determines per-ray spp:

$$\mathrm{spp}(\theta) = \mathrm{spp}_\mathrm{max}\,\exp\left(-\frac{\theta^2}{2\sigma^2}\right)$$

where $\theta$ is the angle from gaze center and $\sigma\approx R/2$ controls the falloff. Only rays inside $R$ are processed at high spp; outside, spp approaches zero. The initialization phase applies stand-alone NVIDIA OptiX denoising, while streaming employs temporal denoising with motion vectors and albedo for real-time quality preservation. This design enables perceptual optimization, allocating compute to regions of active visual scrutiny while dynamically reducing rendering cost in the periphery.

3. Peripheral Gaussian Splatting

Peripheral scene regions are rendered using a parametric Gaussian cloud:

$G(p) = \sum_i G_i(p)\,T_i$

where each $G_i$ is a 3D Gaussian parameterized by position $\mu_i$, covariance $\Sigma_i$, opacity $\alpha_i$, and color $T_i$. Generation proceeds by casting rays from $\leq16$ wide-angle poses at low spp, collecting first-hit points and coloring, followed by position-based simplification to produce an initial cloud. Fast minibatch rasterization and gradient descent (Adam optimizer with per-Gaussian LR increased by 50%) drive the Mini-Splatting2 optimization:

$$w^{(t+1)} = w^{(t)} - \eta \frac{m^{(t)}}{\sqrt{v^{(t)}+\epsilon}}$$

Continuous peripheral refinement leverages newly acquired foveal views. A view is considered "sufficiently novel" if

$$\forall\,v_i\in V_\mathrm{train}:\; (\|pos_\mathrm{new}-pos_i\|>\delta_\mathrm{pos})\,\lor\,( \langle vec_\mathrm{new},\,vec_i\rangle < \cos\theta_\mathrm{view})$$

with $\delta_\mathrm{pos}\approx0.05\,\text{m}$ and $\theta_\mathrm{view}\approx5^\circ$. Periodic retraining ($500$–$1000$ steps, doubling Gaussian count if enabled) ensures peripheral accuracy and responsiveness, with typical update times of $0.8$–$1.0$ s.

4. Depth-Guided Reprojection and Latency Mitigation

To decouple rendering and viewer pose, foveal images are transmitted with linear depth and source pose ($P_\mathrm{old}, V_\mathrm{old}$). The viewer reconstructs mesh geometry via reprojection:

$$p_\mathrm{far} = (P_\mathrm{old}V_\mathrm{old})^{-1}(2uv\!-\!1,\,1,\,1)^\top_w$$

$$p_\mathrm{world} = \mathrm{pos}_\mathrm{old} + d\frac{p_\mathrm{far}-\mathrm{pos}_\mathrm{old}}{\|p_\mathrm{far}-\mathrm{pos}_\mathrm{old}\|}$$

$$p'_\mathrm{clip} = P_\mathrm{new}V_\mathrm{new} p_\mathrm{world}$$

The reconstructed mesh, filled from each depth map frame, is composited with peripheral Gaussians, with small disocclusions treated using the Gaussian cloud, further minimizing perceptible latency artifacts.

5. Streaming Protocols, Performance, and Implementation

Remote pipeline components (foveal tracer and Gaussian trainer) communicate via low-latency TCP, transmitting pose and large texture payloads (4096×4096 px) using MsgPack. Typical timings indicate foveal render & depth receipt at ≈30 ms; reprojection and compositing <1 ms; peripheral rendering <0.5 ms (desktop) or <10 ms (mobile HMD). Peripheral model update is highly efficient (0.8–1.0 s with 700 iterations; "high quality" preset: 16 views×16 spp ≈1.2 s). Resource and quality trade-offs are tunable: initial model construction achieves "normal quality" in ≈300 ms (12 views×8 spp) and "high quality" in ≈400 ms (16 views×16 spp).

A summary of critical parameters is presented below:

Parameter	Value/Range	Context
FoveaFOV	20°	Circular foveal region
σ	10°	Foveation falloff
spp_fovea	16	High-quality rays
spp_periphery	≈0	Peripheral rays
Views_init	16	Initial poses
Optimizer	Adam	(β₁=0.9, β₂=0.99)
MaxGaussians	≈50k	Peripheral model

Viewer-side pseudocode for interactive streaming and model training defines the control flow for pose acquisition, rendering, reprojection, and compositing.

6. Perceptual Quality and Metrics

Foveal mean peak signal-to-noise ratio (PSNR) increases by 3–4 dB compared to standalone path tracing at matched spp. Peripheral Gaussian clouds containing 8k–12k elements render at <0.5 ms, occupying <5 MB of video memory. Quality versus time graphs demonstrate diminishing returns for model fidelity improvement beyond ≈16 views or 16 spp. Scene blending and masking metrics include masked PSNR and LPIPS. This suggests that the hybridization yields both interactive refresh rates and near-optimal foveal perceptual quality while maintaining peripheral visual context.

7. Extension to General Hybrid Rendering Domains

While developed for immersive medical visualization, the streamed foveated path tracing methodology generalizes to interactive hybrid rendering of volumetric and spatial data, including geospatial datasets, fluid flows, and four-dimensional scientific simulations. The path tracer may be substituted with any remote high-quality renderer, while the Gaussian cloud representation is reusable for dynamic scenes and point-based models. The perceptual pipeline—foveation, reprojection, and hybrid composition—is extensible to AR, VR, and XR applications in diverse domains (Kleinbeck et al., 29 Jan 2026).

A plausible implication is that this architecture enables scalable, resource-efficient remote rendering with perceptual optimization for any scenario where localized high-fidelity visualization must be balanced against global contextual awareness and network constraints.