Streaming Real-Time Rendered Scenes as 3D Gaussians

Abstract: Cloud rendering is widely used in gaming and XR to overcome limited client-side GPU resources and to support heterogeneous devices. Existing systems typically deliver the rendered scene as a 2D video stream, which tightly couples the transmitted content to the server-rendered viewpoint and limits latency compensation to image-space reprojection or warping. In this paper, we investigate an alternative approach based on streaming a live 3D Gaussian Splatting (3DGS) scene representation instead of only rendered video. We present a Unity-based prototype in which a server constructs and continuously optimizes a 3DGS model from real-time rendered reference views, while streaming the evolving representation to remote clients using full model snapshots and incremental updates supporting relighting and rigid object dynamics. The clients reconstruct the streamed Gaussian model locally and render their current viewpoint from the received representation. This approach aims to improve viewpoint flexibility for latency compensation and to better amortize server-side scene modeling across multiple users than per-user rendering and video streaming. We describe the system design, evaluate it, and compare it with conventional image warping.

• The paper introduces a live 3D Gaussian streaming system that decouples server rendering from client-side novel-view synthesis and dynamic relighting.
• Continuous optimization leverages multi-view inputs, differentiable rendering, and incremental Draco-compressed updates to refine scene quality in real time.
• Experimental results demonstrate improved perceptual quality over depth-assisted warping with scalability to millions of primitives on commodity GPU hardware.

Streaming Real-Time Rendered Scenes as 3D Gaussians: A Technical Assessment

Problem Formulation and Motivation

Conventional cloud gaming and XR platforms predominantly transmit viewpoint-specific 2D video streams to clients. This approach restricts client-side latency compensation and hinders novel-view rendering, particularly as user viewpoint changes become large or unpredictable. The limitations manifest primarily as artifacts upon reprojection or warping, and notably, problems with disocclusions and dynamic relighting due to the strict coupling between server-rendered frames and client presentation. To address these constraints, the authors propose a fundamentally different strategy: instead of streaming only final rendered images, stream a live-updating 3D Gaussian Splatting (3DGS) representation constructed from engine-native real-time renderings.

System Architecture and Pipeline

The system is implemented as a Unity-based prototype with tight integration between the game engine and a native C++/CUDA/Libtorch plugin responsible for 3DGS construction, optimization, and streaming. The architecture combines the following key modules:

• Reference and Input Views: Multiple input and reference cameras orchestrated by Unity provide multi-view observations, feeding per-pixel world-space positions, appearance, normals, and object IDs into the Gaussian model construction pipeline.
• 3DGS Model Initialization and Expansion: The system initializes new Gaussian primitives directly from online engine data, allowing for region-specific and incremental model growth as the user explores new areas in the scene.
• Continuous Optimization: Differentiable rendering is performed in the native plugin, refining Gaussian parameters (means, scales, appearances, quaternions, etc.) based on reference camera comparisons. Optimization is asynchronous, allowing for seamless interaction even during model updates.
• Handling Scene Dynamics: The representation incorporates both relighting, through SH coefficients and per-Gaussian light visibility estimation, and dynamic rigid object motion via persistent object IDs and transform bridging with Unity.
• Streaming Protocol: Model state synchronization is handled via a hybrid protocol that interleaves full-model Draco-compressed snapshots with frequent incremental tensor-delta updates. This supports efficient synchronization of evolving models without retransmitting redundant data.

  Figure 1: Overview of the proposed live game engine-driven 3DGS pipeline.

Online Gaussian Model Construction, Expansion, and Optimization

A distinctive feature is the system's ability to build and progressively refine the 3DGS model from engine-derived data without dependence on offline pipelines such as multi-view structure-from-motion. This allows near-instantaneous exposure of new geometry and appearance to remote clients and supports exploration-based expansion.

• Online Expansion: The scene is spatially partitioned into a 3D grid. As the user traverses the environment, visibility checks trigger insertion of Gaussians in newly observed but previously unmodeled grid cells, enabling spatially incremental growth.
• Freezing and Preculling: To maintain tractability, the system freezes Gaussians in stabilized regions (switching them from trainable to inference-only) and leverages coarse grid-based visibility for both optimization and rendering culling.

Dynamic Scene Handling: Relighting and Object Motion

The system integrates engine-side relighting by equipping each Gaussian with per-SH-band appearance coefficients and per-splat light visibility terms. Light configuration and shadow maps from Unity enable the model to modulate direct-light response and approximate cast shadows without full scene re-optimization.

Figure 2: View synthesis quality under scene dynamics.

Rigid object dynamics are managed by augmenting each Gaussian with an object identifier and local transform. Rigid motion commands, tracked and streamed from Unity, are applied directly to subsets of the Gaussian set, obviating the need for costly full-geometry diffusion or re-initialization.

Figure 3: Rendered frames from 3D Gaussian model with and without support for dynamics.

Streaming Protocol: Representation Delivery, Synchronization and Bandwidth

The live 3DGS streaming protocol merges full Draco-compressed model snapshots with high-frequency incremental packets (tensors deltas and transform updates) over a socket connection. Snapshots provide robust recovery and synchronization, while delta packets efficiently communicate evolving parameter subsets (such as opacities and low-order SH bands). The protocol maintains object-transform separation, allowing efficient motion tracking without heavy geometry retransmission.

Figure 4: Breakdown of streamed wire bitrate by packet type during the same replay. Full Draco snapshots dominate the largest instantaneous spikes, whereas the steady-state traffic is mainly composed of tensor-delta packets.

Streaming measurements reveal bursty bandwidth profiles dominated by snapshot events, with mean wire rates around 92.5 Mbps and peaks contingent on snapshot and scene complexity. Steady-state traffic is primarily sustained by lightweight delta updates.

Figure 5: Image quality (SSIM) difference over time between server and client during streaming with a moving camera trace. Draco snapshots are shown with check marks.

The SSIM quality gap between server and client remains modest (server: 0.96, client: 0.91 on average), highlighting effective client-side synthesis from streamed representations.

Numerical Results and Comparative Assessment

• Convergence: For static reference camera arrangements, increased reference coverage improves both consistency and absolute quality, but comes at the cost of higher optimization and convergence times. Adequate input camera configuration (in terms of count and resolution) is vital for minimizing residual artifacts especially for multi-user scenarios.

Figure 6: FLIP.

Figure 7: Elapsed wall-clock time until FLIP < 0.07.

• Comparison with Depth-assisted Warping: When compared to view-synthesis via multi-view image + depth warping, 3DGS-based streaming yields notably higher perceptual quality—especially for large novel-view displacements and under dynamic lighting—by sidestepping inherent depth-map quantization and occlusion artifacts.

  Figure 8: Comparison to depth-assisted image warping.

• Scene Exploration and Scalability: The memory and computational profile scales with the number of active/frozen Gaussians. The prototype supports several hundred thousand to millions of primitives on commodity GPU hardware, and the freezing/preculling mechanisms allow performance scaling without sacrificing rendering quality.

  Figure 9: System metrics during the exploration experiment.

Implications, Limitations, and Future Work

Practical Implications

The persistent streaming of live 3DGS representations fundamentally decouples client-side viewpoint rendering from server-side simulation. This allows:

• Significantly improved latency compensation via unrestricted client-side novel-view synthesis
• Amortization of expensive server-side scene modeling across multiple clients, as each can render arbitrary viewpoints from the shared evolving model without per-client video streams
• Greater flexibility in supporting interactive remote graphics for AR/XR and cloud gaming, especially in multi-user, dynamic, and relighted scenes

Theoretical Significance

The work demonstrates that appearance-driven, point-based explicit representations such as 3DGS can act as effective real-time intermediates between traditional game engines and edge client viewers, supporting highly dynamic, relightable, and motion-tolerant rendering.

Limitations and Open Challenges

• History-Dependence of Quality: Regions with longer or richer observation histories attain better convergence; newly revealed or sparsely viewed areas lag in quality.
• Limited Global Illumination: The current relighting implementation is diffuse-only and does not capture view-dependent reflectance or full GI effects.
• Handling of Non-Rigid Dynamics: Only rigid object motion is supported directly.
• End-to-End System Tradeoffs: Full cost-rate-latency-client-CPU comparisons with mature video streaming solutions are not addressed, and real-world bandwidth management (e.g., for mobile networks) requires further exploration.

Future Developments

Future research includes extension to non-rigid and more photometrically complex scenes (see, e.g., "Dynamic 3D Gaussians" (Luiten et al., 2023), "GaussianShader" [CVPR2024: 5322–5332], and "Relightable 3D Gaussians" (Gao et al., 2023)), more aggressive on-the-fly model compression ("Compressed 3D Gaussian Splatting" [CVPR2024: 10349–10358]), bandwidth adaptation, and integration with large-scale streaming architectures for real-world XR and collaborative applications.

Conclusion

The presented system substantiates the feasibility and advantages of streaming real-time, continuously optimized 3D Gaussian splatting representations as an intermediate layer for cloud graphics. By enabling client-side novel-view synthesis, supporting relighting and rigid object dynamics, and offering scalable multi-user synchronization protocols, this work opens practical opportunities for next-generation interactive cloud rendering paradigms that transcend the limitations of traditional video-based delivery (2604.02851).