- The paper introduces DSGS, eliminating explicit geometry transmission by using compressed atlases and camera metadata to directly infer 3D Gaussian parameters.
- The method achieves notable improvements with a +5.79 dB IV-PSNR gain and reduced inter-view inconsistency from 17.2 to 6.4 dB, while halving the bandwidth requirements.
- The approach replaces expensive cost-volume estimation with feed-forward volumetric inference, enabling real-time rendering at over 500 fps on client GPUs.
Decoder-Side Gaussian Splatting for Immersive Video Compression
Context and Challenges in Immersive Video Coding
Immersive video systems require efficient handling of high-resolution multi-view and depth data for real-time 6DoF experiences, yet are limited by bandwidth and decoder-side compute constraints. Traditional approaches, such as MPEG Immersive Video (MIV) with Decoder-Side Depth Estimation (DSDE), shift geometry (depth) computation to the client, maximizing bitrate allocation to texture. However, DSDE depends on accurate stereo or segment-wise depth estimation, which exhibits significant failure modes for non-Lambertian materials, complex geometry, and leads to strong inter-view inconsistency, perceptual artifacts, and high computational cost for accurate geometry inference on the client. Recent 3D Gaussian Splatting (3DGS) methods achieve high fidelity and support real-time synthesis for arbitrary viewpoints, but efficient compression and transmission of splat attributes remains poorly aligned with video codec pipelines, and splat attribute projection to 2D maps often induces severe geometric artifacts via standard video compression.
Decoder-Side Gaussian Splatting (DSGS): Methodology
The paper introduces Decoder-Side Gaussian Splatting (DSGS), leveraging compressed video atlases and camera metadata, transmitted under the unmodified MIV DSDE profile, to directly initialize and infer 3DGS on the client. Key deviations and advances include:
- Elimination of Explicit Geometry Transmission: By discarding both explicit depth maps and projected 3DGS attribute streams, DSGS utilizes only video atlases and metadata to reconstruct scene geometry via Gaussian splatting, removing dependencies on depth estimators and their domain-specific limitations.
- Direct Volumetric Inference: Lossy-compressed atlases are decoded and input to a feed-forward 3DGS predictor (e.g., ReSplat), generating the 3D Gaussian parameters (center, covariance, opacity, SH) without explicit gradient-based optimization at inference. This redefines the decoder pipeline, completely bypassing cost-volume depth estimation and DIBR.
- Codec-Bounded Regularization: A counterintuitive and critical finding is that standard lossy video compression acts as a beneficial low-pass regularizer for the feed-forward 3DGS predictor. This filtering suppresses spatial micro-texture and noise that otherwise induce redundant, high-variance splats (floater artifacts), thus improving geometric and visual consistency of the synthesized scene—in sharp contrast to the monotonic quality degradation observed with increasing compression in DSDE.
- Standard-Compliant Deployability: DSGS achieves this without syntax or infrastructure changes to the standardized MIV DSDE ecosystem. Camera poses and intrinsic/extrinsic calibration, present in the side metadata, are converted as needed (e.g., to OpenCV conventions), avoiding computationally expensive pose estimation steps.
Experimental Results and Analysis
Objective Quality and Consistency
Experiments follow the MPEG Immersive Video (MIV) Common Test Conditions, focusing on the single-atlas (4-view) regime where data sparsity aggravates the weaknesses of DSDE and exposes the benefits of volumetric 3DGS-based generative synthesis. The main findings are:
- Substantial Gains Over DSDE: Averaged over 14 test sequences, DSGS yields a +5.79 dB increase in IV-PSNR and a +0.054 gain in IV-SSIM, compared to the baseline DSDE anchor under the same bandwidth conditions.
- Superior Inter-View Consistency: Maximum Delta IV-PSNR (reflecting inter-view flicker/artifacts) decreases from 17.2 dB in DSDE to 6.4 dB in DSGS. This evidence points to a significant reduction in domain shift between transmitted and synthesized viewpoints, delivering a notably more stable and artifact-free immersive navigation experience.
- Bandwidth Efficiency: A single compressed atlas (4 views, ~297 kB) with DSGS yields higher or comparable IV-PSNR and IV-SSIM than DSDE operating with two atlases (8 views, ~570 kB), cutting bandwidth by half.
Codec-Quality Interaction
Unlike conventional pipelines, the rate-distortion curve for DSGS is non-monotonic; moderate lossy compression (i.e., RP1) actually improves IV-SSIM compared to the lossless baseline, only degrading at high compression (e.g., RP3+). This validates the hypothesis that video codec quantization synergizes with feed-forward 3DGS inference by discarding nuisance detail and regularizing the volumetric field.
Computational Complexity
DSGS replaces computationally intensive cost volume evaluation and refinement (O(N²)) with feed-forward volumetric inference (O(N)), matched to resource-constrained client GPUs/NPUs. The end-to-end view synthesis pipeline executes nearly in real time, with rendering at >500 fps on high-end consumer GPUs.
Failure and Degeneracy Points
DSGS underperforms DSDE in sequences with minimal view pool size (typically 9 cameras), explained by the reduced challenge for DSDE's cost-volume matching (tighter baselines) and the impact of limited evaluation viewpoints in average metric computation. These degeneracies are not observed for larger camera rigs or denser content.
Implications and Prospects
Practical Impacts
DSGS enables efficient, scalable immersive video delivery, removing geometric data transmission, improving visual stability, and reducing computational latency at the decoder. It is of immediate relevance for low-bandwidth, real-time scenarios and can be retrofitted to existing MIV-based immersive video systems without encoder updates. The codec regularization effect highlights the necessity to co-design generative volumetric pipelines and compression standards, as their interaction determines the domain of achievable synthesis quality.
Theoretical Significance
The finding that codec-induced denoising can outperform lossless inputs by regularizing volumetric predictors challenges standard assumptions in rate-distortion optimization and immersive media coding. It points toward a new paradigm where information bottlenecks, imposed "upstream" in the texture domain, are strategically exploited to improve generative synthesis stability and generalization.
Future Directions
Notably, the current approach processes each frame independently, resulting in limited temporal consistency; minor jitter or flicker may arise across video frames. Future research should emphasize 4D dynamic modeling (e.g., GIFStream-style time-dependent feature anchoring or control-point-guided D-FCGS motion compensation) to address temporal artifacts. Additionally, incorporating sparse geometric hints or edge-computed refinement on the client could further elevate the practical performance and robustness of decoder-side splatting.
Conclusion
DSGS represents a significant evolution in immersive video decoder pipelines: explicit transmission and estimation of 3D geometry are made obsolete by direct, compressed-texture-driven volumetric inference using standardized bitstreams. The ability of video codec quantization to regularize 3DGS predictors and improve synthesis quality shifts the paradigm of compression trade-offs in immersive media. Prospective research will focus on extending DSGS to temporally coherent 4D reconstructions and hybridizing with geometric assistance signals for even greater bandwidth and fidelity efficiency.
