---
title: Decoder-Side Gaussian Splatting (DSGS)
url: https://www.emergentmind.com/topics/decoder-side-gaussian-splatting-dsgs
type: topic
---

# Decoder-Side Gaussian Splatting (DSGS)

Searching arXiv for the cited DSGS-related papers to ground the article in current preprints.
Decoder-Side Gaussian Splatting (DSGS) is a decoder-side view-synthesis framework for immersive video that replaces the depth-estimation and depth-image-based rendering (DIBR) pipeline used in MPEG Immersive Video (MIV) with feed-forward 3D Gaussian Splatting (3DGS). In the formulation described in "A Single Atlas is All You Need: Decoder-Side Gaussian Splatting for Immersive Video" [2605.17002], the encoder remains an unchanged MIV DSDE profile and transmits only 2D atlases plus camera metadata, while all 3D volumetric structure is inferred at the decoder. The central claim is that geometry transmission becomes redundant: instead of sending depth maps or Gaussian attributes, the decoder reconstructs a 3DGS state directly from compressed textures and poses. Within the broader literature, DSGS also names a more general decoder-centric design pattern in which compressed intermediate representations or compressed Gaussian fields are decoded into renderable Gaussian primitives at the receiver, as illustrated by CodecSplat [2605.25563] and CSGaussian [2601.12814].

## 1. Definition and problem setting

DSGS emerges from a specific bottleneck in immersive video delivery: pixel-rate constraints make the transmission of high-resolution depth maps or explicit 3D volumetric data expensive [2605.17002]. In MPEG Immersive Video, a conventional approach is Decoder-Side Depth Estimation (DSDE), where the encoder sends sparse color views packed into 2D atlases together with camera metadata, and the decoder estimates depth maps and synthesizes novel views via DIBR. This shifts geometry computation to the client and frees bitrate for textures, but the underlying depth-based representation remains fragile under sparse viewpoints, occlusions, and view-dependent effects [2605.17002].

The motivation for DSGS is the mismatch between DSDE’s scalar-per-pixel depth representation and the volumetric complexity of real scenes. The DSDE pipeline struggles with complex geometries and occlusions, exhibits inter-view flickering because each view’s depth is independently estimated, and is poorly matched to non-Lambertian effects such as specularities and translucency [2605.17002]. A plausible implication is that DSDE’s error mode is structural rather than merely photometric: errors manifest as ghosting, fragmentation, and inconsistent quality across neighboring viewpoints.

3D Gaussian Splatting is attractive in this setting because it represents a scene as 3D Gaussian primitives with geometry, opacity, and view-dependent appearance. In the notation used for DSGS, each primitive is
\[
G = \{\mu, \Sigma, \alpha, SH\},
\]
where \(\mu \in \mathbb{R}^3\) is the Gaussian center, \(\Sigma \in \mathbb{R}^{3\times 3}\) is the covariance matrix, \(\alpha\) is opacity, and \(SH\) denotes spherical harmonics coefficients for view-dependent color [2605.17002]. However, directly transmitting splats or projected 2D splat-attribute maps is described as bandwidth- and codec-unfriendly, requiring large attribute streams or complex packing procedures not aligned with standard video codecs [2605.17002]. DSGS is therefore defined not as Gaussian transmission, but as Gaussian inference at the decoder from a standard atlas-and-metadata bitstream.

## 2. Core formulation and system architecture

The conceptual substitution introduced by DSGS is direct. Instead of mapping decoded atlases and poses to depth maps and then to DIBR, DSGS maps them to a 3DGS state and renders novel views by splatting [2605.17002]. The paper defines the decoder-side mapping as
\[
F : (J, P) \mapsto G,
\]
where \(J\) is the set of decoded 2D texture atlases, \(P\) denotes camera poses and metadata, and \(G\) is the 3D Gaussian Splatting state space [2605.17002].

At the encoder, the bitstream remains standard MIV DSDE. The encoder selects a subset of camera views from a multiview sequence, packs selected views into one or more 2D atlases, compresses the atlases using VVenc in the experiments, and transmits the atlas video bitstream plus camera intrinsics and extrinsics [2605.17002]. No extra syntax is introduced for Gaussian attributes or depth maps.

At the decoder, the processing chain changes completely. The decoder first performs standard MIV DSDE bitstream decoding to recover atlases \(J\) and metadata \(P\), converts MIV metadata into an OpenCV-style camera representation, feeds decoded images and poses into a feed-forward 3DGS inference network, obtains Gaussians \(G=\{\mu,\Sigma,\alpha,SH\}\), and renders requested viewpoints using a standard 3DGS splat rasterizer [2605.17002]. In the reported implementation, the feed-forward 3DGS backbone is ReSplat, described as recurrent and using rendering error as a feedback signal for iterative refinement without explicit gradient descent; it initializes a compact Gaussian set in a subsampled space, yielding fewer Gaussians than per-pixel representations and faster tile-based rasterization [2605.17002].

A succinct comparison with the DSDE anchor clarifies the architectural shift:

| Aspect | DSDE | DSGS |
|---|---|---|
| Decoder-side intermediate representation | Depth maps | \(G=\{\mu,\Sigma,\alpha,SH\}\) |
| Rendering backend | DIBR | Splat rasterizer |
| Bitstream syntax | 2D atlases + camera metadata | Unchanged: 2D atlases + camera metadata |

This replacement is especially important in the “extreme view sparsity” regime, defined in the paper as a single atlas comprising 4 input views, with all views rescaled to \(960 \times 540\) to match ReSplat’s internal resolution [2605.17002]. The title phrase “A Single Atlas is All You Need” refers to the empirical observation that feed-forward DSGS remains effective under this sparse-view condition, whereas standard DSDE tends to break down [2605.17002].

## 3. Mathematical and algorithmic properties

The mathematical description of DSGS is intentionally compact and inherits standard 3DGS rendering assumptions. The central representational distinction is that DSDE emits per-pixel scalar depths, whereas DSGS emits a continuous volumetric field parameterized by Gaussian covariance and opacity [2605.17002]. This suggests a more expressive geometric prior: covariance and opacity can encode structural uncertainty and some non-Lambertian phenomena more naturally than a single depth sample per pixel.

The paper does not reproduce full rendering equations, but relies on standard 3DGS semantics: each Gaussian defines a density kernel in 3D; after projection into 2D it produces an elliptical footprint; and pixel color is accumulated by alpha compositing contributions along a ray, with view-dependent color modeled by spherical harmonics [2605.17002]. An interpretive but standard expression included in the source material is
\[
I(\mathbf{x}) = \sum_i T_i \alpha_i c_i(\mathbf{d}),
\]
where \(T_i\) is accumulated transparency, \(\alpha_i\) is opacity, and \(c_i(\mathbf{d})\) is SH-based view-dependent color for viewing direction \(\mathbf{d}\) [2605.17002].

A common misconception is that DSGS performs per-scene optimization or stochastic gradient descent at the decoder in the style of classical 3DGS training. The paper explicitly rejects this: to avoid the latency of standard SGD, the framework integrates a feed-forward recurrent Gaussian splatting model at the decoder, and no decoder-side fine-tuning equations or additional regularizers are introduced [2605.17002]. Training of the predictor \(F\) is offline and inherited from the pre-trained ReSplat model. Accordingly, the phrase “optimizing volumetric scenes entirely on the decoder side” refers to generating all 3D geometry at the decoder from transmitted atlases and metadata, not to running scene-specific training there [2605.17002].

Complexity is also framed in algorithmic terms. Traditional DSDE depth estimation is described as using cost-volume matching with runtime roughly \(O(N^2)\) in disparity search steps and spatial resolution, whereas feed-forward 3DGS inference is effectively \(O(N)\) in depth complexity because the decoder performs inference rather than training or iterative optimization [2605.17002]. This does not imply zero decoder cost; it implies a different asymptotic and implementation profile.

## 4. Compression as implicit regularization

One of the most distinctive findings associated with DSGS is that lossy 2D video compression of the input atlases can improve 3DGS inference quality relative to lossless inputs [2605.17002]. The rationale given is that dynamic Gaussian initialization is highly sensitive to high-frequency noise and micro-textures, which can induce redundant, high-variance splats and “floater” Gaussians. Standard lossy 2D codecs preferentially discard high-frequency content, thereby acting as a low-pass filter that emphasizes coherent large-scale structure [2605.17002].

The paper states that “By using lossy 2D video compression, the encoded atlases effectively undergo low-pass filtering. This implicit denoising forces the feed-forward network to focus on the dominant, coherent geometry, thereby stabilizing the covariance matrix \(\Sigma\) and reducing the variance of generated attributes” [2605.17002]. It further relates this effect to explicit geometric regularization and to the information bottleneck principle used in ZPressor [2605.17002]. This suggests that DSGS should be understood not only as a rendering substitution but also as a decoder-side generative pipeline whose quality depends on the spectral properties of its compressed inputs.

The empirical pattern is non-monotonic for DSGS and monotonic for DSDE. For single-atlas DSGS, at RP0 (lossless) the reported IV-SSIM is approximately \(0.925\), while at RP1, with a bitstream about tenfold smaller than RP0, IV-SSIM increases to \(0.927\), and PSNR and IV-PSNR also increase at RP1–RP2 before degrading beyond RP3 [2605.17002]. In contrast, DSDE behaves conventionally: IV-SSIM decreases monotonically from \(0.860\) at RP0 to \(0.843\) at RP4 [2605.17002]. The factual significance is precise: for DSGS, lossless RP0 is not the best operating point, whereas for DSDE compression only harms quality.

This observation is central to the identity of DSGS because it reverses an assumption common in classical geometry coding. In DSGS, standard video compression is not merely a transport layer; it can function as an implicit regularizer for downstream volumetric inference [2605.17002]. A plausible implication is that future immersive-video codec design may need to co-design quantization and decoder-side generative backends rather than treating them as separable modules.

## 5. Experimental characterization in immersive video

The experimental evaluation in [2605.17002] uses 14 MIV CTC multiview test sequences from Classes J, W, D, E, and L, with computer-generated omnidirectional content from Classes A, B, and C excluded because of equirectangular distortions and the pinhole-camera assumptions of current feed-forward architectures [2605.17002]. Configurations include one atlas with at most 4 views and two atlases with at most 8 views; atlases are compressed with VVenc in All-Intra mode; and the baseline is the MIV DSDE profile with the reference IVDE depth estimator [2605.17002].

In the single-atlas, four-view regime, average gains over the DSDE anchor are reported as **+5.79 dB BD-PSNR** and **+0.0538 BD-SSIM** over 14 sequences [2605.17002]. The abstract rounds the latter to **+0.054 BD-SSIM** and highlights the same result [2605.17002]. These figures quantify rate-distortion superiority under the “extreme view sparsity” condition that motivates the method.

Inter-view consistency is a second major axis of evaluation. The average maximum \(\Delta\)IV-PSNR is reported as **17.20 dB** for DSDE and **6.40 dB** for DSGS, while average \(\Delta\)IV-SSIM is **0.16** for DSDE and **0.05** for DSGS [2605.17002]. The interpretation supplied by the paper is that a 17.2 dB gap indicates some views are dramatically worse than others, causing severe flickering during navigation, whereas synthesis from a unified Gaussian model greatly reduces the domain shift between transmitted and virtual views [2605.17002].

Equal-bitrate comparisons sharpen this claim. At RP1 with a single atlas, both DSDE and DSGS use an average bitstream size of approximately **297 kB**, but DSDE achieves IV-PSNR of approximately **24.08 dB** and IV-SSIM of approximately **0.858**, while DSGS reaches IV-PSNR of approximately **31.77 dB** and IV-SSIM of approximately **0.927** [2605.17002]. Moreover, one-atlas DSGS at RP1, with a bitstream of approximately **297 kB**, outperforms two-atlas DSDE at RP1, which uses approximately **570 kB** and reaches IV-PSNR of approximately **30.89 dB** [2605.17002]. This is the factual basis for the claim that one-atlas DSGS can exceed two-atlas DSDE while using about half the bitrate.

The paper also identifies failure or boundary cases. DSGS underperforms DSDE on E02 (Carpark), E03 (Street), and L01 (Fencing), which are described as having the smallest camera counts, namely 9 cameras; in those sequences, 4 of 9 views provide relatively narrow baselines and DSDE cost-volume depth estimation benefits from denser viewpoints [2605.17002]. This matters because it limits overgeneralization: DSGS is not presented as uniformly dominant across all capture geometries.

Runtime measurements emphasize deployment implications. DSDE depth estimation with IVDE on CPU is reported as taking more than **15 minutes** on AMD Ryzen 9 for the sequences, whereas DSGS feed-forward inference takes **<1 second** on GPU (H100), rendering exceeds **500 fps**, and peak GPU memory is **<10 GB** [2605.17002]. These are reported as sequence-level timings under the tested hardware assumptions, not universal constants.

## 6. Position within the broader DSGS landscape

Within the broader literature, DSGS can denote more than the MIV-specific substitution in [2605.17002]. CodecSplat [2605.25563] can be read as a concrete instantiation of a decoder-side Gaussian splatting scheme in which the decoder receives an entropy-coded latent tensor plus camera metadata and reconstructs 3D Gaussian primitives from that latent. The pipeline there is explicit: entropy decode \(\hat Z\), reconstruct \(\hat F_{\text c}\) through a synthesis transform, predict depth, perform post-codec depth-guided multi-view refinement, predict Gaussian attributes, map them from 2D to 3D using camera geometry, and render with a standard 3DGS rasterizer [2605.25563]. The defining difference from [2605.17002] is where compression is applied: CodecSplat compresses an intermediate structured 2D Gaussian-generation feature rather than ordinary video atlases or final irregular Gaussian sets.

This intermediate-feature strategy yields very low scene-level rates. On DL3DV and RealEstate10K, CodecSplat reports **23.56–26.36 dB** and **24.76–27.05 dB** PSNR with only **20.00–107.77 KiB** and **3.37–12.51 KiB per scene**, respectively [2605.25563]. The paper characterizes this as roughly one order of magnitude smaller than compressing feed-forward generated Gaussian primitives while maintaining controllable rate-distortion behavior [2605.25563]. In a decoder-centric taxonomy, this suggests a variant of DSGS where the transmitted object is neither video atlases nor Gaussian attributes, but a compact learned latent that remains structurally aligned with Gaussian generation.

CSGaussian [2601.12814] represents a different branch of the decoder-side paradigm: instead of inferring Gaussians from compressed 2D observations, it compresses a 3D Gaussian representation that jointly carries appearance and semantic features so that the decoder can render and also perform segmentation and manipulation [2601.12814]. Built on Scaffold-GS, it organizes the scene into anchors on a voxel grid, with each anchor carrying offsets, a scaling vector, an anchor feature, and a semantic feature \(\boldsymbol{s}\in\mathbb{R}^6\) shared across its \(K\) Gaussians [2601.12814]. The framework introduces a lightweight implicit neural representation-based hyperprior for entropy coding and a compression-guided segmentation learning strategy based on quantization-aware training and a quality-aware weighting mechanism [2601.12814].

The decoder-side consequences are substantive. CSGaussian is reported to support real-time rendering, 3D semantic segmentation, open-vocabulary queries, click-based selection, and scene manipulation directly from the compressed Gaussian representation, without access to training images, SAM, or other heavy foundation models at the receiver [2601.12814]. Quantitatively, the paper states that adding semantics increases bitrate by only **2–4%** with negligible PSNR change, while over LERF and 3D-OVS it achieves **>140× bitrate reduction** versus OpenGaussian and **>23×** versus InstanceGS, with **+2–5 percentage points of mIoU** [2601.12814]. In this sense, CSGaussian extends DSGS from decoder-side view synthesis to decoder-side semantic functionality.

Taken together, these works define a coherent landscape. In [2605.17002], DSGS is a standards-compatible MIV decoder replacement that infers a volumetric scene from compressed atlases and metadata. In [2605.25563], it becomes latent-coded decoder-side Gaussian reconstruction from structured intermediate features. In [2601.12814], it denotes decoder-side deployment of compressed Gaussian fields carrying both appearance and semantics. The common denominator is that the receiver reconstructs or consumes Gaussian primitives as its primary scene representation, and that downstream rendering or semantic functionality is enabled without transmitting dense geometry in conventional form.

## 7. Limitations, assumptions, and open directions

The MIV-oriented DSGS formulation in [2605.17002] is explicitly conditioned on standard MIV capture assumptions: pinhole cameras, known intrinsics and extrinsics, and static scenes [2605.17002]. The reported experiments target static scenes, and the paper notes that current feed-forward 3DGS models are designed per frame; applying them independently to video frames can introduce temporal jitter in Gaussian attributes and slight inter-frame flicker even when inter-view consistency within a frame is good [2605.17002]. This establishes an important distinction between inter-view coherence and temporal coherence.

Sparse-camera edge cases remain a limitation. The sequences where DSGS underperforms DSDE are characterized by relatively few cameras and narrower baselines, conditions under which cost-volume depth estimation is less challenged and can retain an advantage [2605.17002]. The method also deliberately avoids pose-free architectures because MIV already transmits poses [2605.17002], so its deployment premise is calibrated multiview capture rather than unconstrained view synthesis.

Future work identified in [2605.17002] includes temporal consistency through dynamic 4D modeling and motion compensation, with GIFStream and D-FCGS named as relevant directions; lightweight geometric assistance via MIV’s Geometry Assistance SEI; and standard co-design that explicitly accounts for the interaction between lossy video compression and decoder-side generative reconstruction [2605.17002]. CodecSplat suggests another direction: moving the coding bottleneck into structured intermediate Gaussian-generation features and controlling quality by a rate parameter \(\lambda\) within a hierarchical learned codec [2605.25563]. CSGaussian points toward multi-function decoder-side Gaussian representations in which rendering, segmentation, and editing are all supported from a single RD-optimized bitstream [2601.12814].

A final misconception worth addressing is that DSGS is simply “3DGS compression.” The MIV formulation is almost the opposite: it sends no 3DGS attributes at all and instead reconstructs Gaussians entirely at the decoder from standard compressed video plus metadata [2605.17002]. CodecSplat and CSGaussian show that adjacent decoder-side designs may instead transmit compressed latents or compressed Gaussian fields [2605.25563; 2601.12814]. The broader concept therefore names a family of decoder-centric systems, but the specific term DSGS in immersive-video standards discourse most precisely refers to decoder-side replacement of DSDE by feed-forward 3DGS inference on an unchanged MIV bitstream [2605.17002].

Source: https://www.emergentmind.com/topics/decoder-side-gaussian-splatting-dsgs