Decoder-Side Gaussian Splatting (DSGS)

• DSGS is a view-synthesis framework that infers 3D Gaussian primitives from compressed video atlases and camera metadata, replacing traditional depth estimation pipelines.
• It employs a feed-forward, recurrent 3D Gaussian splatting network that reconstructs volumetric geometry efficiently, reducing inter-view inconsistencies and enhancing rendering speed.
• By leveraging lossy video compression as an implicit regularizer, DSGS achieves superior rate-distortion performance and lower bitrate requirements compared to DSDE methods.

Searching arXiv for the cited DSGS-related papers to ground the article in current preprints. Decoder-Side Gaussian Splatting (DSGS) is a decoder-side view-synthesis framework for immersive video that replaces the depth-estimation and depth-image-based rendering (DIBR) pipeline used in MPEG Immersive Video (MIV) with feed-forward 3D Gaussian Splatting (3DGS). In the formulation described in "A Single Atlas is All You Need: Decoder-Side Gaussian Splatting for Immersive Video" (Mieloch et al., 16 May 2026), the encoder remains an unchanged MIV DSDE profile and transmits only 2D atlases plus camera metadata, while all 3D volumetric structure is inferred at the decoder. The central claim is that geometry transmission becomes redundant: instead of sending depth maps or Gaussian attributes, the decoder reconstructs a 3DGS state directly from compressed textures and poses. Within the broader literature, DSGS also names a more general decoder-centric design pattern in which compressed intermediate representations or compressed Gaussian fields are decoded into renderable Gaussian primitives at the receiver, as illustrated by CodecSplat (Yu et al., 25 May 2026) and CSGaussian (Tseng et al., 19 Jan 2026).

1. Definition and problem setting

DSGS emerges from a specific bottleneck in immersive video delivery: pixel-rate constraints make the transmission of high-resolution depth maps or explicit 3D volumetric data expensive (Mieloch et al., 16 May 2026). In MPEG Immersive Video, a conventional approach is Decoder-Side Depth Estimation (DSDE), where the encoder sends sparse color views packed into 2D atlases together with camera metadata, and the decoder estimates depth maps and synthesizes novel views via DIBR. This shifts geometry computation to the client and frees bitrate for textures, but the underlying depth-based representation remains fragile under sparse viewpoints, occlusions, and view-dependent effects (Mieloch et al., 16 May 2026).

The motivation for DSGS is the mismatch between DSDE’s scalar-per-pixel depth representation and the volumetric complexity of real scenes. The DSDE pipeline struggles with complex geometries and occlusions, exhibits inter-view flickering because each view’s depth is independently estimated, and is poorly matched to non-Lambertian effects such as specularities and translucency (Mieloch et al., 16 May 2026). A plausible implication is that DSDE’s error mode is structural rather than merely photometric: errors manifest as ghosting, fragmentation, and inconsistent quality across neighboring viewpoints.

3D Gaussian Splatting is attractive in this setting because it represents a scene as 3D Gaussian primitives with geometry, opacity, and view-dependent appearance. In the notation used for DSGS, each primitive is

$G = \{\mu, \Sigma, \alpha, SH\},$

where $\mu \in \mathbb{R}^3$ is the Gaussian center, $\Sigma \in \mathbb{R}^{3\times 3}$ is the covariance matrix, $\alpha$ is opacity, and $SH$ denotes spherical harmonics coefficients for view-dependent color (Mieloch et al., 16 May 2026). However, directly transmitting splats or projected 2D splat-attribute maps is described as bandwidth- and codec-unfriendly, requiring large attribute streams or complex packing procedures not aligned with standard video codecs (Mieloch et al., 16 May 2026). DSGS is therefore defined not as Gaussian transmission, but as Gaussian inference at the decoder from a standard atlas-and-metadata bitstream.

2. Core formulation and system architecture

The conceptual substitution introduced by DSGS is direct. Instead of mapping decoded atlases and poses to depth maps and then to DIBR, DSGS maps them to a 3DGS state and renders novel views by splatting (Mieloch et al., 16 May 2026). The paper defines the decoder-side mapping as

$F : (J, P) \mapsto G,$

where $J$ is the set of decoded 2D texture atlases, $P$ denotes camera poses and metadata, and $G$ is the 3D Gaussian Splatting state space (Mieloch et al., 16 May 2026).

At the encoder, the bitstream remains standard MIV DSDE. The encoder selects a subset of camera views from a multiview sequence, packs selected views into one or more 2D atlases, compresses the atlases using VVenc in the experiments, and transmits the atlas video bitstream plus camera intrinsics and extrinsics (Mieloch et al., 16 May 2026). No extra syntax is introduced for Gaussian attributes or depth maps.

At the decoder, the processing chain changes completely. The decoder first performs standard MIV DSDE bitstream decoding to recover atlases $J$ and metadata $\mu \in \mathbb{R}^3$0, converts MIV metadata into an OpenCV-style camera representation, feeds decoded images and poses into a feed-forward 3DGS inference network, obtains Gaussians $\mu \in \mathbb{R}^3$1, and renders requested viewpoints using a standard 3DGS splat rasterizer (Mieloch et al., 16 May 2026). In the reported implementation, the feed-forward 3DGS backbone is ReSplat, described as recurrent and using rendering error as a feedback signal for iterative refinement without explicit gradient descent; it initializes a compact Gaussian set in a subsampled space, yielding fewer Gaussians than per-pixel representations and faster tile-based rasterization (Mieloch et al., 16 May 2026).

A succinct comparison with the DSDE anchor clarifies the architectural shift:

Aspect	DSDE	DSGS
Decoder-side intermediate representation	Depth maps	$\mu \in \mathbb{R}^3$2
Rendering backend	DIBR	Splat rasterizer
Bitstream syntax	2D atlases + camera metadata	Unchanged: 2D atlases + camera metadata

This replacement is especially important in the “extreme view sparsity” regime, defined in the paper as a single atlas comprising 4 input views, with all views rescaled to $\mu \in \mathbb{R}^3$3 to match ReSplat’s internal resolution (Mieloch et al., 16 May 2026). The title phrase “A Single Atlas is All You Need” refers to the empirical observation that feed-forward DSGS remains effective under this sparse-view condition, whereas standard DSDE tends to break down (Mieloch et al., 16 May 2026).

3. Mathematical and algorithmic properties

The mathematical description of DSGS is intentionally compact and inherits standard 3DGS rendering assumptions. The central representational distinction is that DSDE emits per-pixel scalar depths, whereas DSGS emits a continuous volumetric field parameterized by Gaussian covariance and opacity (Mieloch et al., 16 May 2026). This suggests a more expressive geometric prior: covariance and opacity can encode structural uncertainty and some non-Lambertian phenomena more naturally than a single depth sample per pixel.

The paper does not reproduce full rendering equations, but relies on standard 3DGS semantics: each Gaussian defines a density kernel in 3D; after projection into 2D it produces an elliptical footprint; and pixel color is accumulated by alpha compositing contributions along a ray, with view-dependent color modeled by spherical harmonics (Mieloch et al., 16 May 2026). An interpretive but standard expression included in the source material is

$\mu \in \mathbb{R}^3$4

where $\mu \in \mathbb{R}^3$5 is accumulated transparency, $\mu \in \mathbb{R}^3$6 is opacity, and $\mu \in \mathbb{R}^3$7 is SH-based view-dependent color for viewing direction $\mu \in \mathbb{R}^3$8 (Mieloch et al., 16 May 2026).

A common misconception is that DSGS performs per-scene optimization or stochastic gradient descent at the decoder in the style of classical 3DGS training. The paper explicitly rejects this: to avoid the latency of standard SGD, the framework integrates a feed-forward recurrent Gaussian splatting model at the decoder, and no decoder-side fine-tuning equations or additional regularizers are introduced (Mieloch et al., 16 May 2026). Training of the predictor $\mu \in \mathbb{R}^3$9 is offline and inherited from the pre-trained ReSplat model. Accordingly, the phrase “optimizing volumetric scenes entirely on the decoder side” refers to generating all 3D geometry at the decoder from transmitted atlases and metadata, not to running scene-specific training there (Mieloch et al., 16 May 2026).

Complexity is also framed in algorithmic terms. Traditional DSDE depth estimation is described as using cost-volume matching with runtime roughly $\Sigma \in \mathbb{R}^{3\times 3}$0 in disparity search steps and spatial resolution, whereas feed-forward 3DGS inference is effectively $\Sigma \in \mathbb{R}^{3\times 3}$1 in depth complexity because the decoder performs inference rather than training or iterative optimization (Mieloch et al., 16 May 2026). This does not imply zero decoder cost; it implies a different asymptotic and implementation profile.

4. Compression as implicit regularization

One of the most distinctive findings associated with DSGS is that lossy 2D video compression of the input atlases can improve 3DGS inference quality relative to lossless inputs (Mieloch et al., 16 May 2026). The rationale given is that dynamic Gaussian initialization is highly sensitive to high-frequency noise and micro-textures, which can induce redundant, high-variance splats and “floater” Gaussians. Standard lossy 2D codecs preferentially discard high-frequency content, thereby acting as a low-pass filter that emphasizes coherent large-scale structure (Mieloch et al., 16 May 2026).

The paper states that “By using lossy 2D video compression, the encoded atlases effectively undergo low-pass filtering. This implicit denoising forces the feed-forward network to focus on the dominant, coherent geometry, thereby stabilizing the covariance matrix $\Sigma \in \mathbb{R}^{3\times 3}$2 and reducing the variance of generated attributes” (Mieloch et al., 16 May 2026). It further relates this effect to explicit geometric regularization and to the information bottleneck principle used in ZPressor (Mieloch et al., 16 May 2026). This suggests that DSGS should be understood not only as a rendering substitution but also as a decoder-side generative pipeline whose quality depends on the spectral properties of its compressed inputs.

The empirical pattern is non-monotonic for DSGS and monotonic for DSDE. For single-atlas DSGS, at RP0 (lossless) the reported IV-SSIM is approximately $\Sigma \in \mathbb{R}^{3\times 3}$3, while at RP1, with a bitstream about tenfold smaller than RP0, IV-SSIM increases to $\Sigma \in \mathbb{R}^{3\times 3}$4, and PSNR and IV-PSNR also increase at RP1–RP2 before degrading beyond RP3 (Mieloch et al., 16 May 2026). In contrast, DSDE behaves conventionally: IV-SSIM decreases monotonically from $\Sigma \in \mathbb{R}^{3\times 3}$5 at RP0 to $\Sigma \in \mathbb{R}^{3\times 3}$6 at RP4 (Mieloch et al., 16 May 2026). The factual significance is precise: for DSGS, lossless RP0 is not the best operating point, whereas for DSDE compression only harms quality.

This observation is central to the identity of DSGS because it reverses an assumption common in classical geometry coding. In DSGS, standard video compression is not merely a transport layer; it can function as an implicit regularizer for downstream volumetric inference (Mieloch et al., 16 May 2026). A plausible implication is that future immersive-video codec design may need to co-design quantization and decoder-side generative backends rather than treating them as separable modules.

5. Experimental characterization in immersive video

The experimental evaluation in (Mieloch et al., 16 May 2026) uses 14 MIV CTC multiview test sequences from Classes J, W, D, E, and L, with computer-generated omnidirectional content from Classes A, B, and C excluded because of equirectangular distortions and the pinhole-camera assumptions of current feed-forward architectures (Mieloch et al., 16 May 2026). Configurations include one atlas with at most 4 views and two atlases with at most 8 views; atlases are compressed with VVenc in All-Intra mode; and the baseline is the MIV DSDE profile with the reference IVDE depth estimator (Mieloch et al., 16 May 2026).

In the single-atlas, four-view regime, average gains over the DSDE anchor are reported as +5.79 dB BD-PSNR and +0.0538 BD-SSIM over 14 sequences (Mieloch et al., 16 May 2026). The abstract rounds the latter to +0.054 BD-SSIM and highlights the same result (Mieloch et al., 16 May 2026). These figures quantify rate-distortion superiority under the “extreme view sparsity” condition that motivates the method.

Inter-view consistency is a second major axis of evaluation. The average maximum $\Sigma \in \mathbb{R}^{3\times 3}$7IV-PSNR is reported as 17.20 dB for DSDE and 6.40 dB for DSGS, while average $\Sigma \in \mathbb{R}^{3\times 3}$8IV-SSIM is 0.16 for DSDE and 0.05 for DSGS (Mieloch et al., 16 May 2026). The interpretation supplied by the paper is that a 17.2 dB gap indicates some views are dramatically worse than others, causing severe flickering during navigation, whereas synthesis from a unified Gaussian model greatly reduces the domain shift between transmitted and virtual views (Mieloch et al., 16 May 2026).

Equal-bitrate comparisons sharpen this claim. At RP1 with a single atlas, both DSDE and DSGS use an average bitstream size of approximately 297 kB, but DSDE achieves IV-PSNR of approximately 24.08 dB and IV-SSIM of approximately 0.858, while DSGS reaches IV-PSNR of approximately 31.77 dB and IV-SSIM of approximately 0.927 (Mieloch et al., 16 May 2026). Moreover, one-atlas DSGS at RP1, with a bitstream of approximately 297 kB, outperforms two-atlas DSDE at RP1, which uses approximately 570 kB and reaches IV-PSNR of approximately 30.89 dB (Mieloch et al., 16 May 2026). This is the factual basis for the claim that one-atlas DSGS can exceed two-atlas DSDE while using about half the bitrate.

The paper also identifies failure or boundary cases. DSGS underperforms DSDE on E02 (Carpark), E03 (Street), and L01 (Fencing), which are described as having the smallest camera counts, namely 9 cameras; in those sequences, 4 of 9 views provide relatively narrow baselines and DSDE cost-volume depth estimation benefits from denser viewpoints (Mieloch et al., 16 May 2026). This matters because it limits overgeneralization: DSGS is not presented as uniformly dominant across all capture geometries.

Runtime measurements emphasize deployment implications. DSDE depth estimation with IVDE on CPU is reported as taking more than 15 minutes on AMD Ryzen 9 for the sequences, whereas DSGS feed-forward inference takes <1 second on GPU (H100), rendering exceeds 500 fps, and peak GPU memory is <10 GB (Mieloch et al., 16 May 2026). These are reported as sequence-level timings under the tested hardware assumptions, not universal constants.

6. Position within the broader DSGS landscape

Within the broader literature, DSGS can denote more than the MIV-specific substitution in (Mieloch et al., 16 May 2026). CodecSplat (Yu et al., 25 May 2026) can be read as a concrete instantiation of a decoder-side Gaussian splatting scheme in which the decoder receives an entropy-coded latent tensor plus camera metadata and reconstructs 3D Gaussian primitives from that latent. The pipeline there is explicit: entropy decode $\Sigma \in \mathbb{R}^{3\times 3}$9, reconstruct $\alpha$0 through an overview transform, predict depth, perform post-codec depth-guided multi-view refinement, predict Gaussian attributes, map them from 2D to 3D using camera geometry, and render with a standard 3DGS rasterizer (Yu et al., 25 May 2026). The defining difference from (Mieloch et al., 16 May 2026) is where compression is applied: CodecSplat compresses an intermediate structured 2D Gaussian-generation feature rather than ordinary video atlases or final irregular Gaussian sets.

This intermediate-feature strategy yields very low scene-level rates. On DL3DV and RealEstate10K, CodecSplat reports 23.56–26.36 dB and 24.76–27.05 dB PSNR with only 20.00–107.77 KiB and 3.37–12.51 KiB per scene, respectively (Yu et al., 25 May 2026). The paper characterizes this as roughly one order of magnitude smaller than compressing feed-forward generated Gaussian primitives while maintaining controllable rate-distortion behavior (Yu et al., 25 May 2026). In a decoder-centric taxonomy, this suggests a variant of DSGS where the transmitted object is neither video atlases nor Gaussian attributes, but a compact learned latent that remains structurally aligned with Gaussian generation.

CSGaussian (Tseng et al., 19 Jan 2026) represents a different branch of the decoder-side paradigm: instead of inferring Gaussians from compressed 2D observations, it compresses a 3D Gaussian representation that jointly carries appearance and semantic features so that the decoder can render and also perform segmentation and manipulation (Tseng et al., 19 Jan 2026). Built on Scaffold-GS, it organizes the scene into anchors on a voxel grid, with each anchor carrying offsets, a scaling vector, an anchor feature, and a semantic feature $\alpha$1 shared across its $\alpha$2 Gaussians (Tseng et al., 19 Jan 2026). The framework introduces a lightweight implicit neural representation-based hyperprior for entropy coding and a compression-guided segmentation learning strategy based on quantization-aware training and a quality-aware weighting mechanism (Tseng et al., 19 Jan 2026).

The decoder-side consequences are substantive. CSGaussian is reported to support real-time rendering, 3D semantic segmentation, open-vocabulary queries, click-based selection, and scene manipulation directly from the compressed Gaussian representation, without access to training images, SAM, or other heavy foundation models at the receiver (Tseng et al., 19 Jan 2026). Quantitatively, the paper states that adding semantics increases bitrate by only 2–4% with negligible PSNR change, while over LERF and 3D-OVS it achieves >140× bitrate reduction versus OpenGaussian and >23× versus InstanceGS, with +2–5 percentage points of mIoU (Tseng et al., 19 Jan 2026). In this sense, CSGaussian extends DSGS from decoder-side view synthesis to decoder-side semantic functionality.

Taken together, these works define a coherent landscape. In (Mieloch et al., 16 May 2026), DSGS is a standards-compatible MIV decoder replacement that infers a volumetric scene from compressed atlases and metadata. In (Yu et al., 25 May 2026), it becomes latent-coded decoder-side Gaussian reconstruction from structured intermediate features. In (Tseng et al., 19 Jan 2026), it denotes decoder-side deployment of compressed Gaussian fields carrying both appearance and semantics. The common denominator is that the receiver reconstructs or consumes Gaussian primitives as its primary scene representation, and that downstream rendering or semantic functionality is enabled without transmitting dense geometry in conventional form.

7. Limitations, assumptions, and open directions

The MIV-oriented DSGS formulation in (Mieloch et al., 16 May 2026) is explicitly conditioned on standard MIV capture assumptions: pinhole cameras, known intrinsics and extrinsics, and static scenes (Mieloch et al., 16 May 2026). The reported experiments target static scenes, and the paper notes that current feed-forward 3DGS models are designed per frame; applying them independently to video frames can introduce temporal jitter in Gaussian attributes and slight inter-frame flicker even when inter-view consistency within a frame is good (Mieloch et al., 16 May 2026). This establishes an important distinction between inter-view coherence and temporal coherence.

Sparse-camera edge cases remain a limitation. The sequences where DSGS underperforms DSDE are characterized by relatively few cameras and narrower baselines, conditions under which cost-volume depth estimation is less challenged and can retain an advantage (Mieloch et al., 16 May 2026). The method also deliberately avoids pose-free architectures because MIV already transmits poses (Mieloch et al., 16 May 2026), so its deployment premise is calibrated multiview capture rather than unconstrained view synthesis.

Future work identified in (Mieloch et al., 16 May 2026) includes temporal consistency through dynamic 4D modeling and motion compensation, with GIFStream and D-FCGS named as relevant directions; lightweight geometric assistance via MIV’s Geometry Assistance SEI; and standard co-design that explicitly accounts for the interaction between lossy video compression and decoder-side generative reconstruction (Mieloch et al., 16 May 2026). CodecSplat suggests another direction: moving the coding bottleneck into structured intermediate Gaussian-generation features and controlling quality by a rate parameter $\alpha$3 within a hierarchical learned codec (Yu et al., 25 May 2026). CSGaussian points toward multi-function decoder-side Gaussian representations in which rendering, segmentation, and editing are all supported from a single RD-optimized bitstream (Tseng et al., 19 Jan 2026).

A final misconception worth addressing is that DSGS is simply “3DGS compression.” The MIV formulation is almost the opposite: it sends no 3DGS attributes at all and instead reconstructs Gaussians entirely at the decoder from standard compressed video plus metadata (Mieloch et al., 16 May 2026). CodecSplat and CSGaussian show that adjacent decoder-side designs may instead transmit compressed latents or compressed Gaussian fields (Yu et al., 25 May 2026, Tseng et al., 19 Jan 2026). The broader concept therefore names a family of decoder-centric systems, but the specific term DSGS in immersive-video standards discourse most precisely refers to decoder-side replacement of DSDE by feed-forward 3DGS inference on an unchanged MIV bitstream (Mieloch et al., 16 May 2026).