DiffusionGS: 3D Gaussian Splatting Diffusion

• DiffusionGS is a diffusion-based framework that replaces variable-length discrete Gaussians with continuous functions in latent space for 3D representation.
• It factorizes the process into Gaussian probability, color, and transform functions, leveraging a Gaussian VAE and latent diffusion model for flexible conditioning.
• The method employs octree-guided discretization to extract an arbitrary number of optimized Gaussian primitives, balancing quality (PSNR) and efficiency.

DiffusionGS most commonly denotes a diffusion-based framework for generating native 3D Gaussian Splatting representations by replacing a variable-length, discrete set of Gaussians with continuous functions that can be modeled in latent space. In the formulation introduced as "DiffGS: Functional Gaussian Splatting Diffusion," the method factorizes Gaussian Splatting into a Gaussian Probability Function, a Gaussian Color Function, and a Gaussian Transform Function, trains a Gaussian VAE plus a latent diffusion model over these fields, and then recovers an arbitrary number of Gaussian primitives through octree-guided discretization and optimization (Zhou et al., 2024). The resulting system targets unconditional generation, text- and image-conditioned generation, partial-3DGS completion, and point-to-Gaussian generation while preserving rasterization-based rendering.

1. Problem setting and 3D Gaussian Splatting basis

DiffusionGS arises from a structural mismatch between 3D Gaussian Splatting and standard generative modeling. Native 3DGS represents a scene as an unstructured, discrete set of Gaussian primitives with variable cardinality, whereas most generative frameworks are designed for regular grids, tensors, or fixed-length latent codes. Prior approaches that voxelize Gaussians impose fixed resolution and introduce information loss, which limits fidelity and the number of splats that can be represented (Zhou et al., 2024).

A 3D Gaussian primitive is parameterized by a mean $\mu \in \mathbb{R}^3$, covariance $\Sigma \in \mathbb{R}^{3\times3}$, opacity $\alpha \in [0,1]$, and color $\mathbf{c} \in \mathbb{R}^3$ or spherical harmonics coefficients. Its spatial form is

$$G(\mathbf{x}) \;=\; \alpha \,\mathcal{N}(\mathbf{x};\, \mu, \Sigma) \;=\; \alpha \,\exp\!\left(-\tfrac{1}{2}(\mathbf{x}-\mu)^\top \Sigma^{-1}(\mathbf{x}-\mu)\right).$$

In practice, $\Sigma$ is parameterized by a rotation $R$ and axis-aligned scales $\sigma_x,\sigma_y,\sigma_z$,

$$\Sigma \;=\; R \,\mathrm{diag}(\sigma_x^2, \sigma_y^2, \sigma_z^2)\, R^\top.$$

For rasterization, local linearization of the perspective projection $\pi$ at $\Sigma \in \mathbb{R}^{3\times3}$0 yields the Jacobian $\Sigma \in \mathbb{R}^{3\times3}$1, and the image-plane covariance becomes

$\Sigma \in \mathbb{R}^{3\times3}$2

Per-pixel splats are composited front-to-back with transmittance

$\Sigma \in \mathbb{R}^{3\times3}$3

This rendering model is efficient and supports real-time interaction on modern GPUs, but its underlying representation remains discrete and unstructured, which is precisely the obstacle DiffusionGS attempts to remove.

2. Functionalization of Gaussian splats

The central idea of DiffusionGS is to replace the discrete Gaussian list with three continuous fields over $\Sigma \in \mathbb{R}^{3\times3}$4. This reformulation converts Gaussian Splatting into a representation that is structured enough for diffusion modeling while preserving the semantics of native 3DGS (Zhou et al., 2024).

The first field is the Gaussian Probability Function, or GauPF,

$\Sigma \in \mathbb{R}^{3\times3}$5

It expresses the probability that a spatial location is a Gaussian center. Its supervision is defined from nearest-center distance with truncation $\Sigma \in \mathbb{R}^{3\times3}$6 and projection $\Sigma \in \mathbb{R}^{3\times3}$7,

$\Sigma \in \mathbb{R}^{3\times3}$8

Farther points receive lower probability, and local maxima indicate likely Gaussian centers.

The second field is the Gaussian Color Function, or GauCF,

$\Sigma \in \mathbb{R}^{3\times3}$9

At a Gaussian center $\alpha \in [0,1]$0, the function returns the corresponding color $\alpha \in [0,1]$1, either as RGB or spherical harmonics coefficients. If view dependence is used, color may depend on viewing direction $\alpha \in [0,1]$2 via $\alpha \in [0,1]$3.

The third field is the Gaussian Transform Function, or GauTF,

$\alpha \in [0,1]$4

where $\alpha \in [0,1]$5 is a rotation, $\alpha \in [0,1]$6 produces scales or axis variances, and $\alpha \in [0,1]$7 is opacity. The Gaussian parameters implied by GauTF are

$\alpha \in [0,1]$8

Taken together, $\alpha \in [0,1]$9 provide a continuous geometry likelihood, continuous appearance, and continuous local transform-plus-opacity. The significance of this factorization is not merely representational. It converts the “unstructured set” problem into a continuous-field generation problem, after which a finite Gaussian set can be reintroduced only at the final discretization stage.

3. Gaussian VAE and latent diffusion model

DiffusionGS implements generation through a Gaussian VAE followed by a latent diffusion model. The encoder $\mathbf{c} \in \mathbb{R}^3$0 consumes a fitted 3DGS $\mathbf{c} \in \mathbb{R}^3$1 and maps it to a latent vector,

$\mathbf{c} \in \mathbb{R}^3$2

A triplane decoder $\mathbf{c} \in \mathbb{R}^3$3 expands $\mathbf{c} \in \mathbb{R}^3$4 into a feature triplane $\mathbf{c} \in \mathbb{R}^3$5 with orthogonal planes $\mathbf{c} \in \mathbb{R}^3$6,

$\mathbf{c} \in \mathbb{R}^3$7

For a query point $\mathbf{c} \in \mathbb{R}^3$8, triplane interpolation yields

$\mathbf{c} \in \mathbb{R}^3$9

Three predictor heads then recover the functional fields:

$$G(\mathbf{x}) \;=\; \alpha \,\mathcal{N}(\mathbf{x};\, \mu, \Sigma) \;=\; \alpha \,\exp\!\left(-\tfrac{1}{2}(\mathbf{x}-\mu)^\top \Sigma^{-1}(\mathbf{x}-\mu)\right).$$0

The VAE objective combines reconstruction and KL regularization,

$$G(\mathbf{x}) \;=\; \alpha \,\mathcal{N}(\mathbf{x};\, \mu, \Sigma) \;=\; \alpha \,\exp\!\left(-\tfrac{1}{2}(\mathbf{x}-\mu)^\top \Sigma^{-1}(\mathbf{x}-\mu)\right).$$1

with prior $$G(\mathbf{x}) \;=\; \alpha \,\mathcal{N}(\mathbf{x};\, \mu, \Sigma) \;=\; \alpha \,\exp\!\left(-\tfrac{1}{2}(\mathbf{x}-\mu)^\top \Sigma^{-1}(\mathbf{x}-\mu)\right).$$2.

A latent diffusion model is then trained on $$G(\mathbf{x}) \;=\; \alpha \,\mathcal{N}(\mathbf{x};\, \mu, \Sigma) \;=\; \alpha \,\exp\!\left(-\tfrac{1}{2}(\mathbf{x}-\mu)^\top \Sigma^{-1}(\mathbf{x}-\mu)\right).$$3 to reverse Gaussian noise. Conditioning $$G(\mathbf{x}) \;=\; \alpha \,\mathcal{N}(\mathbf{x};\, \mu, \Sigma) \;=\; \alpha \,\exp\!\left(-\tfrac{1}{2}(\mathbf{x}-\mu)^\top \Sigma^{-1}(\mathbf{x}-\mu)\right).$$4 is encoded by $$G(\mathbf{x}) \;=\; \alpha \,\mathcal{N}(\mathbf{x};\, \mu, \Sigma) \;=\; \alpha \,\exp\!\left(-\tfrac{1}{2}(\mathbf{x}-\mu)^\top \Sigma^{-1}(\mathbf{x}-\mu)\right).$$5 and injected via cross-attention. The training loss is the DDPM-style noise-prediction objective

$$G(\mathbf{x}) \;=\; \alpha \,\mathcal{N}(\mathbf{x};\, \mu, \Sigma) \;=\; \alpha \,\exp\!\left(-\tfrac{1}{2}(\mathbf{x}-\mu)^\top \Sigma^{-1}(\mathbf{x}-\mu)\right).$$6

The latent diffusion architecture follows DALLE-2’s latent diffusion design, with a UNet backbone, sinusoidal time embeddings, and cross-attention blocks. CLIP encoders are used for text and image conditioning, while partial 3DGS conditioning uses a modified PointNet; in the point-to-Gaussian setting, the Gaussian encoder is replaced with a PointNet-style encoder and no LDM is used (Zhou et al., 2024).

This architecture has two consequences. First, diffusion operates in a compressed latent space rather than directly in the high-dimensional function domain. Second, the representation is flexible with respect to conditioning modality because all conditions act through $$G(\mathbf{x}) \;=\; \alpha \,\mathcal{N}(\mathbf{x};\, \mu, \Sigma) \;=\; \alpha \,\exp\!\left(-\tfrac{1}{2}(\mathbf{x}-\mu)^\top \Sigma^{-1}(\mathbf{x}-\mu)\right).$$7 rather than through changes to the Gaussian parameterization itself.

4. Octree-guided discretization and arbitrary-splat extraction

DiffusionGS does not generate a fixed Gaussian set directly. Instead, after diffusion sampling, it decodes continuous fields and then extracts Gaussians at a chosen cardinality $$G(\mathbf{x}) \;=\; \alpha \,\mathcal{N}(\mathbf{x};\, \mu, \Sigma) \;=\; \alpha \,\exp\!\left(-\tfrac{1}{2}(\mathbf{x}-\mu)^\top \Sigma^{-1}(\mathbf{x}-\mu)\right).$$8 through an octree-guided discretization pipeline (Zhou et al., 2024).

The process begins by building an octree over the 3D bounding volume. At each depth, GauPF is evaluated at cell centers and only high-probability cells are subdivided. After $$G(\mathbf{x}) \;=\; \alpha \,\mathcal{N}(\mathbf{x};\, \mu, \Sigma) \;=\; \alpha \,\exp\!\left(-\tfrac{1}{2}(\mathbf{x}-\mu)^\top \Sigma^{-1}(\mathbf{x}-\mu)\right).$$9 levels, the method uniformly samples $\Sigma$0 proxy points $\Sigma$1 within high-probability leaf cells. These proxies are then optimized to maximize GauPF:

$\Sigma$2

The optimized points $\Sigma$3 are treated as refined Gaussian centers.

Attributes are recovered by querying GauCF and GauTF at each refined center:

$\Sigma$4

The final Gaussian set is

$\Sigma$5

with covariance assembled as $\Sigma$6.

A defining property of this stage is variable-$\Sigma$7 support. By choosing $\Sigma$8 at extraction time, the same generated functional representation can yield different Gaussian counts. The reported ablations show a quality–density trade-off: PSNR rises from 28.61 at 50K Gaussians to 34.01 at 350K, and octree depth 10 improves over depth 7 from PSNR 29.77 to 34.01 while sample time increases from 0.15s to 0.58s. Skipping center refinement also degrades performance, lowering PSNR to 30.34. These results indicate that discretization is not a peripheral post-processing step; it is an essential part of the representation-to-rendering bridge.

5. Training protocol, tasks, and empirical characteristics

The training data preparation for the ShapeNet chair and airplane experiments consists of rendering 100 multi-view images per mesh with Blender and fitting 3DGS for each object using vanilla 3DGS optimization. For stable fitting, initialization uses 100K surface points sampled from the ground-truth mesh instead of COLMAP initial points, and Gaussian scales are clipped to a maximum of 0.01. Training is implemented in PyTorch Lightning with Adam at learning rate $\Sigma$9 on $R$0 RTX 3090 GPUs, with convergence reported at approximately five days per class (Zhou et al., 2024).

The method is evaluated on several tasks. In unconditional generation on ShapeNet, it reports Airplane FID-50K = 47.03 with KID-50K = 3.436‰, and Chair FID-50K = 35.28 with KID-50K = 2.148‰. The corresponding baselines include GET3D on chairs at FID 59.51 and KID 2.414, and DiffTF on airplanes and chairs at FID 110.8 and 93.02, with KID 9.173 and 6.708. For Gaussian VAE reconstruction on ShapeNet chair, 350K Gaussians achieve PSNR 34.01, SSIM 0.9879, and LPIPS 0.0149; 100K yields PSNR 30.35, SSIM 0.9838, and LPIPS 0.0151; 50K yields PSNR 28.61, SSIM 0.9787, and LPIPS 0.0251.

The same framework supports conditional generation from text, image, and partial 3DGS, as well as point-to-Gaussian generation. Text conditioning uses a frozen CLIP text encoder and can employ classifier-free guidance at inference. Image conditioning uses a frozen CLIP image encoder and is reported to produce high-fidelity 3DGS matching input-view semantics and geometry. Partial-3DGS conditioning uses a modified PointNet and completes occluded regions to full 3DGS. In the point-to-Gaussian setting, a PointNet-like encoder replaces the GS encoder and the LDM is omitted, focusing instead on fitting GauPF, GauCF, and GauTF fields from point-cloud geometry alone.

The ablation study also isolates several design decisions. Truncation $R$1 in GauPF is beneficial, with “w/o truncation” reducing PSNR to 29.39. A linear projection $R$2 outperforms $R$3, with the “Exponent” variant reporting PSNR 29.74 versus 34.01 for the full method. A triplane decoder improves detail and geometric fidelity relative to an MLP decoder. In efficiency terms, DiffGS reports 127.4M parameters and 9.5s generation time on a single RTX 3090, compared with DiffTF at 929.9M and 99.7s, DreamGaussian at 258.7M and 197s, and LGM at 429.8M and 10.5s.

6. Position in the literature, naming ambiguity, and limitations

Within 3D generative modeling, DiffusionGS occupies a specific position. It is not a 3DGS reconstruction method in the sense of optimizing Gaussians directly from a fixed image set; rather, it is a learned generative model that outputs native Gaussian representations without voxelization (Zhou et al., 2024). Relative to NeRF-based generators such as DiffRF, Volumediffusion, and SSD-NeRF, its advantage is that it emits Gaussians for fast rasterization rather than relying on volume rendering at test time. Relative to Gaussian-based generators that voxelize or discretize Gaussians into volumetric grids, it avoids fixed-resolution bottlenecks and supports arbitrary Gaussian counts. Relative to SDS-based systems such as DreamGaussian, it is data-driven rather than optimization-per-scene.

A common misconception is that “DiffusionGS” refers uniquely to one algorithm. The name has been used for multiple distinct systems. "Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable Single-stage Image-to-3D Generation and Reconstruction" presents a different DiffusionGS that outputs 3D Gaussian point clouds at each denoising timestep and “bakes” Gaussian splatting directly into the denoiser for single-stage image-to-3D generation and scene reconstruction (Cai et al., 2024). "Diffusion-Guided Gaussian Splatting for Large-Scale Unconstrained 3D Reconstruction and Novel View Synthesis" instead uses a multi-view diffusion model to generate pseudo-observations that guide 3DGS optimization under sparse, unconstrained capture conditions (Mithun et al., 2 Apr 2025). "S2D: Sparse to Dense Lifting for 3D Reconstruction with Minimal Inputs" further argues that geometry-aware one-step diffusion lifting and robust optimization are crucial for scene-level consistency under extreme sparsity, and explicitly contrasts its two-fold lifting strategy with DiffusionGS-type single-stage pipelines in general scenes (Ji et al., 11 Mar 2026). Outside 3D graphics, the title "DiffusionGS" has also been used for a query-conditioned diffusion framework for personalized search ranking at Kuaishou, which is unrelated to Gaussian Splatting (Li et al., 25 Aug 2025). This suggests that the label is semantically overloaded and must be interpreted from context.

The principal limitation reported for functional DiffusionGS is appearance failure rather than geometric collapse: it can occasionally generate overly creative or unnatural color schemes, including multicolor chairs. A plausible implication is that the disentanglement and diffusion machinery are stronger at modeling Gaussian placement and local transform than at enforcing appearance priors. The paper therefore identifies stronger appearance priors or constraints as a future direction. At the same time, later scene-level work suggests a broader limitation of direct diffusion-based Gaussian generation under sparse in-the-wild inputs: maintaining long-range 3D consistency across diverse, minimally observed scenes remains difficult, and may require explicit geometry-aware conditioning or robust optimization strategies beyond latent generative modeling alone (Ji et al., 11 Mar 2026).