Splatent: 3D Diffusion Latent Refinement

• Splatent is a diffusion-based enhancement framework for novel view synthesis that leverages 3D Gaussian Splatting and latent diffusion to refine view-inconsistent details.
• It separates the coarse 3D low-frequency structure from high-frequency details recovered via a 2D multi-view attention diffusion step at render time.
• Empirical results show Splatent improves PSNR, SSIM, and FID scores on benchmarks like DL3DV-10K and LLFF, establishing a new state-of-the-art in VAE latent radiance-field reconstruction.

Splatent is a diffusion-based enhancement framework for novel view synthesis that operates on top of 3D Gaussian Splatting in the latent space of variational autoencoders used by diffusion models. It was introduced in “Splatent: Splatting Diffusion Latents for Novel View Synthesis” (Hirschorn et al., 10 Dec 2025). The method addresses a fundamental limitation of VAE-latent radiance-field reconstruction: modern continuous VAEs encode fine details in a way that is not equivariant across views, so when a 3D representation is fit directly to per-view latents, high-frequency components cancel out and decoded renderings exhibit blurred textures and missing small features. Splatent’s key insight is to let the 3D model recover only the low-frequency structure in latent space and to restore the missing high frequencies in 2D, at render time, by conditioning a single-step latent diffusion model on both the novel-view latent and a small set of nearby reference latents (Hirschorn et al., 10 Dec 2025).

1. Conceptual basis

Radiance field representations have recently been explored in the latent space of VAEs that are commonly used by diffusion models. This direction offers efficient rendering and seamless integration with diffusion-based pipelines (Hirschorn et al., 10 Dec 2025). The central difficulty, however, is that the VAE latent space lacks multi-view consistency. High-frequency latent components disagree from one camera to the next, even when the underlying RGB observations depict the same scene content.

When each observed RGB view $I_i$ is encoded through a frozen VAE encoder $\mathcal{E}$, the resulting latent is

$$z_i = \mathcal{E}(I_i)\;\in\mathbb{R}^{h\times w\times d}\,.$$

A latent radiance field based on 3D Gaussian Splatting represents the scene by Gaussians

$$G_j = (\mu_j\in\mathbb{R}^3,\;\Sigma_j\in\mathbb{R}^{3\times3},\;\alpha_j\in\mathbb{R},\;f^z_j\in\mathbb{R}^d)\,.$$

A novel-view latent $\hat z$ is rendered by splatting all Gaussians into camera $C$. In this setting, the procedure faithfully reconstructs low-frequency content, but high frequencies encoded by $\mathcal{E}$ differ across training views; as a result, rendered latents lose high-frequency detail and the decoder outputs blurred textures and often misses small features altogether (Hirschorn et al., 10 Dec 2025).

Existing approaches in the paper’s framing expose a trade-off. LRF attempts to fix the issue by fine-tuning the VAE for 3D consistency, but this degrades the VAE’s native reconstruction quality. Other works use video diffusion on RGB outputs, but risk hallucination and are not end-to-end latent pipelines (Hirschorn et al., 10 Dec 2025). Splatent is designed to avoid both failure modes by keeping the underlying VAE frozen and shifting detail recovery from 3D latent fitting to 2D multi-view latent refinement.

2. Problem formulation in latent 3D Gaussian Splatting

In Splatent, the 3D reconstruction stage uses latent-space 3D Gaussian Splatting. For a pixel $u$, the rendered latent is expressed as

$\hat z(u) \;=\;\sum_j w_j(u)\,f^z_j,$

with weights proportional to Gaussian opacity and projected distance:

$w_j(u)\;\propto\;\alpha_j\exp\!\Bigl[-\tfrac12\|\pi_j(u)-\mu_j\|_{\Sigma_j^{-1}^2\Bigr].$

In practice one normalizes the $\mathcal{E}$0. The paper also gives the normalized splatting weights as

$\mathcal{E}$1

The practical consequence is a separation between coarse scene structure and view-inconsistent detail. Because fine details conflict across views in latent space, the optimized $\mathcal{E}$2 capture only coarse, low-frequency geometry and appearance (Hirschorn et al., 10 Dec 2025). The paper characterizes this directly: the spectral magnitude of rendered latents shows all high frequencies washed out, while the ground-truth latents contain rich detail.

This suggests that the limiting factor is not merely the expressivity of 3D Gaussian Splatting, but the interaction between a frozen, view-inconsistent latent space and a 3D representation that enforces cross-view agreement. A plausible implication is that attempting to restore these details directly in 3D can be structurally mismatched to the representation.

3. Two-stage architecture

Splatent consists of two stages (Hirschorn et al., 10 Dec 2025).

First, input views $\mathcal{E}$3 are encoded to $\mathcal{E}$4, and a set of Gaussians $\mathcal{E}$5 is optimized so that the rendered latents $\mathcal{E}$6 match the training latents in an $\mathcal{E}$7 sense plus optional perceptual VGG loss in latent space. Because fine details conflict, this stage captures only coarse, low-frequency geometry and appearance.

Second, for each novel camera, the method renders the degraded latent $\mathcal{E}$8, selects $\mathcal{E}$9 nearest training views, and encodes them to reference latents $$z_i = \mathcal{E}(I_i)\;\in\mathbb{R}^{h\times w\times d}\,.$$0. The $$z_i = \mathcal{E}(I_i)\;\in\mathbb{R}^{h\times w\times d}\,.$$1 latents are tiled into a single grid tensor

$$z_i = \mathcal{E}(I_i)\;\in\mathbb{R}^{h\times w\times d}\,.$$2

and reshaped to $$z_i = \mathcal{E}(I_i)\;\in\mathbb{R}^{h\times w\times d}\,.$$3. A single-step denoising diffusion model $$z_i = \mathcal{E}(I_i)\;\in\mathbb{R}^{h\times w\times d}\,.$$4 with self-attention is then applied:

$$z_i = \mathcal{E}(I_i)\;\in\mathbb{R}^{h\times w\times d}\,.$$5

$$z_i = \mathcal{E}(I_i)\;\in\mathbb{R}^{h\times w\times d}\,.$$6

plus feed-forward layers. By attending across all $$z_i = \mathcal{E}(I_i)\;\in\mathbb{R}^{h\times w\times d}\,.$$7 views, information—especially high-frequency edges and textures—flows from the reference latents into the top-left slot, which corresponds to $$z_i = \mathcal{E}(I_i)\;\in\mathbb{R}^{h\times w\times d}\,.$$8 (Hirschorn et al., 10 Dec 2025). The enhanced latent $$z_i = \mathcal{E}(I_i)\;\in\mathbb{R}^{h\times w\times d}\,.$$9 is extracted from that slot and decoded via $$G_j = (\mu_j\in\mathbb{R}^3,\;\Sigma_j\in\mathbb{R}^{3\times3},\;\alpha_j\in\mathbb{R},\;f^z_j\in\mathbb{R}^d)\,.$$0 to RGB.

The paper emphasizes that the architecture never touches 3D during refinement: all geometry comes from the frozen 3DGS model, and detail recovery is purely a 2D multi-view attention diffusion step (Hirschorn et al., 10 Dec 2025). This division of labor is the defining architectural property of Splatent.

4. Diffusion refinement and optimization objective

The refinement stage uses a single-step DDIM variant. The diffusion update is written as

$$G_j = (\mu_j\in\mathbb{R}^3,\;\Sigma_j\in\mathbb{R}^{3\times3},\;\alpha_j\in\mathbb{R},\;f^z_j\in\mathbb{R}^d)\,.$$1

but in practice only one step is fine-tuned at noise level $$G_j = (\mu_j\in\mathbb{R}^3,\;\Sigma_j\in\mathbb{R}^{3\times3},\;\alpha_j\in\mathbb{R},\;f^z_j\in\mathbb{R}^d)\,.$$2, so that a single call of $$G_j = (\mu_j\in\mathbb{R}^3,\;\Sigma_j\in\mathbb{R}^{3\times3},\;\alpha_j\in\mathbb{R},\;f^z_j\in\mathbb{R}^d)\,.$$3 suffices to recover $$G_j = (\mu_j\in\mathbb{R}^3,\;\Sigma_j\in\mathbb{R}^{3\times3},\;\alpha_j\in\mathbb{R},\;f^z_j\in\mathbb{R}^d)\,.$$4 (Hirschorn et al., 10 Dec 2025).

Attention across views is defined as

$$G_j = (\mu_j\in\mathbb{R}^3,\;\Sigma_j\in\mathbb{R}^{3\times3},\;\alpha_j\in\mathbb{R},\;f^z_j\in\mathbb{R}^d)\,.$$5

with $$G_j = (\mu_j\in\mathbb{R}^3,\;\Sigma_j\in\mathbb{R}^{3\times3},\;\alpha_j\in\mathbb{R},\;f^z_j\in\mathbb{R}^d)\,.$$6. In operational terms, the model uses self-attention over a concatenated grid of the degraded novel-view latent and nearby reference latents, allowing cross-view transfer of high-frequency structure without modifying the 3D representation itself.

The total training loss is

$$G_j = (\mu_j\in\mathbb{R}^3,\;\Sigma_j\in\mathbb{R}^{3\times3},\;\alpha_j\in\mathbb{R},\;f^z_j\in\mathbb{R}^d)\,.$$7

A central design choice is that the VAE remains frozen. The paper states that this preserves the reconstruction fidelity and generalization capabilities of a large pretrained autoencoder (Hirschorn et al., 10 Dec 2025). This suggests that Splatent should be read not as a replacement for latent-space 3D reconstruction, but as a compensatory refinement layer specialized for the view-inconsistent part of the latent spectrum.

5. Benchmarks and empirical results

Splatent is evaluated on three benchmarks: DL3DV-10K, described as real-world indoor/outdoor with 140 scenes; LLFF with 8 forward-facing scenes; and Mip-NeRF360 with 9 complex scenes (Hirschorn et al., 10 Dec 2025). The reported metrics are PSNR$$G_j = (\mu_j\in\mathbb{R}^3,\;\Sigma_j\in\mathbb{R}^{3\times3},\;\alpha_j\in\mathbb{R},\;f^z_j\in\mathbb{R}^d)\,.$$8, SSIM$$G_j = (\mu_j\in\mathbb{R}^3,\;\Sigma_j\in\mathbb{R}^{3\times3},\;\alpha_j\in\mathbb{R},\;f^z_j\in\mathbb{R}^d)\,.$$9, LPIPS$\hat z$0, and FID$\hat z$1.

In the dense setting, with 30 training views except that LLFF uses 1/8 sampling, Splatent establishes a new state-of-the-art for VAE latent radiance field reconstruction (Hirschorn et al., 10 Dec 2025).

Benchmark	Splatent	Comparison in the paper
DL3DV-10K PSNR	21.94	LRF 20.19
DL3DV-10K SSIM	0.692	LRF 0.619
DL3DV-10K LPIPS	0.265	LRF 0.322
DL3DV-10K FID	35.6	LRF 75.3
LLFF PSNR	19.57	17.98
Mip-NeRF360 PSNR	20.42	19.08

In the sparse setting with 5 views, Splatent again leads by $\hat z$2 dB in PSNR, halves FID relative to baselines, and shows markedly sharper brick textures, foliage and text in the decoded images (Hirschorn et al., 10 Dec 2025). The paper further reports integration with MVSplat360, a feed-forward latent 3DGS model. On DL3DV-10K with 5 views, the MVSplat360 baseline has PSNR 16.69 and SSIM 0.514; after Splatent refinement these rise to 17.98 and 0.531, with LPIPS dropping from 0.431 to 0.378 (Hirschorn et al., 10 Dec 2025). Qualitatively, MVSplat360 alone hallucinates windows, trees or small props; with Splatent it recovers them faithfully from the few input images.

The ablation on the number of reference views $\hat z$3 isolates the role of multi-view attention:

Reference views $\hat z$4	PSNR	FID
0	19.47	83.7
1	21.61	38.0
3	21.94	35.6
5	21.96	35.2

The paper states that this confirms that multi-view attention is the key to hallucination avoidance and high-frequency recovery (Hirschorn et al., 10 Dec 2025). A plausible implication is that the reference set primarily supplies consistent high-frequency evidence rather than coarse geometric information, since the geometry is already fixed by the 3DGS stage.

6. Limitations, scope, and prospective extensions

Splatent operates in a compressed VAE latent, so it often must reconstruct details that have been discarded at encoding time; the paper characterizes this as inherently harder than refining RGB splats, where all pixels remain in the decoded image (Hirschorn et al., 10 Dec 2025). In scenarios where traditional RGB-space Gaussian splatting already produces artifact-free renderings, one may prefer that simpler pipeline. In extreme cases of view sparsity or highly reflective surfaces, the 3DGS geometry can be too coarse, limiting what even the diffusion step can recover. The method is also upper-bounded by the pre-trained VAE’s capacity: any systematic bias in $\hat z$5 or $\hat z$6 cannot be removed without fine-tuning.

The future directions identified in the paper are jointly fine-tuning the VAE for better multi-view latent consistency, for example by adding equivariance regularizers; extending Splatent to dynamic and deformable scenes; and exploring multi-step diffusion refinements that trade off speed for even sharper detail (Hirschorn et al., 10 Dec 2025). Another avenue is to combine Splatent with text- or geometry-guided priors, enabling interactive editing of the recovered high frequencies in novel views.

Within the paper’s own framing, Splatent is the first principled framework to “splat” diffusion latents in 3D and then use 2D multi-view attention to restore the lost high frequencies, while never fine-tuning the underlying autoencoder (Hirschorn et al., 10 Dec 2025). Its significance lies in making VAE-latent radiance-field reconstruction compatible with faithful detail recovery without abandoning frozen pretrained VAEs or resorting to RGB-space hallucination-prone post-processing.