---
title: 'DeepVoxels: 3D Scene Representation'
url: https://www.emergentmind.com/topics/deepvoxels
type: topic
---

# DeepVoxels: 3D Scene Representation

DeepVoxels is a learned representation for novel view synthesis that introduces a persistent 3D feature embedding organized as a Cartesian 3D grid of persistent embedded features. It encodes the view-dependent appearance of a 3D scene without having to explicitly model its geometry, while combining 3D geometric computer vision with image-to-image mappings based on adversarial loss functions. The model is supervised through a 2D re-rendering loss, enforces perspective and multi-view geometry in a differentiable manner, and is commonly situated within neural rendering as a 3D-aware scene representation between unstructured latent scene models and more explicitly geometric methods [1812.01024][2004.03805].

## 1. Origins and problem formulation

DeepVoxels was proposed against the observation that contemporary generative neural networks can synthesize realistic single images yet fundamentally lack an understanding of 3D structure. In the formulation associated with DeepVoxels, this deficiency appears as an inability to render coherent, multi-view images of the same scene from arbitrary camera poses. The method therefore targets novel view synthesis under known camera geometry, seeking a representation that can preserve multi-view consistency, perspective effects, and occlusion ordering without requiring explicit depth maps, meshes, or 3D models as supervision [1812.01024].

Within the broader neural rendering literature, DeepVoxels is treated as a milestone in constructing 3D-aware, differentiable scene representations. The survey treatment places it among approaches that bridge physically grounded representations and learned priors, in contrast both to classical image-based rendering, which is sensitive to geometry proxy quality, and to unstructured neural scene representations such as GQN, which are limited in geometric consistency. In that framing, DeepVoxels is a voxel-based novel view synthesis method with explicit camera control, differentiable rendering, and instance-specific training [2004.03805].

## 2. Persistent 3D feature embedding

At the core of DeepVoxels is a persistent, high-dimensional feature vector stored in each voxel of a uniform 3D grid covering the scene’s spatial extent in world coordinates. The representation is persistent in the sense that the 3D grid is maintained as a latent state across training steps and views, so the volume is shared and refined over the course of optimization into a scene-specific, canonical 3D representation. This persistent scene representation is intended to encode appearance and structure jointly, while remaining agnostic to explicit geometry supervision [1812.01024].

The construction pipeline begins with a 2D convolutional encoder, described as a U-Net CNN, that extracts deep features from a sampled source image. These 2D features are then lifted into a temporary 3D volume using known camera geometry: each 3D voxel is projected into the source image using camera intrinsics and extrinsics, and its feature is sampled from the source features via bilinear sampling. In the survey description, this operation is also characterized as un-projection into the 3D world coordinate frame by replicating features along camera rays. The persistent grid is then updated by voxel-wise 3D GRUs with parameters shared across the grid; the hidden state of these GRUs constitutes the DeepVoxels representation. A 3D U-Net subsequently performs inpainting and context aggregation over the volume [1812.01024][2004.03805].

The projective relationship used throughout the method is expressed as

$$
\mathbf{u} = \mathbf{K}(\mathbf{R}\mathbf{x} + \mathbf{t}),
$$

where a world-space position $\mathbf{x} \in \mathbb{R}^3$ is mapped to camera-space coordinates $\mathbf{u} = (u,v,d)$. This same differentiable geometry underlies both lifting into the world-aligned grid and projection into target-view volumes [1812.01024].

## 3. View-conditioned rendering and occlusion reasoning

For synthesis from a novel camera pose, DeepVoxels places a virtual camera in world coordinates and resamples the persistent 3D grid into a canonical view volume aligned with that target viewpoint. The projection layer inverse-projects each canonical voxel center back into the world grid and interpolates its feature vector, yielding a target-view-aligned feature volume. This separates scene representation from viewpoint-conditioned rendering: the world-aligned grid stores the scene, while the canonical view volume organizes the scene for a specific query camera [1812.01024].

Occlusion is handled explicitly by a trainable occlusion reasoning module. In the survey description, this module is a 3D U-Net that receives all features along each camera ray and computes per-voxel visibility weights along the ray; these weights are normalized so that along each ray they sum to one. In the original formulation, the occlusion network predicts softmax visibility weights for depth layers in the canonical view volume, encouraging the model to select a single or small number of plausible surfaces along each ray. The final feature for a target pixel is therefore a weighted sum over the ray:

$$
\mathbf{f}_{\text{output}} = \sum_d w_d \mathbf{f}_d .
$$

The resulting 2D feature map is then passed through a 2D U-Net rendering network to generate the RGB image [1812.01024][2004.03805].

A recurrent misconception is that the use of a voxel grid implies explicit geometry reconstruction. DeepVoxels does not require explicit geometry supervision; instead, geometry is induced by the architecture through perspective projection, multi-view consistency, and occlusion reasoning. The occlusion module learns unsupervised depth maps as a byproduct, but those depth estimates emerge from rendering supervision rather than from ground-truth depth labels [2004.03805].

## 4. Supervision, objectives, and operating regime

Training is fully end-to-end and supervised only via 2D image reconstruction. The survey gives the photometric objective in the form

$$
\mathcal{L}_{\text{recon}}(G) = \mathbb{E}_{x,y}\|G(x)-y\|_p,
$$

with $\ell_1$ or $\ell_2$ loss typically used. The original description specifies an $\ell_1$ re-rendering loss over predicted and target images,

$$
L_{\text{recon}} = \frac{1}{N}\sum_{i=1}^{N} \left|\hat{\mathbf{I}}_i - \mathbf{I}_i\right|,
$$

and states that no explicit 3D or depth supervision is required [1812.01024][2004.03805].

DeepVoxels also incorporates a conditional adversarial objective with a patch-based discriminator to promote sharper image synthesis. The adversarial term is written as

$$
\min_G \max_D \; \mathbb{E}_{\mathbf{I}}[\log D(\mathbf{I})] + \mathbb{E}_{\hat{\mathbf{I}}}[\log (1 - D(\hat{\mathbf{I}}))].
$$

In the experimental description, the $\ell_1$ and adversarial losses are combined with a weighting of $200\!:\!1$. Operationally, the training set consists of registered, posed multi-view RGB images of a static scene, with intrinsic and extrinsic camera parameters known from structure-from-motion. During each training step, one source view and two target views are sampled: the source view is lifted and integrated into the persistent grid, and the target views are rendered and compared to ground-truth images. At test time, only the learned DeepVoxels grid is needed; no input images are required for inference [1812.01024].

The survey’s summary table further characterizes DeepVoxels as requiring videos, taking images and camera parameters as inputs, producing images as outputs, operating on single objects, providing explicit camera control, using a differentiable graphics module, and being instance-specific rather than general. Multi-modal synthesis and temporal coherence are not supported or enforced in that summary [2004.03805].

## 5. Empirical performance and the DeepVoxels benchmark

In the original evaluation, DeepVoxels significantly outperformed several baselines on synthetic scenes. The reported average PSNR was **30.55dB** for DeepVoxels versus **23.63dB** for the best baseline, Pix2Pix, and the reported average SSIM was **0.97** versus **0.92**. The baselines listed in the experimental summary include nearest neighbor retrieval, 2D-to-2D image translation with Pix2Pix, deep autoencoders with pose-conditioned latent codes, and rotation-equivariant embeddings. Qualitatively, the model was reported to generate sharp images, avoid “shining through” artifacts from occluded surfaces, and generalize to nearby unseen viewpoints; it was also demonstrated on real scenes such as busts and globes [1812.01024].

A later benchmark convention uses the name “DeepVoxels dataset” for a standard single-scene novel view synthesis testbed. In the description used by subsequent work, this dataset consists of four objects—vase, pedestal, chair, and cube—rendered from a dense set of viewpoints, with **479 poses for training on the northern hemisphere, 30 for validation, and 1000 for testing on an Archimedean spiral**, each at a resolution of **$512 \times 512$**. The objects are described as having relatively simple geometry, making the benchmark suitable for analyzing scene representation accuracy and rendering precision [2207.14741].

The benchmark remained relevant in later comparisons. On this DeepVoxels dataset, NeRFA reported **PSNR 40.31**, **SSIM 0.992**, and **LPIPS 0.020**, compared with **40.14 / 0.990 / 0.021** for NeRF, **35.51 / 0.975 / 0.030** for NerFormer, **34.21 / 0.986 / 0.045** for LLFF, **33.15 / 0.947 / 0.069** for SRN, and **29.57 / 0.930 / 0.094** for Neural Volumes. In the accompanying qualitative discussion, NeRFA is described as producing sharper object boundaries and finer details than NeRF, NerFormer, or DeepVoxels-related approaches, which indicates the continued use of DeepVoxels as a reference point in single-scene view synthesis even after the emergence of NeRF-style methods [2207.14741].

## 6. Limitations, significance, and later developments

The principal limitations attributed to DeepVoxels are instance-specific training, memory consumption, and restriction to static scenes. The survey states that a new grid must be learned for each new scene, which limits broad applicability and real-time adaptation. Because voxel grids scale cubically with spatial resolution, memory usage restricts the ability to represent large scenes or very fine detail. The original method also assumes static scenes with known posed images and does not readily extend to dynamic content [2004.03805].

At the same time, DeepVoxels is repeatedly described as a bridge between explicit 3D volumetric representations and later neural implicit models. A later comparison states that DeepVoxels formed such a bridge but that its reliance on explicit geometric computation and memory-heavy grids limited scalability and flexibility. This historical role is visible in how subsequent methods define themselves against its design choices. NeRFA contrasts an explicit 3D voxel feature grid with sampled ray points processed by transformer-based ray and pixel attention and replaces explicit volumetric rendering variables with soft latent feature modulation [2207.14741]. Dynamic voxel grid optimization contrasts a fixed canonical voxel grid storing persistent 3D features with a dynamically subdivided grid that stores SDF and color under RGB-D supervision [2304.06178]. Vox-Surf contrasts voxel feature storage for view synthesis with a sparse voxel-based implicit surface representation that stores geometry and appearance at voxel corner vertices and supports surface extraction and normal computation [2208.10925]. VoxNeRF contrasts dense 3D voxel grids and learned 2D-to-3D feature projection with a Sparse Voxel Octree geometry prior, voxel-guided sampling, and multiresolution hash grids for indoor view synthesis [2311.05289].

DeepVoxels therefore occupies a specific position in neural rendering: it established that a persistent scene-specific 3D feature grid, trained only from posed RGB observations and 2D re-rendering losses, can enforce perspective, visibility, and multi-view consistency without direct 3D supervision. A plausible implication is that its enduring importance lies less in the exact voxel-grid implementation than in the architectural claim that 3D-aware latent structure can be learned as an internal scene representation and then rendered differentiably into novel views.

Source: https://www.emergentmind.com/topics/deepvoxels