DeepDVR: Neural Volume Rendering

• DeepDVR is a generalized direct volume rendering framework that replaces manual transfer functions with end-to-end neural modules.
• It employs 3D CNNs and MLPs to extract features and compute emission, opacity, and latent colors, enabling semantically rich volumetric visualization.
• The architecture supports diverse designs and training strategies like stepsize annealing, optimizing rendering quality and efficiency for scientific and medical applications.

Deep Direct Volume Rendering (DeepDVR) is a generalization of the classical Direct Volume Rendering (DVR) paradigm, enabling the integration of deep neural networks into the core DVR pipeline for rendering scientific and medical volumetric data. Unlike traditional DVR, which relies on explicit, hand-designed transfer functions for mapping scalar field values to emission and absorption properties, DeepDVR employs neural feature extractors and multilayer perceptrons (MLPs) to learn these mappings end-to-end from example images. The approach introduces a latent color space and supports architectures that can be optimized directly from image space, eliminating the need for manual transfer function design and facilitating the extraction of semantically meaningful features from volumetric data (Weiss et al., 2021).

1. Mathematical Formulation and Latent Color Space

Classical DVR computes image color using the emission–absorption model, where each position $x \in \mathbb{R}^3$ in the volume is associated with emissive color %%%%1%%%% and absorption coefficient $\kappa(x) \in \mathbb{R}$. Rendering proceeds by integrating along viewing rays:

$$C = \int_{0}^{\infty} c(x(t))\exp\left(-\int_0^t \kappa(x(\tau)) d\tau \right) dt,$$

with $x(t) = x_0 + t d$ defining the ray.

Discretization with step $\Delta t$ enables practical front-to-back alpha compositing:

• $$C_i = c_i \Delta t,\quad A_i = 1 - \exp(-\kappa_i \Delta t)$$
• $$C'_i = C'_{i-1} + (1 - A'_{i-1}) C_i,\quad A'_i = A'_{i-1} + (1 - A'_{i-1})A_i$$
• Initialization: $C'_0 = A'_0 = 0$

DeepDVR generalizes this to an $n_C$-dimensional latent color space. The algorithm replaces all hand-crafted transfer functions with neural modules:

• Feature extraction: $F(x) = \mathcal{E}(I(x))$
• Latent emission and opacity: $$c_i = \mathcal{C}(F_i),\;\; \kappa_i = \mathcal{K}(F_i)$$
• Alpha blending: as in classical DVR, but with latent $C_i$
• Decoding: $C^{RGB} = \mathcal{D}(C'_\infty)$

Here, $I(x)$ denotes the raw input volume, $\mathcal{E}$ is a 3D CNN feature extractor, $\mathcal{C}$ and $\mathcal{K}$ are two-layer MLPs for emission and opacity, and $\mathcal{D}$ maps accumulated latent color to RGB. Joint optimization of these modules obviates the need for hand-tuned transfer functions and enables feature learning directly from data (Weiss et al., 2021).

2. DeepDVR Network Architectures

DeepDVR is a family of architectures instantiating $\mathcal{E}$, $\mathcal{C}$, $\mathcal{K}$, $\mathcal{D}$, and the blending procedure in different ways:

Model	Brief Description	Parameters
Lookup	Identity extractor; 1D LUTs for $\mathcal{C}, \mathcal{K}$	$\approx$1K
RenderNet	Heavy 3D CNN $\rightarrow$ MLP $\rightarrow$ 2D CNN (baseline)	$\approx$226M
VNet-4-4	4-level 3D VNet, RGB+$\alpha$ direct output	$\approx$45M
VNetL16-4	Light VNet (16 ch.), 3-layered $\mathcal{C}$, $\mathcal{K}$ MLPs	$\approx$12.3M
VNetL16-17	Same VNet, $\mathcal{C}$ identity, 16$\rightarrow$3 $\mathcal{D}$	$\approx$12.3M
DVRNet	Multiscale: 3D VNet encoder, multi-scale raymarching, 2D UNet	$\approx$25M

In VNet-x-x models, the feature extractor is a 3D VNet CNN producing voxel-wise latent vectors; emission $\mathcal{C}$ and opacity $\mathcal{K}$ are MLPs with ReLU and sigmoid activations; the decoder $\mathcal{D}$ varies from identity to an MLP or a 2D CNN for view-dependent effects. DVRNet, the multi-scale hybrid, encodes volume features at four scales using a VNet, raymarches each, and aggregates results using a 2D UNet—a structure optimal for capturing both semantic volume context and view-dependent phenomena (Weiss et al., 2021).

3. Stepsize Annealing and Efficient Training

A critical bottleneck in DeepDVR training is the ray sampling rate $s$ (voxels per unit length): higher values improve quality, but increase linear computational cost. Small fixed $s$ enables fast convergence but leads to underfitting, while large $s$ is accurate but slow. DeepDVR introduces stepsize annealing to accelerate training:

$s(e) = s_l (1 - (e/E)^2) + s_h (e/E)^2,$

where $e$ is the epoch index, $E$ is total epochs, with $s_l$ (start) $\approx 0.1$, $s_h$ (end) $\approx 2.0$ for practical regimes.

Typically, epochs progress from very coarse to fine raymarching, with batches optionally jittering ray positions for regularization. Stepsize annealing reduces training time by $\approx$33% compared to fixed high $s$, without compromising rendering quality in transfer function learning tasks (Weiss et al., 2021).

4. Supervision Protocols and Data Regimes

Training supervises the output RGB image against reference images, using:

$$\mathcal{L} = \text{MSE}(Y, \hat{Y}) + [1 - \text{SSIM}(Y, \hat{Y})]$$

where $\text{MSE}$ is mean squared error and $\text{SSIM}$ denotes structural similarity. Validation uses SSIM, while evaluation includes perceptual metrics such as Fréchet Inception Distance (FID) and Learned Perceptual Image Patch Similarity (LPIPS), none of which enter the loss.

Three dataset regimes are used:

• Image-based TF reconstruction: single-volume transfer function inversion using expert hand-tuned LUTs for 5 volumetric datasets (from the Volume Library), generating training/validation image pairs by GPU rendering.
• Hand-painted reference inversion: two volumes with expert-edited semantic labeling via painting of rendered images; network trained to invert label edits.
• Generalizable multi-volume rendering: 27 CT angiography datasets with semantic coloring and shading (label- and view-dependent lighting provided for data generation only, not for network input).

DeepDVR thus supports both single-volume targeted transfer learning and more complex multi-volume, multi-view generalization tasks (Weiss et al., 2021).

5. Empirical Evaluation and Comparative Performance

Extensive experiments compare DeepDVR variants, classical lookup-based networks, and RenderNet baselines across all regimes:

(a) Image-based Transfer Function Learning

• LUT-based TFs train rapidly, with time $\propto s$; no accuracy benefit beyond $s=1$.
• Stepsize annealing achieves fixed-$s=2$ quality in 33% less time.
• MLP-based TFs are slower (2–6$\times$), exhibit higher variance, and frequently collapse to "dead-opacity" (black output).

(b) Hand-painted Reference Inversion

Model	LPIPS↓ Bonsai3/Pig	FID↓ Bonsai3/Pig	SSIM↑ Bonsai3/Pig	Time
Lookup	0.29/0.25	234/164	0.53/0.64	6 min
RenderNet	0.49/0.34	274/274	0.27/0.43	1 h33 m
VNet4-4	0.16/0.19	208/157	0.83/0.87	8 h32 m
VNetL16-4	0.10/0.12	149/112	0.92/0.93	9 h45 m
VNetL16-17	0.10/0.08	171/107	0.92/0.93	9 h46 m
DVRNet	0.08/0.10	137/93	0.91/0.93	2 h25 m

DVRNet and VNetL16 variants most accurately recover edited color semantics, outperforming both LUT and RenderNet baselines in detail and perceptual quality.

(c) Multi-volume Generalization (“Kidney”/“Shaded” test sets)

Model	LPIPS↓ / SSIM↑ Kidney/Shaded	FID↓ Kidney/Shaded	Training
Lookup	0.263/0.706	0.280/0.677	188/233
RenderNet	0.562/0.537	0.480/0.622	387/263
VNet4-4	0.268/0.699	0.287/0.664	280/236
VNetL16-4	0.215/0.694	0.241/0.677	256/226
VNetL16-17	0.211/0.690	0.274/0.656	245/231
DVRNet	0.273/0.699	0.249/0.734	314/228

VNetL16-4/17 are optimal for semantic coloring (“Kidney”), while DVRNet substantially outperforms others for view-dependent lighting (“Shaded”), due to its 2D decoder’s ability to model view-space effects.

6. Architectural Insights and Comparative Analysis

Experiments and ablations reveal several robust patterns:

• Stepsize annealing consistently accelerates convergence (by 30–40%).
• Ray jittering is only beneficial at very low sampling rates ($s < 0.5$).
• 1D intensity MLP TFs are often unstable and suboptimal relative to LUTs for scalar-to-opacity/emission mappings.
• Explicit DVR modeling (separating feature extraction, transfer, compositing, and decoding) outperforms “black-box” approaches like RenderNet.
• Hybrid architectures (DVRNet) with mixed 3D/2D processing are essential for capturing view-dependent effects, such as lighting and shadow, that simple volume-encoding architectures cannot easily model (Weiss et al., 2021).

7. Context and Applications

DeepDVR unifies scientific volume rendering and deep learning, making it possible to learn feature-extracting and visualization policies directly from expertly generated images. Applications include medical imaging, scientific data visualization, and custom rendering tasks requiring either transfer function inversion (e.g., from expert-adjusted images) or the creation of generalizable renderers that can extrapolate to unseen volumetric data for tasks like semantic colorization or physically motivated shading (Weiss et al., 2021).

The framework’s primary distinction is its combinatory, modular architecture and latent color space, which facilitate end-to-end trainable DVR generalizations without manually defined transfer functions or hand-engineered feature spaces. This suggests potential for extension to more advanced semantic, interactive, or physically-based visualization workflows.