Controllable Volumetric Rendering

• Controllable volumetric rendering is a suite of techniques that decouples geometry, appearance, and lighting for explicit, user-driven scene editing.
• The approach employs advanced neural representations with editable primitives, enabling operations like texture editing, geometric deformation, and style transfer.
• It leverages differentiable rendering pipelines and acceleration structures to achieve high-fidelity, real-time control over complex volumetric scenes.

Controllable volumetric rendering encompasses a family of computational techniques that enable explicit, flexible manipulation of scene geometry, appearance, lighting, and temporal behaviors within volumetric representations. Distinguished from conventional black-box neural radiance fields (NeRFs), which entangle appearance and geometry in inscrutable latent spaces, controllable models architect separable, interpretable representations and editing primitives. This enables a range of operations including texture editing, geometric deformation, multi-object composition, style transfer, visibility control, pose-conditioned rendering, and efficient spatiotemporal effects—all within an end-to-end differentiable volume rendering pipeline.

1. Foundational Principles and Formulations

Modern volumetric rendering is grounded in the principle of differentiable ray integration through a parameterized radiance field $$V : \mathbb{R}^3 \times S^2 \to \mathbb{R}^3 \times \mathbb{R}_+$$, where each query produces radiance $c(\mathbf{p},\mathbf{v})$ and density $\sigma(\mathbf{p})$. For a camera ray $r(t) = \mathbf{o} + t\mathbf{d}$, pixel color is computed as

$$C(r) = \int_{t_n}^{t_f} T(t) \sigma\bigl(r(t)\bigr) c\bigl(r(t), \mathbf{d}\bigr) \, dt$$

with $T(t) = \exp(-\int_{t_n}^t \sigma(r(s)) ds)$. Discretized quadrature yields

$$C \approx \sum_{i=1}^N T_i\, [1 - \exp(-\sigma_i \Delta_i)]\, c_i$$

where $T_i = \exp(-\sum_{j < i} \sigma_j \Delta_j)$.

Controllability is achieved by explicitly decoupling components:

• Separating geometry and appearance (as in NeuTex's 3D–2D UV unwrapping (Xiang et al., 2021)),
• Constructing editable feature volumes (as in Control-NeRF (Lazova et al., 2022)),
• Encoding deformation fields or graphical cages for shape control (as in VolTeMorph (Garbin et al., 2022)),
• Factorizing radiance fields for spatial, angular, and temporal edits (as in NeuVV (Zhang et al., 2022)),
• Coupling geometry primitives with per-splat textures and shading for non-photorealistic or local effects (as in TexGS-VolVis (Tang et al., 18 Jul 2025)).

2. Techniques for Disentanglement and Control

Disentanglement is a recurring design choice for enabling user-driven or programmatic editing:

• NeuTex introduces explicit 3D–2D mappings via a parameterization network $F_{uv}(x) \to u \in \mathbb{R}^2$ and a separate neural texture $tex(u, d) \to c$, with a cycle-consistency loss enforced by an inverse mapping $F_{uv}^{-1}(u) \to x$ (Xiang et al., 2021).
• Control-NeRF represents each scene as a dense 3D feature volume $V_s:\mathbb{R}^3 \to \mathbb{R}^F$—subject to object-level mixing, geometric warps, or sub-volume cropping and insertion—while the rendering network $R_\theta$ remains scene-agnostic and fixed after pretraining (Lazova et al., 2022).
• TexGS-VolVis decouples Gaussian splat geometry from per-splat textures and programmable shading attributes, enabling stylization or partial editing by manipulating only $\{T_i, k_{*,i}, \beta_i\}$, while keeping geometry frozen for consistency (Tang et al., 18 Jul 2025).
• HVTR fuses low-resolution volumetric cues with high-resolution 2D textural features, where pose and shape are controlled via SMPL parameters and the main user edits flow through the UV manifold encoding and GAN-based textural renderer (Hu et al., 2021).

These frameworks often leverage cycle-consistency, total variation, or geometry- or appearance-specific regularization losses to enforce fidelity and editability.

3. Editing, Deformation, and Stylization Workflows

Controllable volumetric rendering supports an extensive suite of editing paradigms:

• Texture editing via 2D map manipulation: In NeuTex, after scene unwrapping, neural textures can be repainted, pattern-multiplied, or style-swapped; changes propagate through all rendered views (Xiang et al., 2021).
• Geometric deformation: VolTeMorph applies piecewise-linear (e.g., tetrahedral) cages atop static radiance fields. User or physics engine manipulations to cage vertices $X'$ are barycentrically inverted to canonical coordinates for querying the radiance field, supporting real-time, artist-driven or simulation-based deformation (Garbin et al., 2022).
• Scene feature mixing and modular composition: Control-NeRF allows scene mixing via spatial masks $\alpha(x)$, geometric transformation via invertible warps $T$, and feature grid cropping/insertion for object-level operations; these edits remain differentiable and composable (Lazova et al., 2022).
• Non-photorealistic scene editing (NPSE), image/text-driven: TexGS-VolVis integrates image- and text-conditioned stylization losses on per-splat textures, enabling both global and region-restricted style transfers by backpropagating through rendered results under VGG/CLIP features or paired diffusion models. Fine-grained control is achieved via adjustable style weights, lighting, and segmentation thresholds (Tang et al., 18 Jul 2025).
• Visibility management and interactive sparsification: Volume Conductor exposes predicate-based grouping and view-dependent per-instance visibility ratios; importance sorting and context-preserving sparsification allow dynamic, user-driven decluttering of voluminous datasets (Lesar et al., 2022).

4. Acceleration Structures and Real-Time Inference

Efficient inference is a prerequisite for interactive control:

• Sparse octree structures: NeuVV factorizes dynamic neural radiance into spatial–angular–temporal bases, stored in Video Octrees (VOctrees). Octree traversal and per-ray front-to-back compositing produce frame rates in excess of 30 Hz with low memory footprint (Zhang et al., 2022).
• 2D Gaussian splatting: TexGS-VolVis maintains >30 fps for scenes with $60\,\mathrm{K}$ splats at $800^2$ output by rendering camera-facing quads with closed-form differentiable alpha and per-splat shading on the GPU. Depth sorting or hierarchical Z-buffering ensures proper alpha compositing (Tang et al., 18 Jul 2025).
• Tetrahedral acceleration: VolTeMorph constructs GPU-optimized ray-tracing acceleration structures (TLAS) over tetrahedral cages, minimizing per-sample primitive lookup while ensuring numerical stability under deformation (Garbin et al., 2022).
• Hybrid approaches: HVTR reduces computation by using a pose-conditioned, downsampled NeRF (PD-NeRF) for occlusion and geometry, fusing with high-frequency 2D features for rendering in GAN-based U-Nets, thus enabling real-time and high-quality outputs, particularly on human avatars (Hu et al., 2021).

5. Quantitative Evaluation and Limitations

Evaluations typically report image similarity (PSNR, SSIM, LPIPS), rendering speed, and edit consistency metrics. For example:

Method	PSNR (dB)	SSIM	LPIPS
NeRF	30.73	0.938	—
NeuTex	28.23	0.894	—
Control-NeRF	25.635	0.853	0.181
TexGS-VolVis	—	—	—

• NeuTex achieves near-NeRF fidelity with a modest drop (about 2.5 dB PSNR), but enables direct 2D texture editing via the UV parameterization (Xiang et al., 2021).
• Control-NeRF demonstrates <5% metric drop after complex edits (mixing, deformation, object insertion), with average LPIPS of 0.181 and PSNR of 25.635 dB (Lazova et al., 2022).
• VolTeMorph delivers real-time performance and better LPIPS compared to learned deformation NeRFs, with numerically superior novel-view PSNR (∼30.2 dB) in avatar scenarios (Garbin et al., 2022).
• NeuVV supports interactive spatial/temporal manipulations of dynamic volumetric video at 30+ Hz after acceleration (Zhang et al., 2022).

Limitations reported include the need for complete multi-view/segmentation coverage for accurate UV mapping (Xiang et al., 2021), potential drop in photorealism under aggressive stylization (Tang et al., 18 Jul 2025), and challenges in handling topologically complex or dynamic scenes without significant prior construction or optimization time (Zhang et al., 2022, Hu et al., 2021).

6. Application Domains

Controllable volumetric rendering finds diverse applications:

• Photoreal and stylized visualization of complex internal or medical volumes, high-value in scientific visualization (TexGS-VolVis) (Tang et al., 18 Jul 2025).
• Editable avatars and dynamic scene compositing for telepresence, XR, and entertainment, leveraging pose/shape control and spatiotemporal montage (NeuVV, HVTR) (Zhang et al., 2022, Hu et al., 2021).
• Crowded data exploration with smart-visibility controls for biomedical and materials data (Volume Conductor) (Lesar et al., 2022).
• Physics-based simulation and artist-driven shape manipulation, as in VolTeMorph's mesh cage workflows for animation or telepresence (Garbin et al., 2022).
• Hybrid scene modeling for combining scanned data, procedural edits, and stylization, as unified in Control-NeRF's modular editing capabilities (Lazova et al., 2022).

7. Current Challenges and Future Directions

Ongoing research identifies several critical trajectories:

• Scalability to dynamic and open-world scenes: Future extensions of UV mapping, cycle consistency, and neural field factorization over categories, articulated objects, and dynamic environments are required for real-time SLAM and robust look transfer (Xiang et al., 2021, Hu et al., 2021).
• Improved editing granularity: Fine-grained region selection (e.g., 2D-lift-3D segmentation in TexGS-VolVis) and content-aware regularization promise higher-fidelity and less intrusive edits (Tang et al., 18 Jul 2025).
• Incorporation of learned priors and structural constraints: Better UV shape regularization, structure-aware octree pruning, and composable neural descriptors may facilitate more robust and semantically meaningful control (Xiang et al., 2021, Zhang et al., 2022).
• Efficient real-time streaming and low-latency rendering: Optimized GPU kernels, cache-aware hierarchical data structures, and hybrid neural–rasterization pipelines will be crucial for the integration with XR/VR and bandwidth-limited deployments (Zhang et al., 2022, Hu et al., 2021).
• User interface and automation for editing: Bridging the gap between graphical artist tools and programmatic API-driven control remains an active area, especially for integrating text/image-based instructions, visibility predicates, and semantic region detection (Lesar et al., 2022, Tang et al., 18 Jul 2025).

A plausible implication is that the frontier of controllable volumetric rendering will coincide with advances in both geometric representation learning and interfaces for high-level, semantic scene manipulation.