C³G-NeRF: Continuous Conditional NeRF

• The paper introduces a class-continuous conditional NeRF that enables photorealistic multi-view image synthesis with finely controlled, real-valued semantic features.
• It employs a GAN-based framework with separate generators and discriminators, integrating elementwise conditioned latent codes to maintain image fidelity and 3D consistency.
• Experimental results demonstrate state-of-the-art performance with smooth attribute interpolation and robust scene composability across datasets like CelebA, AFHQ, and Cars.

Class-Continuous Conditional Generative Neural Radiance Field (C³G-NeRF) is a 3D-aware generative model that enables photorealistic synthesis of multi-view-consistent images with finely controlled, continuous class-conditional features. Unlike previous conditional NeRF variants limited to discrete or binary labels, C³G-NeRF allows arbitrary, real-valued control over semantic attributes or categories at inference time, maintaining state-of-the-art image fidelity and 3D consistency (Kim et al., 2023).

1. Background and Motivation

3D-aware image generation tasks require models that preserve inter-view consistency while supporting high-resolution and semantically meaningful image synthesis. Neural Radiance Fields (NeRF) represent scenes as a continuous 5D function

$$(x, y, z,\, \theta, \phi) \mapsto (\sigma, R, G, B)$$

mapping each spatial position and viewing direction to a density $\sigma$ and view-dependent color. Volume rendering integrates these outputs to produce color images from arbitrary viewpoints:

$$C(\mathbf r) = \int_{t_n}^{t_f} T(t)\, \sigma(\mathbf x(t), \mathbf d)\, \mathbf c(\mathbf x(t), \mathbf d)\, dt, \quad T(t) = \exp\left(-\int_{t_n}^t \sigma(\mathbf x(s), \mathbf d)\, ds\right)$$

Earlier works (e.g., GRAF, pi-GAN, GIRAFFE) deploy NeRF within a GAN framework but only allow unconditional sampling or discrete class control, precluding smooth manipulation along arbitrary semantic axes. CG-NeRF (Jo et al., 2021) introduced conditioning via image or text, but lacked support for numeric, continuous conditional vectors. C³G-NeRF addresses this limitation by introducing real-valued conditional feature vectors, expanding the space of generatable images to a continuous manifold parameterized by user-specified attributes.

2. Model Architecture

C³G-NeRF adopts a GAN-based architecture with separate modules for the generator and discriminator.

• Generator Inputs
  • Shape noise $(z_s \sim \mathcal N(0, I) \in \mathbb R^{M_s})$
  • Appearance noise $(z_a \sim \mathcal N(0, I) \in \mathbb R^{M_a})$
  • Camera pose $\xi$ (azimuth, elevation)
  • Continuous class condition $c \in \mathbb R^{M_c}$
• Conditional Feature Projection Two linear projections $L_s$, $L_a$ modulate shape and appearance respectively:

$$c_s = L_s(c) \odot z_s, \qquad c_a = L_a(c) \odot z_a$$

where $\odot$ denotes elementwise multiplication, and $M_s$, $M_a$ are shape/appearance latent dimensions. These codes are injected as input to the NeRF backbone, ensuring the generated geometry and photometry are smoothly controllable by $c$.

• Neural Feature Field A multilayer perceptron $h_\theta$ returns a density and feature vector for each spatial query:

$$h_\theta\left(\gamma(\mathbf x), \gamma(\mathbf d), c_s, c_a\right) = (\sigma, f)$$

Here, $\gamma(\cdot)$ denotes Fourier positional encoding, and $f \in \mathbb R^{M_f}$ is an $M_f$-dimensional feature vector.

• Scene Composition For compositional scenes with multiple entities, $N$ separate feature-field decoders produce $(\sigma_i, f_i)$, which are combined:

$$\sigma = \sum_{i=1}^N \sigma_i, \qquad f = \frac{1}{\sigma} \sum_{i=1}^N \sigma_i f_i$$

This weighted average operates per-ray sample, supporting object combinations and backgrounds.

• Volume Rendering & 2D Neural Rendering Volume rendering integrates along rays to produce a low-resolution feature image:

$$v = \sum_{j=1}^T T_j (1 - e^{-\sigma_j \delta_j}) f_j, \qquad T_j = \exp\left(-\sum_{k<j} \sigma_k \delta_k\right)$$

A 2D neural renderer $\pi_{\text{neural}}$ (residual blocks + upsampling) then maps features to RGB.

3. Training Objectives

Training follows an adversarial paradigm with a standard GAN loss and R₁ gradient penalty. Specifically:

$$\mathcal L_{\text{GAN}}(G, D) = \mathbb E_{z_s, z_a, \xi, c}\left[-\log D(G(z_s,z_a,\xi, c), c)\right] \ + \mathbb E_{I \sim p_{\text{data}}} \left[-\log(1-D(I, c)) + \lambda \|\nabla_I D(I, c)\|^2\right]$$

Alternative losses (WGAN-GP, hinge) are also compatible.

To promote smooth semantic interpolation in conditional space, an interpolation consistency loss is optionally introduced:

$$\mathcal L_{\text{interp}} = \left\|G(z, \alpha c_1 + (1-\alpha) c_2) - [\alpha G(z, c_1) + (1-\alpha) G(z, c_2)] \right\|_1$$

This regularizes the generator to respond linearly (in image space) to convex combinations of condition vectors, ensuring a smooth latent manifold.

4. Implementation and Data

• Datasets:
  • AFHQ (≈16k animal faces, 7 classes)
  • CelebA (≈200k faces, 40 binary attributes)
  • Cars (≈9.7k car images, 13 models)
• Resolutions: $64^2$, $128^2$ (all), $256^2$ (AFHQ, Cars)
• Latent dimensions: $M_s=M_a=256$, $M_c$ equals number of classes or attributes, $M_f=128$ or $256$
• Optimization: RMSprop ($3 \times 10^{-4}$ for $G$, $1 \times 10^{-4}$ for $D$), batch size 32, NVIDIA A6000 GPU
• Architecture: Residual modules in all main decoders (3D MLPs, 2D renderer, discriminator) are crucial to avoid vanishing gradients and ensure conditional signal propagation.

5. Experimental Results

C³G-NeRF achieves superior quantitative and qualitative performance in continuous conditional 3D-aware image synthesis:

Dataset/Condition	FID ($128^2$)	Notable Qualitative Results
CelebA (faces)	7.64	Interpolations between facial expressions, e.g., smile strength; consistent novel views
AFHQ (cat)	13.65	Smooth class transitions (e.g., cat↔dog), composition under view rotation
AFHQ (dog)	43.37	Fine detail and consistency across interpolated features
Cars (Peugeot-207i)	71.06	State-preserving generation at full $360^\circ$ rotation
Cars (all)	43.3	Outperforms pi-GAN: FID 43.3 vs. 104.5, KID 0.039 vs. 0.083 (at $128^2$)

Qualitatively, C³G-NeRF demonstrates high-fidelity, multi-view-consistent synthesis, producing smooth transitions even outside the original label range (e.g., extrapolating “smile” strength or mixing animal classes). Unlike prior works, C³G-NeRF does not face notable degradation in single-class FID when conditioned, and composition of multiple entities is supported natively.

6. Comparison to Prior Conditional NeRF Approaches

C³G-NeRF extends beyond previous conditional 3D-aware GANs, such as GRAF, pi-GAN, and GIRAFFE, which lack explicit, continuous control. In contrast to CG-NeRF (Jo et al., 2021), which conditions on image or textual input (via CLIP) but lacks fine-grained manipulability, C³G-NeRF employs real-valued condition vectors, enabling unlimited interpolation and attribute scaling with consistent geometry and appearance.

The use of elementwise projected conditional vectors $L_s(c), L_a(c)$ to modulate latent codes is a key architectural distinction, preventing vanishing gradients and supporting disentanglement. Residual blocks are essential across the architecture for stable conditional training and effective semantic control.

7. Limitations and Future Directions

C³G-NeRF requires accurately labeled datasets and does not support zero-shot generalization or free-form text conditioning. Scaling to higher resolutions ($512^2$ and above), as well as more complex or dynamic scenes (e.g., articulated bodies, lighting), is not yet demonstrated. Inverse inference—recovering latent and condition vectors from real images to enable unsupervised editing—remains an open challenge. Future research may seek to bridge these gaps by integrating zero-shot or text-based conditioning, scalable architectures, and enhanced scene compositionality (Kim et al., 2023).