---
title: Neural Irradiance Volume for Real-Time Rendering
url: https://www.emergentmind.com/topics/neural-irradiance-volume-niv
type: topic
---

# Neural Irradiance Volume for Real-Time Rendering

Neural Irradiance Volume (NIV) is a neural representation designed for real-time rendering of diffuse global illumination, aiming to replace conventional probe-based volumetric irradiance schemes with a learned, highly compact, and memory-efficient model. NIV achieves this by directly regressing a continuous 5D field of irradiance conditioned on position and direction, using neural compression strategies that circumvent the cubic memory costs and structural artifacts of dense probe grids. The approach is distinguished by its suitability for real-time constraints, enabling high-fidelity rendering of indirect lighting effects for both static and dynamic content with a minimal hardware and runtime footprint [2602.12949].

## 1. Formulation of the Irradiance Field

Diffuse irradiance at a given location and orientation, denoted $E(x, n)$, is defined as
$$
E(x,n) = \int_{\Omega} L_i(x, \omega) \, \max(\langle \omega, n \rangle, 0) \, d\omega,
$$
where $x \in \mathbb{R}^3$ is a spatial point, $n \in S^2$ is the target normal, and $L_i$ is the incoming radiance. Conventional probe grids discretize $E(x, n)$ at regular grid locations $x_i$ and encode its angular dependence via spherical harmonics, requiring $O(N^3)$ per-scene memory for a cubic grid of $N$ resolution. NIV models $E(x, n)$ as a continuous function parameterized by a neural network:
$$
E_\theta : (x, n) \mapsto \hat{E} \approx E(x, n),
$$
where $E_\theta$ is instantiated as a compact multilayer perceptron (MLP) with auxiliary position and direction encodings. This architecture decouples representation quality from grid resolution and enables direct volumetric queries at arbitrary $(x, n)$. Embedding strategies include positional (Fourier) encoding for smaller models and learned multi-level hash encoding for scalable compression; the direction $n$ is either embedded with a low-order spherical encoding or fed raw as a 3-vector.

## 2. Model Architecture and Neural Compression

The NIV system comprises two main encoding branches:
- A multi-level hash encoding of $x$, with features $h(x) \in \mathbb{R}^F$ defined as
$$
h(x) = \sum_{\ell=1}^L T_\ell[\mathrm{hash}(\lfloor x \cdot 2^\ell \rfloor)],
$$
where $T_\ell$ are trainable lookup tables and $L$ is the number of hash levels.
- Direction encoding, either via a low-dimensional spherical basis or directly as a 3D vector.

The concatenated feature vector is processed by a 4-layer fully connected MLP (with ReLU activations except the output), whose hidden width (typically 64–256 units) and hash levels ($L$) are adjusted for desired memory and inference speed. No normalization layers are used. Model sizes vary from 0.003 MB (positionally encoded only) up to 5.4 MB (8-level hash, 64-width MLP). Example pseudocode structure is:

```python
function query_NIV(x, n):
    hx = hash_encode(x)           # F-dimensional
    dx = dir_encode(n)            # 3 or higher-dim
    z = concat(hx, dx)
    for i in 1..3:
        z = ReLU(W[i] * z + b[i])
    return W[4]*z + b[4]          # scalar irradiance
```

## 3. Data Preparation and Training Objectives

Training samples $(x_i, n_i, E_i)$ are drawn from path-traced ground-truth indirect irradiance. 20% of the dataset samples are drawn precisely at surface points (with $n_i$ set to the true normal), supporting sharp shadow boundaries and accurate surface details; the remainder are sampled volumetrically. Samples inside geometry, as determined by back-facing normals at first hit, are excluded. The loss function is a stabilized relative mean squared error:
$$
\mathcal{L}(\theta) = \frac{1}{N} \sum_{i=1}^N \frac{(E_\theta(x_i, n_i) - E_i)^2}{\mathrm{stopgrad}(E_\theta(x_i, n_i)^2) + \varepsilon}, \quad \varepsilon = 10^{-2}.
$$
Optimization employs Adam with learning rate decayed from $10^{-2}$ to $10^{-4}$ over approximately 50k iterations. No explicit regularizers are required to manage hash collisions; gradient competition provides implicit spatial allocation.

## 4. Memory Scaling and Empirical Quality Comparison

Traditional probe grids with $N^3$ nodes and 9 SH coefficients at half-precision require $54 N^3$ bytes. NIV total memory consumption is calculated as
$$
\mathrm{Mem}_{\text{NIV}} = T \times 4\ \mathrm{bytes} \times 4 + \sum_{\ell=1}^{4} (\#\mathrm{params}_\ell \times 2\ \mathrm{bytes}) \approx 1\text{–}5\,\mathrm{MB}.
$$
Empirical results on the Sponza scene show that a 1 MB NIV achieves MSE $\approx 9\times10^{-6}$ compared to probe grid MSE $\approx 10^{-4}$ at equivalent budget—a quality improvement of at least $10\times$. MSE-to-memory scaling consistently favors NIV over probe-based approaches across multiple budgets.

| Levels of Hash | Memory (MB) | Full-HD (ms) | MSE (Sponza) |
|:--------------:|:-----------:|:------------:|:------------:|
| 0 (PE only)    | 0.003       | 0.19         | 1.2e–4       |
| 2 levels       | 0.16        | 0.31         | 4.5e–5       |
| 4 levels       | 1.20        | 0.67         | 2.1e–5       |
| 6 levels       | 3.30        | 1.06         | 1.2e–5       |
| 8 levels       | 5.40        | 1.35         | 9.1e–6       |

## 5. Real-Time Inference and Integration Pipeline

At runtime, the scene is rasterized into a G-buffer that records per-pixel $(x, n, \rho)$ (world-space position, normal, and albedo). For each pixel, the system:
1. Computes hash and direction encodings for $(x, n)$.
2. Runs the 4-layer MLP to predict $\hat{E}(x, n)$.
3. Calculates indirect diffuse radiance $L_r(x) = \frac{\rho(x)}{\pi} \hat{E}(x, n)$.
4. Adds direct illumination and emitted light as applicable.

A single full-screen pass suffices. Optional features include half-resolution shading followed by bilinear upsampling (0.37 ms for 8 hash levels) and a dynamic ambient occlusion pass (0.2 ms) focused on dynamic geometry. Complexity scales as $O(L)$ hash lookups and $O(4W^2)$ FLOPs per pixel, with measured latencies of 0.19 ms to 1.35 ms per frame (full HD, RTX 4090, FP16). Integration into graphics pipelines replaces probe-interpolation shaders with a neural volume evaluation on G-buffer inputs.

## 6. Support for Dynamic and Temporal Effects

NIV is volumetric and independent of mesh topology or surface parameterization, permitting queries for novel or moving objects without retraining. Temporal effects, such as time-of-day changes, are supported natively by extending the input with a scalar parameter $\tau$ (e.g., sun angle), encoded by a small Fourier basis:
$$
\hat{E}_\theta(x, n, \tau).
$$
The network is trained on randomized $\tau$ values, and at inference, the current $\tau$ is passed directly to the model. This enables real-time rendering of smoothly varying illumination due to dynamic scene factors without additional runtime cost or model retraining.

## 7. Performance Analysis, Limitations, and Directions for Extension

NIV’s key performance metrics include frame times of approximately 1 ms at full HD resolution and memory footprints between 0.003 MB and 5.4 MB, substantially lower than $64^3$ probe grids ($\sim$14 MB) or production DDGI ($\sim$35 MB). Quality improvements are consistently observed, with NIV exhibiting more accurate indirect lighting, fewer light leaks, and sharper contact shadows. Training ablations demonstrate that culling samples inside geometry reduces error by $\sim$15\%, and surface-biased sampling helps preserve shadow detail. Pre-integrating $E$ rather than regressing raw $L_i$ further accelerates convergence and improves runtime stability.

Current limitations include:
- Restriction to indirect diffuse illumination; direct illumination still requires conventional shadow mapping, and scaling up to scenes with many shadow-mapped lights may incur added cost or noise.
- Large dynamic occluders are not fully modeled; existing AO approximates only local occlusion.
- Glossy transport is not addressed except via deferral to NIV at diffuse bounces. Extension to support glossy materials would require augmenting the model, for example by encoding surface roughness.
- Capacity saturation in very large scenes, suggesting a need for tiled or level-of-detail variants.
- Opportunities remain for further compression (e.g., hash probing, pruning, quantization) and for online updates via fine-tuning on recent samples, trading accuracy for latency as necessary.

NIV is fully differentiable and compatible with inverse-rendering applications, including light optimization and mixed-reality pipelines, contingent on appropriate integration with downstream assets [2602.12949].

Source: https://www.emergentmind.com/topics/neural-irradiance-volume-niv