Polarization-Aware NeRF

• Polarization-aware NeRFs are neural 3D reconstruction techniques that integrate polarization measurements to improve the recovery of complex geometries and material properties.
• They combine implicit scene representations with polarization cues (e.g., Stokes parameters, DoP, and AoP) to disentangle diffuse and specular radiance contributions.
• Frameworks like GNeRP and PANDORA demonstrate superior performance on reflective scenes, offering promising applications in robotics, industrial inspection, and photorealistic rendering.

Polarization-aware Neural Radiance Fields (NeRF) are a class of neural implicit 3D reconstruction algorithms that integrate polarization measurements into the neural radiance field or signed-distance field (SDF) framework. These methods combine the modeling power of neural fields for geometric and photometric scene representation with the unique normal and material cues provided by polarimetric imaging, enabling more accurate and robust inference of 3D shape and reflectance, especially for reflective or mixed-material scenes. Two key frameworks in this line of research are GNeRP (Gaussian-guided Neural Reconstruction with Polarization) (Yang et al., 18 Mar 2024) and PANDORA (Polarization-Aided Neural Decomposition of Radiance) (Dave et al., 2022).

1. Architectural Foundations

Polarization-aware NeRF models are built on implicit scene representations combining geometry and radiance fields, enhanced with polarization supervision mechanisms.

• GNeRP augments a standard SDF-based NeRF (such as NeuS) with two MLPs:
  • The geometry network $f_{geo}: \mathbb{R}^3 \to (d(x), f(x))$ outputs a signed distance $d(x)$ and a local feature $f(x)$.
  • The radiance network $c_{rad}: (x, n, d, f(x), d_{view}) \to c$ predicts color, where $n = \nabla_x d(x)$ and $d_{view}$ is the viewing direction.

The SDF-derived density $\sigma(x)$ is Laplace-based:

$$\Phi_s(d) = \frac{1}{1+e^{-s d}} \ \alpha_i = 1 - \exp\left(-\sigma_i \delta_i\right) \ \sigma_i \equiv s \, \Phi_s(d(x_i)) (1-\Phi_s(d(x_i)))$$

Volume rendering uses the radiance prediction:

$$\hat C(r) = \sum_{i=1}^K T_i \alpha_i c_i(x_i, n_i, d, d_f)$$

where $T_i = \exp\left(-\sum_{j<i} \alpha_j\right)$.

• PANDORA extends a VolSDF backbone with neural decomposition modules:
  • SDFNet (8-layer MLP) models implicit surface via $d(x)=\mathrm{SDFNet}(x)$.
  • DiffuseNet, RoughNet, IllumNet, and MaskNet predict diffuse/specular radiance, roughness, illumination, and object masks.

The radiance field is replaced by a Stokes field, $S(o, d)$, to encode intensity and polarization state. Rendering integrates the radiance and polarization using an explicit physically-based polarization reflectance model (pBRDF/Mueller-matrix formalism).

2. Polarization Measurement and Modeling

Key to polarization-aware NeRF is the utilization of Stokes vector measurements, which encode polarization state:

• Stokes parameters are $S = [S_0, S_1, S_2]^T$, where $S_0$ represents total intensity; Degree of Polarization (DoP) and Angle of Polarization (AoP) are given by

$$\rho(i, j) = \frac{\sqrt{S_1^2 + S_2^2}}{S_0} \ \phi(i, j) = \frac{1}{2} \arctan\left(\frac{S_2}{S_1}\right)$$

• In GNeRP, AoP and DoP are computed from raw Stokes channels to derive image-plane constraints on surface normal azimuth:

$\psi(u) + \frac{\pi}{2} \equiv \phi(u) \pmod \pi$

• In PANDORA, the outgoing light's Stokes vector for each sampled 3D point is modeled as the sum of diffuse and specular contributions:

$$S_o(x, \omega_o) = L_d \begin{bmatrix} 1 \ \beta_d(\theta_n)\cos(2\phi_n) \ -\beta_d(\theta_n)\sin(2\phi_n) \end{bmatrix} + L_s \begin{bmatrix} 1 \ \beta_s(\theta_n)\cos(2\phi_n) \ -\beta_s(\theta_n)\sin(2\phi_n) \end{bmatrix}$$

• Polarization cues strongly depend on surface normals and are distinct between diffuse and specular reflection—enabling disambiguation of radiance contributions.

3. Normal Encoding and Gaussian Field Priors

GNeRP introduces a Gaussian representation over local normal distributions to robustly encode normal uncertainty and variation, constrained by polarization priors:

• At each sample $x_i$, the 3D normal distribution is

$$\mathcal{G}(n \mid x_i) = \mathcal{N}(n(x_i), \Sigma(x_i))$$

with unbiased covariance estimated from $M$ neighbors:

$$\hat \Sigma(x_i) = \frac{1}{M-1} \sum_{j=1}^M (n(x_i^j) - n(x_i))(n(x_i^j) - n(x_i))^T$$

• Projection to 2D image space is performed using the world-to-camera transform and Jacobian, producing a local Gaussian prior for image normal orientation.
• Per-pixel Gaussian fields along rays are composited:

$$\hat{\mathcal{G}}(u) = \mathcal{N}\left(\sum_i T_i\alpha_i\bar{n}_p^{(i)}, \sum_i T_i\alpha_i\Sigma_p^{(i)}\right)$$

This representation enables the system to be supervised directly by polarization-derived angular priors, providing detailed guidance in specular-dominated regions without explicit BRDF fitting.

4. Optimization Objectives and Reweighting

Learning in polarization-aware NeRF relies on multi-objective losses that couple radiance rendering with polarization-derived normal supervision:

• GNeRP uses:
  • $\mathcal{L}_{\mathrm{color}}$ (radiance rendering; $\ell_2$ loss)
  • $\mathcal{L}_{\mathrm{mean}}$ (azimuth angle; $\ell_1$ between predicted/image AoP)
  • $\mathcal{L}_{\mathrm{cov}}$ (covariance; anisotropy/eigenvector alignment)
  • $\mathcal{L}_{\mathrm{eik}}$ (eikonal regularizer for SDF)
  • $\mathcal{L}_{\mathrm{mask}}$ (optional, for foreground)

The total loss for pixel $u$ is reweighted:

$$\mathcal{L}(u) = w_{\mathrm{rad}}(u)\mathcal{L}_{\mathrm{color}} + w_{\mathrm{pol}}(u)(\mathcal{L}_{\mathrm{mean}} + \mathcal{L}_{\mathrm{cov}}) + \gamma\mathcal{L}_{\mathrm{eik}} + \delta\mathcal{L}_{\mathrm{mask}}$$

where $w_{\mathrm{rad}}(u) = 1 - \rho(u)$, $w_{\mathrm{pol}}(u) = \rho(u)$ with $\rho(u)$ the measured DoP.

• PANDORA uses a Stokes vector rendering loss, eikonal loss, and mask loss:

$$L_{\mathrm{total}} = L_{\mathrm{stokes}} + 0.1 L_{\mathrm{SDF}} + L_{\mathrm{mask}}$$

Only polarization-dominant regions (high DoP) strongly affect the polarization loss term, making the optimization robust to noisy or ambiguous polarization measurements.

5. Implementation and Experimental Results

Both methods are implemented as deep MLPs with positional encoding, using standard Adam optimization, and are validated across synthetic and real data.

• Network configurations: GNeRP and PANDORA employ 8-layer networks for SDF/geometry; auxiliary radiance/specularity/illumination networks are 4–5 layers deep. Meshes are extracted by Marching Cubes.
• Data: GNeRP introduces the PolRef dataset (four highly reflective objects, mesh error ±0.1mm) and evaluates on PANDORA's dataset.
• Quantitative performance: On PolRef (Chamfer Distance $10^{-3}$), GNeRP achieves 1.35, outperforming prior methods (e.g., NeuS 2.58, Ref-NeuS 2.34, PANDORA 5.90).
• Qualitative observations: GNeRP yields visibly crisper edges and high-frequency geometric detail in specular regions. PANDORA provides radiance decompositions that avoid specular bleed and model illumination with improved accuracy.

Method	Chamfer Dist. ($10^{-3}$, PolRef)	Normal MAE ($^\circ$, PANDORA)
Unisurf	10.89	—
VolSDF	4.26	—
NeuS	2.58	3.9 (Bust scene)
Ref-NeuS	2.34	—
NeRO	13.32	—
PANDORA	5.90	3.9 (Bust), 1.4 (Sphere)
GNeRP	1.35	—

These results demonstrate that incorporating polarization-aware losses and priors enables significant improvements in surface recovery and radiance separation, particularly in the presence of challenging reflectance.

6. Relation to Underlying Physics and Broader Significance

Polarization encodes view-dependent information about surface orientation and material composition that is largely inaccessible to conventional RGB imaging. The unique signatures in Degree of Polarization and Angle of Polarization, especially for specular surfaces, allow neural fields to localize normal directions behind highlight regions, disambiguate specular from diffuse radiance, and recover high-frequency shape details without manually modeling the full BRDF.

A plausible implication is that polarization-aware NeRFs obviate the need for explicit BRDF disentanglement or restrictive lighting control in multi-view stereo recovery of non-Lambertian objects. This suggests applications in industrial inspection, robotics, and relighting, whenever glossy, metallic, or otherwise reflective objects are involved.

7. Datasets, Practical Considerations, and Future Directions

• Datasets: Both frameworks rely on multi-view polarization image sets, captured either by commodity on-chip polarization sensors (e.g., Sony IMX250MYR) or rendered in simulation. GNeRP introduces the PolRef dataset with highly accurate 3D ground-truth on reflective objects.
• Training stability: Loss reweighting by DoP is necessary to handle variable signal-to-noise ratios and to avoid overfitting to ambiguous polarization measurements in diffuse regions.
• Performance factors: Covariance-based normal encoding enables local high-frequency shape recovery in GNeRP, while explicit Stokes rendering and pBRDF modeling are key for PANDORA's radiance decomposition.

Further advances may target joint global lighting estimation, improved robustness to noise, real-time inference, or extension of polarization modeling to transmission or scattering media. Recent results indicate that direct polarization supervision is a critical and robust signal for extending neural radiance fields to non-Lambertian and complex material scenarios (Yang et al., 18 Mar 2024, Dave et al., 2022).