Shape-from-Polarization Networks

• Shape-from-Polarization (SfP) networks are physics-guided deep learning methods that estimate per-pixel 3D geometry from polarization-encoded images.
• They integrate raw polarization images, Stokes-derived features, and physics-based priors into encoder–decoder architectures to resolve imaging ambiguities.
• Extensions to underwater, event-based, LWIR, and lidar-polarization fusion enhance performance under complex material properties and challenging illumination.

Shape-from-Polarization (SfP) Networks are a class of data-driven and hybrid computational imaging methods that estimate three-dimensional shape (usually in the form of per-pixel surface normals, and in advanced cases, depth or even spatially-varying reflectance parameters) from polarization-encoded images captured by single- or multi-view camera systems. These networks exploit the physics of light polarization on reflection and transmission, integrating them with deep neural architectures to resolve ambiguities associated with conventional polarization imaging, and to extend applicability to challenging scenes, diverse materials (diffuse, specular, mixed, transluscent, or even transparent), complex illumination, and non-ideal acquisition modalities (e.g., underwater, event-based, long-wave infrared, lidar fusion).

1. Polarization Physics and Shape Inference

SfP networks are grounded in polarimetric image formation, where the intensity observed at each pixel under a linear polarizer at angle $\theta$ is modeled as

$$I(\theta) = I_\text{unpol}\,[1 + \rho\,\cos(2\phi - 2\theta)]$$

with $I_\text{unpol}$ the unpolarized intensity, $\rho$ the degree of polarization (DoP), and $\phi$ the angle of polarization (AoP). For a given acquisition (typically four angles $\{0^\circ,45^\circ,90^\circ,135^\circ\}$), per-pixel Stokes parameters $S_0$, $S_1$, $S_2$ are computed, yielding

$$\rho = \sqrt{S_1^2 + S_2^2} / S_0, \qquad \phi = \frac{1}{2}\arctan2(S_2, S_1)$$

The DoP relates to the surface normal’s zenith angle $$I(\theta) = I_\text{unpol}\,[1 + \rho\,\cos(2\phi - 2\theta)]$$0, dependent on Fresnel reflection/transmission models and the refractive index $$I(\theta) = I_\text{unpol}\,[1 + \rho\,\cos(2\phi - 2\theta)]$$1. For diffuse and specular reflection, Fresnel theory predicts distinct nonlinear relationships,

$$I(\theta) = I_\text{unpol}\,[1 + \rho\,\cos(2\phi - 2\theta)]$$2

with known parametric forms (see (Lei et al., 2021, Mortazavi et al., 2024, Ba et al., 2019)).

The AoP constrains the azimuth of the normal up to a $$I(\theta) = I_\text{unpol}\,[1 + \rho\,\cos(2\phi - 2\theta)]$$3 (diffuse) or $$I(\theta) = I_\text{unpol}\,[1 + \rho\,\cos(2\phi - 2\theta)]$$4 (specular) ambiguity. Subsurface scattering, depolarization by transmission, and measurement noise introduce additional challenges for direct shape recovery.

2. Network Architectures and Input Representations

SfP networks ingest polarization cues through various encoding strategies:

• Raw polarization images: The canonical setup provides four single-shot polarization channels (Ba et al., 2019, Mortazavi et al., 2024).
• Stokes-derived representations: Some architectures explicitly featurize degree and angle of polarization, e.g., $$I(\theta) = I_\text{unpol}\,[1 + \rho\,\cos(2\phi - 2\theta)]$$5 (Mortazavi et al., 2024, Lei et al., 2021).
• Physics-based priors: Hybrid methods concatenate multiple per-pixel normal hypotheses—derived from diffuse and specular Fresnel inverses—as “ambiguous normal maps” (Ba et al., 2019, Zou et al., 2020).
• View encoding: Essential in non-orthographic or scene-level SfP to resolve spatially-varying viewing directions (Lei et al., 2021).

Most networks pursue encoder–decoder (U-Net or ResNet-UNet) backbones, often augmented with:

• Self-attention bottlenecks: Multi-head transformer or attention modules are key to fusing local and global context and handling ambiguities (Lei et al., 2021, Scheuble et al., 2024).
• Multi-branch decoders: Partitioning geometry, reflectance, and additional tasks (e.g., subsurface scattering, SVBRDF) promotes specialized feature learning (Deschaintre et al., 2021, Li et al., 2024).
• Physics-guided normalization: Spatially-adaptive feature modulation (e.g., SPADE) injects polarization features at multiple decoder stages (Ba et al., 2019).

Event-based SfP leverages cumulative voxel grid representations (CVGR) for asynchronous event streams, enabling U-Net or SNN-based architectures to operate on high-speed data (Muglikar et al., 2023, Kang et al., 2023).

3. Loss Functions, Training Protocols, and Supervision

Most SfP networks optimize a cosine similarity (angular) loss between predicted and ground-truth normals: $$I(\theta) = I_\text{unpol}\,[1 + \rho\,\cos(2\phi - 2\theta)]$$6 Variants include per-pixel L1, per-bin histogram, SSIM, TV, and LPIPS losses, particularly for joint tasks such as denoising, descattering, or photometric refinement (Wang et al., 1 Mar 2026). For photometric consistency or inverse rendering, differentiable rendering losses supervise self-synthesized polarization images according to the physical forward model, closing the loop between output shape and input measurements (Tiwari et al., 2024, Deschaintre et al., 2021, Li et al., 2024).

Supervision regimens span:

• Fully supervised: Jointly with ground-truth normal/depth/reflectance maps, typically from synthetic rendered datasets or 3D scanner-registered real images (Ba et al., 2019, Mortazavi et al., 2024, Deschaintre et al., 2021, Wang et al., 1 Mar 2026).
• Self-supervised/inverse rendering: Learned shape and auxiliary parameters only need to re-synthesize the input polarization images via the physical model (Tiwari et al., 2024).

Training commonly employs the Adam optimizer, learning rates $$I(\theta) = I_\text{unpol}\,[1 + \rho\,\cos(2\phi - 2\theta)]$$7–$$I(\theta) = I_\text{unpol}\,[1 + \rho\,\cos(2\phi - 2\theta)]$$8, 100–1000 epochs, patch-based training, and domain-specific augmentations (random crops, rotations, simulated noise).

4. Extensions: Underwater, Event-based, LWIR, and Lidar-Polarization Fusion

SfP networks have been adapted for several challenging optical regimes:

• Underwater imaging: Joint polarization-based descattering and SfP are addressed in UD-SfPNet, integrating detail-enhanced convolutions (DEConv) and pyramid color embeddings to yield robust normal estimation in scattering media, achieving MAE 15.12° on synthetic underwater datasets (Wang et al., 1 Mar 2026).
• Event-based SfP: Approaches leveraging spatiotemporal voxel representations and spiking neural networks (SNNs) enable low-latency, energy-efficient normal estimation from high-speed streams, matching ANN performance at $$I(\theta) = I_\text{unpol}\,[1 + \rho\,\cos(2\phi - 2\theta)]$$9 lower energy costs (Muglikar et al., 2023, Kang et al., 2023).
• LWIR polarization: Learning-based SfP in the long-wave infrared combines emission and reflection via a Stokes–Mueller formalism. Transformer-augmented U-Nets trained on synthetic ThermoPol data deliver MAE ≈ 10° and robust generalization to materials opaque in the visible spectrum (Kitazawa et al., 23 Jun 2025).
• Lidar-polarization fusion: Polarization-resolved lidar enables simultaneous normal and depth estimation in long-range, outdoor contexts. Transformer U-Nets ingest time-resolved wavefronts and recovered Mueller matrices, improving both normal (8.71° MAE) and depth (19 cm MAE) estimation over pure ToF or vision-based SfP (Scheuble et al., 2024).

5. Advanced Tasks: Mixed Materials, Subsurface Scattering, Transparent Objects, and Human Shape

SfP networks have been extended to address material and geometric complexities:

• Mixed reflection: SS-SfP achieves per-scene self-supervised SfP in the presence of coexisting diffuse and specular components by physically separating reflectance cues, using an inverse-rendered self-supervision loop, and optimizing refractive index as a post-processing step (Tiwari et al., 2024).
• Subsurface scattering and translucency: Polarization cues provide unique constraints for single-shot estimation of surface normals and subsurface scattering (BSSRDF) parameters, outperforming prior SfP baselines (Li et al., 2024).
• Transparent and LWIR-opaque objects: Networks trained under tailored Stokes–Mueller and emission/reflection models provide robust normals for materials challenging for classical visible SfP (Kitazawa et al., 23 Jun 2025).
• 3D human shape reconstruction: By structuring a two-stage pipeline separating normal extraction (via polarization+physics priors) and human shape estimation (SMPL regression + normal-guided mesh refinement), SfP networks yield improved pose and clothed-surface estimation over color- or depth-only baselines (Zou et al., 2020).

6. Benchmarks, Evaluation, and Limitations

Empirical evaluation utilizes both synthetic and real datasets:

• Object-level datasets: DeepSfP (indoor/outdoor), MuS-Polar3D (underwater), ThermoPol (LWIR), ESfP-Real (event-based), and SPW (scene-level).
• Reported MAE: Modern networks achieve MAE in the 10–20° range on matched hardware/calibration settings, significantly surpassing physics-based SfP methods (41–49°) (Mortazavi et al., 2024, Ba et al., 2019).
• Qualitative evaluation: SfP networks preserve high-frequency details and shape continuity under scattering, non-trivial reflectance, or complex scene composition.

Limitations include:

• Extreme materials (mirrors, highly absorptive or scattering), ambiguous refractive index, or complex illumination breaking the polarization model.
• Domain transferability: networks often require retraining for different cameras, wavelength bands, or environmental conditions.
• Self-supervised optimization methods add inference latency.
• Single-view approaches remain limited by local/occlusion ambiguities and phase inversion in the polarization signal.

7. Future Directions and Open Problems

Ongoing research avenues include:

• Integrating multi-view, multi-spectral, and depth cues to resolve normal integration ambiguities and handle severe occlusions or scattering (Wang et al., 1 Mar 2026).
• Meta-learning or domain adaptation to transfer pretrained SfP networks across camera types or spectral bands.
• End-to-end fusion of SfP with ToF, event streams, and classical photometric/semantic cues for holistic 3D scene reconstruction (Scheuble et al., 2024).
• Extending physical models to encompass birefringence, multi-bounce, metals, and layered materials.
• Hardware–algorithm co-design for real-time, robust SfP sensors deployable in robotics, underwater, remote sensing, and biomedical imaging.

A plausible implication is that future SfP network designs will further collapse the traditional separation between physics-based priors and deep data-driven learning, exploiting rich polarization cues for robust, high-fidelity shape inference across an expanding landscape of imaging modalities and environmental contexts.