Differentiable Adaptive 4D Structured Illumination for Joint Capture of Shape and Reflectance

Abstract: We present a differentiable framework to adaptively compute 4D illumination conditions with respect to an object, for efficient, high-quality simultaneous acquisition of its shape and reflectance, with a unified spatial-angular structured light and a single camera. Using a simple histogram-based pixel-level probability model for depth and reflectance, we differentiably link the next illumination condition(s) with a loss that encourages the reduction in depth uncertainty. As new structured illumination is cast, corresponding image measurements are used to update the uncertainty at each pixel. Finally, a fine-tuning-based approach reconstructs the depth map and reflectance parameter maps, by minimizing the differences between all physical measurements and their simulated counterparts. The effectiveness of our framework is demonstrated on physical objects with wide variations in shape and appearance. Our depth results compare favorably with state-of-the-art techniques, while our reflectance results are comparable when validated against photographs.

• The paper introduces a novel differentiable adaptive framework that optimizes illumination conditions to jointly capture shape and reflectance while reducing depth uncertainty.
• It employs histogram-based probability models and Monte Carlo sampling to update pixel-level estimates of depth and GGX BRDF parameters efficiently.
• The system achieves significant efficiency gains, demonstrating up to 100× reduced exposure and 2× faster acquisition with high quantitative fidelity in 3D reconstruction.

Differentiable Adaptive 4D Structured Illumination for Joint Capture of Shape and Reflectance

Introduction

The paper "Differentiable Adaptive 4D Structured Illumination for Joint Capture of Shape and Reflectance" (2605.06214) introduces a novel acquisition framework that enables simultaneous, efficient, and high-quality capture of object geometry and reflectance from a single viewpoint. Leveraging a spatial-angular illumination setup with a single camera, the authors present a differentiable system that adaptively optimizes lighting conditions tailored to the object, thus reducing depth uncertainty and enhancing acquisition throughput. The system is based on a pixel-level histogram probability model for both depth and BRDF parameters, and the adaptive optimization is governed by minimizing a loss related to depth uncertainty.

Acquisition Setup and Hardware

A key element of the framework is its custom spatial-angular illumination setup: a rectangular RGB LED array (3072 LEDs) and a high-resolution LCD mask (1920×1080), with a Canon EOS R5 camera above the mask. The system is fully calibrated for intrinsic and extrinsic parameters in an end-to-end differentiable manner, ensuring accurate mapping between illumination, object geometry, and camera measurements.

Figure 1: Acquisition setup composed of camera, LED array, and LCD mask for spatial-angular structured lighting.

Pipeline Overview

The pipeline consists of two main stages: differentiable adaptive acquisition and fine-tuning. During adaptive acquisition, the next illumination pattern is computed by minimizing depth uncertainty via histogram-based probability models. Photographs under these optimized patterns are used to update the probability distributions. The cycle repeats until a termination criterion is met. Subsequently, a fine-tuning step further minimizes the discrepancies between all physical measurements and simulated counterparts, yielding a depth map and texture maps encoding GGX BRDF parameters.

Figure 2: Two-stage pipeline—adaptive acquisition with probabilistic updates, followed by fine-tuning for shape and reflectance.

Probability Modeling and Differentiable Optimization

For each pixel, joint depth and BRDF distributions are represented independently using histogram-based scores. Depth range is determined per pixel and discretized into bins, each scored using Zero-Normalized Cross-Correlation (ZNCC) between actual and simulated sensor readings. For reflectance, parameter ranges are set using OpenSVBRDF statistics, and bins are scored with inverse $\mathcal{L}_1$ distances. Monte Carlo sampling—600 candidates per pixel—is employed to update the distributions, and new patterns are optimized to maximize cross entropy between probable depth candidates, thus actively driving the illumination towards uncertainty reduction.

Figure 3: Histogram-based probability model for pixel-wise depth estimation.

Figure 4: Visualization of adaptive acquisition—evolution of light/mask patterns, photographs, depth uncertainty, and per-pixel probability distributions.

Fine-Tuning with Neural Latent Representation

Post-acquisition, depth and reflectance parameters are initialized from the highest probability bins, and a simultaneous optimization step is performed to reconcile all captured data with simulated measurements. A 16D neural latent vector parameterizes BRDFs via multiple MLPs, facilitating efficient differentiable optimization of GGX model parameters.

Results: Efficient Joint Capture and Quantitative Analysis

Extensive experiments were conducted on 10 physical objects of varying shapes and material classes. The framework demonstrated remarkable efficiency—adaptive acquisition required only 10 minutes (with a twofold reduction in acquisition time and up to 100× reduction in exposure time compared to prior methods), and fine-tuning was tractable even for high-resolution outputs. Shape quality was benchmarked by RMSE and percentage of inliers ($<$3 mm error), using commercial 3D scans as ground truth. Reflectance accuracy was validated by comparing relighting results against novel photographs, using SSIM, LPIPS, and PSNR metrics.

Figure 5: Comparative results—depth and reflectance reconstructions versus state-of-the-art techniques, with quantitative metrics.

The adaptive use of the LED array allowed significant reduction in shadow regions relative to single-source approaches, enabling more complete geometry acquisition.

Figure 6: Complete depth reconstruction using adaptive multi-source illumination compared to single-source methods.

Reflectance estimations, encoded as GGX parameter maps, showed close correspondence to reference photographs even for challenging material types.

Figure 7: GGX BRDF parameter maps for several objects—diffuse/specular albedo, normals, and roughness.

Ablation Studies

Ablation experiments demonstrated the impact of pattern count, sample count, batch size of patterns, peak candidate selection, histogram bin granularity, and image resolution on depth quality. Increasing pattern and sample counts improved RMSE, at the cost of computation. Optimizing batch size and resolution achieved an empirical balance between performance and numerical accuracy.

Figure 8: Depth accuracy as a function of adaptive pattern count.

Figure 9: Monte Carlo sample count effect on geometric estimation.

Figure 10: Batch size of optimized patterns—balancing acquisition throughput and accuracy.

Figure 11: Candidates used for loss—trade-off between solution coverage and speed.

Figure 12: Bin granularity—finer discretizations improve depth estimation.

Figure 13: Resolution used for pattern computation—accuracy/performance trade-off.

Implications and Future Directions

The approach establishes a differentiable methodology for active, object-adaptive, multiplexed spatial-angular structured light, with practical implications for cultural heritage digitization, e-commerce, visual effects, and game asset acquisition. The numerical results support the claim that adaptive pattern optimization dramatically improves both efficiency and completeness of capture. The method currently assumes direct illumination in pattern optimization and is limited to parametric BRDFs and depth maps; future integration with advanced representations (e.g., Gaussian splatting [bi2024rgs], neural radiance fields [Mildenhall2021nerf]) or free-form scanning [Ma_2021_FreeScan] is anticipated. Further research is needed to address indirect lighting and generalize to full 3D objects and complex scenes.

Conclusion

This paper contributes a differentiable adaptive framework for efficient joint capture of shape and reflectance using 4D spatial-angular structured illumination. By formalizing acquisition as a closed-loop, loss-driven optimization tailored to each object, and coupling this with efficient sampling and neural latent representations, the system achieves strong quantitative metrics on geometric and appearance recovery—substantially outperforming prior art in efficiency and completeness. The methodology provides a promising foundation for further extensions in high-fidelity object digitization and adaptive computational imaging.