Physically-Plausible ISP (PPISP) Frameworks

• PPISP is a framework that enforces physical consistency in image signal processing by integrating explicit camera parameters for spectral and radiance field reconstructions.
• It employs detailed physical forward models and differentiable modules to disentangle photometric effects such as exposure, vignetting, and color correction.
• PPISP demonstrates improved color fidelity and generalization, achieving near-zero ΔE in spectral tasks and higher PSNR in radiance field reconstructions.

The Physically-Plausible ISP (PPISP) framework designates a class of camera data processing models and learning methods that enforce physical consistency between observed photometric measurements, latent high-dimensional representations (such as spectral data or scene radiance), and the results of the image signal processing (ISP) pipeline. Unlike conventional black-box or purely data-driven approaches, PPISP architectures are constructed to guarantee that the recovered latent signals are consistent with observed RGBs under measured or inferred camera characteristics—exposure, color sensitivity, vignetting, and nonlinear response—while also providing explicit control and interpretability. PPISP frameworks enable both exact color reproduction for spectral reconstruction tasks (Lin et al., 2020) and robust, disentangled photometric compensation for advanced multi-view 3D reconstruction tasks (Deutsch et al., 26 Jan 2026).

1. Foundation and Motivation

Physically-plausible ISP concepts emerged as a response to the limitations of purely data-driven and unconstrained approaches in spectral reconstruction and radiance field modeling. Neural spectral reconstruction networks frequently produced spectra which, when recomposed with the known camera spectral sensitivities, did not match the observed RGB measurements, violating physical plausibility. In radiance field reconstruction, performance degraded under real-world photometric variability (exposure, white balance, vignetting), and common fixes—per-frame latent variables or affine color transforms—lacked interpretability and often led to overfitting. PPISP frameworks incorporate physically accurate models of the image formation process to solve these challenges: enforcing color fidelity, enabling photometric robustness, and providing intuitive parameter control (Lin et al., 2020, Deutsch et al., 26 Jan 2026).

2. Forward Imaging Models and ISP Pipelines

All PPISP frameworks are built on detailed physical forward models of the imaging process. The foundational model for spectral reconstruction represents the RGB camera signal as

$r = e\,C^\top S$

where $S\in\mathbb{R}^n$ is the spectral radiance, $C\in\mathbb{R}^{n\times 3}$ are spectral sensitivities, $e$ is scalar exposure, and $r\in\mathbb{R}^3$ are linear RGBs (Lin et al., 2020). For radiance field reconstruction, PPISP pipelines compose radiometric effects as follows:

• Exposure offset: $I^{\rm exp}=L\times 2^{\Delta t}$
• Vignetting: multiplies with a chromatic polynomial radial falloff
• Color correction: chromaticity homography and normalization
• Camera Response Function (CRF): smooth S-shaped or gamma mapping with all modules explicitly parameterized and differentiable (Deutsch et al., 26 Jan 2026). Each effect is disentangled to align with real camera mechanisms.

3. Enforcing Physical Plausibility

PPISP frameworks guarantee that reconstructed signals reproduce observed RGBs when mapped through the camera’s known physical transform. In spectral reconstruction, every candidate spectrum at each patch is decomposed as

$\hat{S} = S_{\parallel} + N\,\hat{\alpha}$

where $S_{\parallel} = C(C^\top C)^{-1}(r_{\rm in}/e)$ gives the unique minimum-norm solution matching the input RGB, $N$ is an orthonormal basis for the null-space of $C^\top$, and $\hat{\alpha}$ are neural network-predicted coefficients (Lin et al., 2020). This ensures $C^\top\hat{S}=r_{\rm in}/e$ identically, i.e., physical color fidelity is enforced by construction. Losses are defined only on the unobservable null-space coefficients:

$L = \|\hat{\alpha} - \alpha_{\rm gt}\|^2_2$

in contrast to conventional color or spectral terms.

For radiance field tasks, similar physical constraints are enforced via modular, parameterized ISP layers, ensuring that photometrically-consistent images are synthesized regardless of camera or scene variations (Deutsch et al., 26 Jan 2026). All modules are optimized within the end-to-end learning pipeline.

4. Exposure and Photometric Robustness

Achieving invariance to exposure and photometric drift is critical for practical deployment. In PPISP spectral reconstruction, robustness is achieved by augmenting input RGB patches and corresponding ground-truth coefficients with random exposure scaling drawn log-uniformly over $[0.1,10]$, with all labels normalized accordingly. This design ensures the model can reconstruct plausible spectra over a broad range of exposures (Lin et al., 2020).

In radiance field frameworks, PPISP incorporates a controller network which, for each input or synthesized view, predicts per-frame exposure and white-balance parameters from the rendered radiance. This permits dynamic compensation for exposure, white balance, and vignetting for both observed and novel, unseen viewpoints (Deutsch et al., 26 Jan 2026). When metadata such as EXIF exposure values is available, it can be directly input to the controller for accelerated and more accurate parameter convergence.

5. Network Architectures and Learning Strategies

Spectral Reconstruction

• Architecture: The backbone is a modified HSCNN-R model (NTIRE2018 competition), with input patches of $50 \times 50$ linear RGBs and outputs in either full spectral space or as null-space coefficients.
• Core blocks: Each block has stacked $3\times 3$ convolutions with ReLU activation and residual connections; three parallel subnetworks are ensembled (Lin et al., 2020).
• Training regime: Data augmentation includes random horizontal/vertical flips and random exposure scaling; $\ell_2$ loss is computed only on null-space coefficients.

Radiance Field Reconstruction

• ISP pipeline: All photometric modules are differentiable and their parameters are trainable end-to-end.
• Controller network: Comprised of $1\times 1$ convolutional layers with ReLU, max-pooling, adaptive pooling, an MLP (with multiple nonlinear layers), and two output heads for exposure and chromatic offsets.
• Two-phase training: First jointly optimize radiance and ISP parameters on training views; then freeze camera/intrinsic ISP and train the controller using photometric loss (Deutsch et al., 26 Jan 2026).

6. Quantitative Performance and Ablation

Spectral reconstruction (Lin et al., 2020):

• Color error (ΔE, CIE1976) and spectral error (MRAE) are primary metrics.
• Under unchanged exposure, PPISP yields perfect color fidelity ($\Delta E_{\rm mean}=0$) and low MRAE ($\sim 1.7\times 10^{-2}$); under exposure changes ($\xi=0.5, 2$), conventional networks degrade ($\Delta E_{\rm mean}>1.9$), while PPISP with exposure augmentation maintains zero color error and stable spectral error across exposure range.

Radiance field reconstruction (Deutsch et al., 26 Jan 2026):

• PPISP achieves higher novel view PSNR (up to $+2$ dB) over BilaRF and ADOP across benchmarks including Mip-NeRF360, Tanks & Temples, Bilateral-RF, and Waymo.
• Full ablation confirms that all four modules (exposure, vignetting, color correction, CRF) contribute to performance, with exposure and vignetting being most critical.
• Controller network boosts novel view PSNR by an additional $\sim0.5$ dB.
• PPISP demonstrates lower train-view PSNR but higher generalization, indicating reduced overfitting compared to less constrained latent approaches.
• Runtime overhead is moderate: ISP pipeline adds $+3\%$ without the controller and $+26\%$ with controller, remaining faster than prior state-of-the-art Bilateral-RF pipelines.

Framework / Condition	Color Error $\Delta E_\mathrm{mean}$	Spectral Error MRAE	Notes
HSCNN-R	~0.49	$1.35 \times 10^{-2}$	Baseline (fixed exposure)
HSCNN-R$^p$ (PPISP)	0	$1.73 \times 10^{-2}$	Perfect color (no exposure invariance)
HSCNN-R$^d$(data-aug)	0.25–0.26	$2.77 \times 10^{-2}$	Stable w.r.t. exposure
HSCNN-R$^{pd}$(PPISP+aug)	0	$2.78$–$2.91 \times 10^{-2}$	Best overall

7. Limitations and Prospects

PPISP frameworks require linear-raw RGB with known, accurate camera response characteristics; any nonlinearity or in-camera processing (e.g., JPEG or proprietary ISP) invalidates strict physical consistency (Lin et al., 2020). The models assume Lambertian scenes and neglect higher-order phenomena (inter-reflections, fluorescence).

Prospective extensions include joint estimation of camera sensitivities and scene spectra, adaptation to unknown/variable ISPs using invertible modules, and integration with complex data (video, HDR, wider spectral range). For radiance field modeling, further expansion is plausible toward end-to-end learning of entire imaging pipelines, generalized video capture, and plug-and-play incorporation of metadata (Deutsch et al., 26 Jan 2026).

• "Physically Plausible Spectral Reconstruction from RGB Images" (Lin et al., 2020)
• "PPISP: Physically-Plausible Compensation and Control of Photometric Variations in Radiance Field Reconstruction" (Deutsch et al., 26 Jan 2026)