Dehaze-then-Splat: 3D Smoke-Free Pipeline

• Dehaze-then-Splat is a two-stage pipeline that first dehazes individual frames using generative models and then reconstructs smoke-free 3D scenes with Gaussian Splatting.
• It implements auxiliary losses including depth supervision, dark channel prior, and dual-source gradient matching to enforce multi-view consistency and enhance metrics like PSNR and SSIM.
• A MCMC-based densification process with early stopping dynamically optimizes Gaussian capacity, balancing reconstruction fidelity with computational efficiency.

Dehaze-then-Splat is a two-stage pipeline for multi-view smoke removal and novel view synthesis, developed for the NTIRE 2026 3D Restoration and Reconstruction Challenge. It combines per-frame generative dehazing with physics-informed 3D Gaussian Splatting (3DGS), introducing auxiliary losses and MCMC-based densification to address the critical challenge of reconciling high per-image fidelity with global multi-view consistency. The pipeline is designed to remove smoke, maximize 2D view restoration accuracy, and produce high-fidelity, smoke-free 3D reconstructions suitable for novel view synthesis (Chen et al., 15 Apr 2026).

1. Two-Stage Pipeline Overview

Dehaze-then-Splat follows a “dehaze-then-reconstruct” philosophy. In the first stage, each smoky input frame $I_\text{smoke}$ is processed independently by Nano Banana Pro, an image-generative model interfaced via Google’s Gemini API (“gemini-3-pro-image-preview”), with a structured prompt targeting comprehensive smoke removal while preserving geometry and photorealism. This achieves high single-frame PSNR (20.07 dB post-normalization on Akikaze), but introduces stochastic, view-specific variations.

Following generative dehazing, per-channel affine brightness normalization is applied to address global photometric inconsistency, effectively eliminating brightness differences of up to 0.12 (in normalized units) between adjacent frames. For each channel $c \in \{\mathrm{R},\mathrm{G},\mathrm{B}\}$:

$$I_\text{norm}(c) = (I_\text{dehazed}(c) - \mu_c) \cdot \left(\frac{\sigma_c'}{\sigma_c}\right) + \mu_c'$$

where $(\mu_c', \sigma_c')$ derive from ground-truth or median statistics.

The second stage constructs a 3D scene via 3D Gaussian Splatting with the gsplat library. Gaussians are optimized such that rendered novel views match the dehazed-normalized frames, initially minimizing a hybrid photometric loss:

$$L_\text{photo} = (1-\lambda_s) \cdot L_1(I_\text{render}, I_\text{norm}) + \lambda_s \cdot L_\text{SSIM}(I_\text{render}, I_\text{norm}),\quad \lambda_s=0.2$$

However, independent frame restoration leads to multi-view inconsistencies, resulting in blur, floaters, and structural instability in traditional 3DGS when auxiliary losses are not included.

2. Physics-Informed Priors and Multi-View Consistency

To overcome the single-view/multi-view tension, Dehaze-then-Splat adds three auxiliary losses, each designed to enforce implicit cross-view constraints:

1. Depth Supervision Loss: Pseudo inverse-depth maps $d_p$ are generated for each input view using Depth Anything V2 on the original smoky frames. The rendered depth $d_r$ is anchored to $d_p$ with a weighted, scale-invariant Pearson correlation:

$$L_\text{depth} = 1 - \text{Corr}_w(d_r, d_p) \ \text{with} \quad \text{Corr}_w(d_r, d_p) = \frac{\sum_i w_i (d_r^i-\bar{d}_r)(d_p^i-\bar{d}_p)}{\sqrt{\sum_i w_i (d_r^i-\bar{d}_r)^2} \cdot \sqrt{\sum_i w_i (d_p^i-\bar{d}_p)^2}},\quad w_i = \alpha_i$$

Depth supervision ($\lambda_p=0.1$) is the most impactful prior; its ablation reduces PSNR by 0.79 dB.

2. Dark Channel Prior (DCP) Regularization: Nonzero dark channel intensities in rendered images $c \in \{\mathrm{R},\mathrm{G},\mathrm{B}\}$0 are penalized, promoting patchwise haze removal as per classic DCP:

$c \in \{\mathrm{R},\mathrm{G},\mathrm{B}\}$1

Regularization weight is $c \in \{\mathrm{R},\mathrm{G},\mathrm{B}\}$2.

3. Dual-source Gradient Matching: High-frequency structure is enforced using an auxiliary dehazed reference $c \in \{\mathrm{R},\mathrm{G},\mathrm{B}\}$3 (from MB-TaylorFormer), brightness-aligned to the Nano Banana output. The $c \in \{\mathrm{R},\mathrm{G},\mathrm{B}\}$4 norm between spatial gradients is minimized:

$c \in \{\mathrm{R},\mathrm{G},\mathrm{B}\}$5

with $c \in \{\mathrm{R},\mathrm{G},\mathrm{B}\}$6.

The overall loss is thus:

$c \in \{\mathrm{R},\mathrm{G},\mathrm{B}\}$7

3. MCMC-Based Densification and Early Stopping

3DGS capacity is regulated via a Monte Carlo Markov Chain (MCMC) densification process, replacing gsplat’s DefaultStrategy. At each step in the designated densification window ($c \in \{\mathrm{R},\mathrm{G},\mathrm{B}\}$8), each Gaussian may be stochastically perturbed, split, or relocated using noise-injection proposals (NOISE_LR$c \in \{\mathrm{R},\mathrm{G},\mathrm{B}\}$9 annealed to zero by step 8000). A hard cap ($$I_\text{norm}(c) = (I_\text{dehazed}(c) - \mu_c) \cdot \left(\frac{\sigma_c'}{\sigma_c}\right) + \mu_c'$$0) limits cloud size.

Early stopping is enforced: after $$I_\text{norm}(c) = (I_\text{dehazed}(c) - \mu_c) \cdot \left(\frac{\sigma_c'}{\sigma_c}\right) + \mu_c'$$1, the probability of further Gaussian splitting $$I_\text{norm}(c) = (I_\text{dehazed}(c) - \mu_c) \cdot \left(\frac{\sigma_c'}{\sigma_c}\right) + \mu_c'$$2. Empirically, this yields stable clouds of approximately $$I_\text{norm}(c) = (I_\text{dehazed}(c) - \mu_c) \cdot \left(\frac{\sigma_c'}{\sigma_c}\right) + \mu_c'$$3–$$I_\text{norm}(c) = (I_\text{dehazed}(c) - \mu_c) \cdot \left(\frac{\sigma_c'}{\sigma_c}\right) + \mu_c'$$4 Gaussians, avoiding capacity “explosion” and overfitting to per-view artifacts.

Throughout densification, all auxiliary losses remain active, ensuring newly introduced Gaussians are regularized for geometric and photometric consistency.

4. Training Regimen and Implementation Details

Training spans 20,000 steps with validation every 1,000 steps. Key hyperparameters include spherical-harmonic degree (0→3 progressive), learning rates ($$I_\text{norm}(c) = (I_\text{dehazed}(c) - \mu_c) \cdot \left(\frac{\sigma_c'}{\sigma_c}\right) + \mu_c'$$5, $$I_\text{norm}(c) = (I_\text{dehazed}(c) - \mu_c) \cdot \left(\frac{\sigma_c'}{\sigma_c}\right) + \mu_c'$$6), scene scale 2.0, and gamma $$I_\text{norm}(c) = (I_\text{dehazed}(c) - \mu_c) \cdot \left(\frac{\sigma_c'}{\sigma_c}\right) + \mu_c'$$7 (gsplat default $$I_\text{norm}(c) = (I_\text{dehazed}(c) - \mu_c) \cdot \left(\frac{\sigma_c'}{\sigma_c}\right) + \mu_c'$$8 reduces PSNR by $$I_\text{norm}(c) = (I_\text{dehazed}(c) - \mu_c) \cdot \left(\frac{\sigma_c'}{\sigma_c}\right) + \mu_c'$$95 dB). Auxiliary loss weights are fixed ($(\mu_c', \sigma_c')$0).

Scene initialization uses $(\mu_c', \sigma_c')$1 (or $(\mu_c', \sigma_c')$2 for dark scenes) Gaussians; backgrounds are white (or black for Hinoki).

The training loop pseudocode: $(\mu_c', \sigma_c')$4

5. Quantitative Results and Ablation

On the Akikaze dataset, the baseline (no auxiliary losses, STOP=20k) achieves PSNR of 19.48 dB and SSIM of 0.632. Incorporating all auxiliary losses and early stopping (STOP=2k) improves performance to 20.98 dB PSNR (+1.50 dB) and 0.683 SSIM (+0.051).

Setting	PSNR (dB)	SSIM	∆PSNR
Baseline (no aux, STOP=20k)	19.48	0.632	—
Full pipeline (all losses, STOP=2k)	20.98	0.683	+1.50

Ablations show:

• Depth supervision: +0.79 dB
• MCMC vs. DefaultStrategy: +0.47 dB at matched STOP
• Early stopping (STOP=3k vs. 15k): +0.35 dB
• Dual-source gradient: +0.06 dB

Qualitatively, Dehaze-then-Splat yields reconstructed views with sharp texture, stable geometry, and minimal haze, while baseline models manifest blur, color drift, and floater artifacts.

6. Failure Modes, Scope, and Extensions

Absent auxiliary priors or early stopping, the 3DGS model tends to catastrophically overfit, producing large numbers of Gaussians (>$(\mu_c', \sigma_c')$3) and severe floater artifacts. End-to-end Koschmieder-based models fail to converge (β→0), resulting in poor PSNR (10.28 dB).

Computationally, the combination of MCMC densification and 20k steps of 3DGS optimization incurs significantly higher resource requirements compared to classical NeRFs, primarily due to larger Gaussian budgets.

Extensions include integrating cross-view consistency directly into the dehazing stage (e.g., video-based generative modeling, multi-view attention), adaptive Gaussian capacity scheduling, and adaptation for other atmospherics (fog, particulate smoke) through modified priors.

7. Broader Impact and Significance

Dehaze-then-Splat demonstrates that explicit separation of per-frame dehazing from multi-view, prior-driven 3D reconstruction can balance state-of-the-art 2D image fidelity with global, physically plausible geometry. The pipeline establishes that per-image generative restoration must be complemented by regularization—both in model capacity and in geometry/appearance consistency—at the 3D stage to avoid degraded rendering outcomes. This architecture achieves substantive quantitative and qualitative gains for smoke-free novel view synthesis and provides a flexible blueprint for addressing a range of atmospheric degradations using physics-informed priors in learning-based 3D reconstruction frameworks (Chen et al., 15 Apr 2026).