---
title: 'Dehaze-then-Splat: 3D Smoke-Free Pipeline'
url: https://www.emergentmind.com/topics/dehaze-then-splat-pipeline
type: topic
---

# Dehaze-then-Splat: 3D Smoke-Free Pipeline

Dehaze-then-Splat is a two-stage pipeline for multi-view smoke removal and novel view synthesis, developed for the NTIRE 2026 3D Restoration and Reconstruction Challenge. It combines per-frame generative dehazing with physics-informed 3D Gaussian Splatting (3DGS), introducing auxiliary losses and MCMC-based densification to address the critical challenge of reconciling high per-image fidelity with global multi-view consistency. The pipeline is designed to remove smoke, maximize 2D view restoration accuracy, and produce high-fidelity, smoke-free 3D reconstructions suitable for novel view synthesis [2604.13589].

## 1. Two-Stage Pipeline Overview

Dehaze-then-Splat follows a “dehaze-then-reconstruct” philosophy. In the first stage, each smoky input frame $I_\text{smoke}$ is processed independently by Nano Banana Pro, an image-generative model interfaced via Google’s Gemini API (“gemini-3-pro-image-preview”), with a structured prompt targeting comprehensive smoke removal while preserving geometry and photorealism. This achieves high single-frame PSNR (20.07 dB post-normalization on Akikaze), but introduces stochastic, view-specific variations.

Following generative dehazing, per-channel affine brightness normalization is applied to address global photometric inconsistency, effectively eliminating brightness differences of up to 0.12 (in normalized units) between adjacent frames. For each channel $c \in \{\mathrm{R},\mathrm{G},\mathrm{B}\}$:
$$
I_\text{norm}(c) = (I_\text{dehazed}(c) - \mu_c) \cdot \left(\frac{\sigma_c'}{\sigma_c}\right) + \mu_c'
$$
where $(\mu_c', \sigma_c')$ derive from ground-truth or median statistics.

The second stage constructs a 3D scene via 3D Gaussian Splatting with the gsplat library. Gaussians are optimized such that rendered novel views match the dehazed-normalized frames, initially minimizing a hybrid photometric loss:
$$
L_\text{photo} = (1-\lambda_s) \cdot L_1(I_\text{render}, I_\text{norm}) + \lambda_s \cdot L_\text{SSIM}(I_\text{render}, I_\text{norm}),\quad \lambda_s=0.2
$$
However, independent frame restoration leads to multi-view inconsistencies, resulting in blur, floaters, and structural instability in traditional 3DGS when auxiliary losses are not included.

## 2. Physics-Informed Priors and Multi-View Consistency

To overcome the single-view/multi-view tension, Dehaze-then-Splat adds three auxiliary losses, each designed to enforce implicit cross-view constraints:

1. **Depth Supervision Loss:** Pseudo inverse-depth maps $d_p$ are generated for each input view using Depth Anything V2 on the original smoky frames. The rendered depth $d_r$ is anchored to $d_p$ with a weighted, scale-invariant Pearson correlation:
   $$
   L_\text{depth} = 1 - \text{Corr}_w(d_r, d_p) \\
   \text{with} \quad \text{Corr}_w(d_r, d_p) = \frac{\sum_i w_i (d_r^i-\bar{d}_r)(d_p^i-\bar{d}_p)}{\sqrt{\sum_i w_i (d_r^i-\bar{d}_r)^2} \cdot \sqrt{\sum_i w_i (d_p^i-\bar{d}_p)^2}},\quad w_i = \alpha_i
   $$
   Depth supervision ($\lambda_p=0.1$) is the most impactful prior; its ablation reduces PSNR by 0.79 dB.

2. **Dark Channel Prior (DCP) Regularization:** Nonzero dark channel intensities in rendered images $\tilde{I}$ are penalized, promoting patchwise haze removal as per classic DCP:
   $$
   L_\text{DCP} = \text{mean}\left(\min_{c\in\{\mathrm{R},\mathrm{G},\mathrm{B}\}} \mathrm{MinPool}_k(\tilde{I}_c)\right),\quad k=15
   $$
   Regularization weight is $\lambda_d=0.01$.

3. **Dual-source Gradient Matching:** High-frequency structure is enforced using an auxiliary dehazed reference $I_\text{struct}$ (from MB-TaylorFormer), brightness-aligned to the Nano Banana output. The $L_1$ norm between spatial gradients is minimized:
   $$
   L_\text{grad} = \|\nabla \tilde{I} - \nabla \text{Norm}(I_\text{struct}, I_\text{nano})\|_1
   $$
   with $\lambda_g=0.1$.

The overall loss is thus:
$$
L_\text{total} = (1-\lambda_s) L_1 + \lambda_s L_\text{SSIM} + \lambda_d L_\text{DCP} + \lambda_p L_\text{depth} + \lambda_g L_\text{grad}
$$

## 3. MCMC-Based Densification and Early Stopping

3DGS capacity is regulated via a Monte Carlo Markov Chain (MCMC) densification process, replacing gsplat’s DefaultStrategy. At each step in the designated densification window ($t \in [500, 3000]$), each Gaussian may be stochastically perturbed, split, or relocated using noise-injection proposals (NOISE_LR$=5\times 10^5$ annealed to zero by step 8000). A hard cap ($\text{CAP}_\text{MAX}=500\,000$) limits cloud size.

Early stopping is enforced: after $t_\text{stop}=3\,000$, the probability of further Gaussian splitting $p_\text{split}(t)=0$. Empirically, this yields stable clouds of approximately $104\,000$–$161\,000$ Gaussians, avoiding capacity “explosion” and overfitting to per-view artifacts.

Throughout densification, all auxiliary losses remain active, ensuring newly introduced Gaussians are regularized for geometric and photometric consistency.

## 4. Training Regimen and Implementation Details

Training spans 20,000 steps with validation every 1,000 steps. Key hyperparameters include spherical-harmonic degree (0→3 progressive), learning rates ($\text{lr}_\text{means}=1.6\times 10^{-4}$, $\text{lr}_\text{SH0}=2.5\times 10^{-3}$), scene scale 2.0, and gamma $\gamma=1.0$ (gsplat default $\gamma=0.5$ reduces PSNR by $\sim$5 dB). Auxiliary loss weights are fixed ($\lambda_s=0.2, \lambda_d=0.01, \lambda_p=0.1, \lambda_g=0.1$).

Scene initialization uses $50\,000$ (or $20\,000$ for dark scenes) Gaussians; backgrounds are white (or black for Hinoki).

The training loop pseudocode:
```python
for t in 1..20000:
    I_norm = BrightnessNormalize(NanoBanana(I_smoke))
    Ĩ, d_r, α = Render3DGS()
    L = (1-0.2)*L1(Ĩ, I_norm) + 0.2*Lssim(Ĩ, I_norm) + 0.01*L_DCP(Ĩ) + \
        0.1*L_depth(d_r, d_p; α) + 0.1*L_grad(Ĩ, I_struct)
    Backpropagate(L)
    if 500 <= t < 3000:
        ApplyMCMCProposals()
    UpdateGaussians()
end
Select checkpoint t* in {2000…4000} maximizing PSNR on hold-out views.
```

## 5. Quantitative Results and Ablation

On the Akikaze dataset, the baseline (no auxiliary losses, STOP=20k) achieves PSNR of 19.48 dB and SSIM of 0.632. Incorporating all auxiliary losses and early stopping (STOP=2k) improves performance to 20.98 dB PSNR (+1.50 dB) and 0.683 SSIM (+0.051).

| Setting                           | PSNR (dB) | SSIM  | ∆PSNR  |
|------------------------------------|-----------|-------|--------|
| Baseline (no aux, STOP=20k)        | 19.48     | 0.632 |  —     |
| Full pipeline (all losses, STOP=2k)| 20.98     | 0.683 | +1.50  |

Ablations show:
- Depth supervision: +0.79 dB
- MCMC vs. DefaultStrategy: +0.47 dB at matched STOP
- Early stopping (STOP=3k vs. 15k): +0.35 dB
- Dual-source gradient: +0.06 dB

Qualitatively, Dehaze-then-Splat yields reconstructed views with sharp texture, stable geometry, and minimal haze, while baseline models manifest blur, color drift, and floater artifacts.

## 6. Failure Modes, Scope, and Extensions

Absent auxiliary priors or early stopping, the 3DGS model tends to catastrophically overfit, producing large numbers of Gaussians (>$900\,000$) and severe floater artifacts. End-to-end Koschmieder-based models fail to converge (β→0), resulting in poor PSNR (10.28 dB).

Computationally, the combination of MCMC densification and 20k steps of 3DGS optimization incurs significantly higher resource requirements compared to classical NeRFs, primarily due to larger Gaussian budgets.

Extensions include integrating cross-view consistency directly into the dehazing stage (e.g., video-based generative modeling, multi-view attention), adaptive Gaussian capacity scheduling, and adaptation for other atmospherics (fog, particulate smoke) through modified priors.

## 7. Broader Impact and Significance

Dehaze-then-Splat demonstrates that explicit separation of per-frame dehazing from multi-view, prior-driven 3D reconstruction can balance state-of-the-art 2D image fidelity with global, physically plausible geometry. The pipeline establishes that per-image generative restoration must be complemented by regularization—both in model capacity and in geometry/appearance consistency—at the 3D stage to avoid degraded rendering outcomes. This architecture achieves substantive quantitative and qualitative gains for smoke-free novel view synthesis and provides a flexible blueprint for addressing a range of atmospheric degradations using physics-informed priors in learning-based 3D reconstruction frameworks [2604.13589].

Source: https://www.emergentmind.com/topics/dehaze-then-splat-pipeline