Variance-Corrected Fusion Overview

• Variance-Corrected Fusion is a set of strategies that integrate noisy measurements by explicitly modeling local variances for minimal uncertainty.
• VCF adapts weighting in applications like 3D reconstruction, image generation, and sensor fusion to enhance reliability and detail.
• Implementations use closed-form and iterative algorithms to calibrate noise parameters, ensuring accurate uncertainty preservation in fused outputs.

Variance-Corrected Fusion (VCF) encompasses a family of statistical and algorithmic strategies for integrating multiple noisy or redundant measurements, with explicit correction or estimation of local variances to achieve minimum-variance, spatially and contextually adaptive fusion. VCF arises in diverse contexts including image generation, 3D reconstruction, and variational inference. It generalizes classical weighted averaging by learning, modeling, or correcting local noise levels, restoring the correct uncertainty structure in fused outputs, and often yields closed-form solutions under Gaussian or Laplace noise models.

1. Mathematical Foundations of Variance-Corrected Fusion

At its core, variance-corrected fusion formulates fusion as an estimation problem under spatially and contextually varying noise. Consider $K$ independent measurements $\{d_k\}_{k=1}^K$ of a quantity $x$ at each location $i$, each corrupted by zero-mean noise with local variance $\sigma_{k,i}^2$. The classical linear minimum variance estimator is

$$x_i = \frac{\sum_k w_{k,i}\,d_{k,i}}{\sum_k w_{k,i}}, \qquad\text{where}\quad w_{k,i}=1/\sigma_{k,i}^2$$

which is the unique unbiased linear estimator minimizing variance when the variances $\{\sigma_{k,i}^2\}$ are known and the noise is Gaussian. This formula arises in settings including sensor data fusion, multi-channel signal processing, and Bayesian inference.

Variance-corrected fusion can extend to:

• Non-Gaussian noise (e.g., Laplacian, Poisson-Gaussian),
• Adaptive spatially varying variance estimation,
• Incorporation of priors, regularization, or constraints such as total variation (TV) or total generalized variation (TGV).

When variances are unknown, VCF frameworks estimate them either jointly with the signal or from empirical residuals, using iterative or alternating minimization schemes.

2. VCF in High-Precision Depth and Phase Fusion

In high-precision 3D imaging tasks, such as structured light (SL) depth reconstruction, raw RGB channel observations have spatially varying noise characteristics, often well modeled by a Poisson–Gaussian process: $$n_i(u,v)\sim\mathcal{N}\bigl(0,\,\sigma_{n,i}^2(I_i)\bigr), \quad \sigma_{n,i}^2(I) = k_{0,i} + k_{1,i} I$$ where $k_{0,i}$ and $k_{1,i}$ encode read and shot noise, estimated via multi-level intensity calibration (Oh et al., 11 Mar 2026).

Each channel yields an independent wrapped/unwrapped phase estimate $\{d_k\}_{k=1}^K$0 with analytic variance

$\{d_k\}_{k=1}^K$1

where $\{d_k\}_{k=1}^K$2 and $\{d_k\}_{k=1}^K$3 are amplitude and modulation parameters derived from the phase scanning protocol.

The optimal fusion is then

$\{d_k\}_{k=1}^K$4

yielding the minimum possible variance among unbiased linear estimators. In LCAMV, this approach corrects for spatially-varying noise and lateral chromatic aberration, enabling accurate 3D reconstructions across color-varying surfaces. Outlier rejection is performed by dropping channels whose phase estimates deviate beyond the $\{d_k\}_{k=1}^K$5 confidence interval of the best channel (Oh et al., 11 Mar 2026).

Empirically, LCAMV reduced mean squared plane error by up to $\{d_k\}_{k=1}^K$6 compared to the next-best baseline. The method is strictly superior to simple averaging (e.g., Mean-RGB, Y′UV, or single-channel fusion), which ignores per-channel noise and induces chromatic artifacts.

3. Variance-Corrected Fusion in Patch-wise Diffusion and Image Generation

Patch-based large-image generation using pretrained diffusion models encounters a distinct fusion challenge: in overlapping regions, each patch generates stochastic samples following a known noise schedule (typically $\{d_k\}_{k=1}^K$7 per patch per diffusion step). Naive averaging in these regions,

$\{d_k\}_{k=1}^K$8

leads to variance underestimation: $\{d_k\}_{k=1}^K$9, instead of the target $x$0. This repeated under-noising yields "over-smooth, blurred outputs" (Sun et al., 2024).

Variance-corrected fusion explicitly restores per-pixel variance by affine remapping: $x$1 where $x$2 are chosen to preserve mean and enforce the correct variance. For $x$3 equivalent overlaps: $x$4 and for weighted fusion with weights $x$5: $x$6 where $x$7, $x$8. The general VCF fusion rule is then

$x$9

This ensures that the fused output at each reverse step in denoising diffusion probabilistic models (DDPMs) maintains the statistical properties of the learned generative process.

Integration into patch-based pipelines is by per-step, per-pixel accumulation of weighted samples and predicted means, followed by application of the above correction. Pseudocode and detailed implementation for panoramic diffusion are presented in (Sun et al., 2024).

4. Joint Estimation and Regularization: Bayesian and Variational Perspectives

Variance-corrected fusion can be embedded in optimization and Bayesian estimation frameworks. The Confidence-Driven TGV Fusion (C-TGV) model (Ntouskos et al., 2016) generalizes VCF by jointly estimating the fused field $i$0 and a spatial confidence (precision) field $i$1 via the biconvex energy: $i$2 The data term uses an adaptive $i$3 fidelity weighted by $i$4, while the prior term imposes an inverse-Wishart hyper-prior on $i$5, circumventing overconfidence and spatial singularities.

Biconvex minimization alternates between closed-form updates of precisions

$i$6

and convex optimization of $i$7 (solved via PDHG) given fixed $i$8.

The spatial regularity of both $i$9 (via TGV) and $\sigma_{k,i}^2$0 (implicitly via residual coupling and hyper-prior) promotes piecewise-polynomial, edge-preserving solutions and adaptively weights residuals. This approach is especially advantageous when the actual noise or uncertainty structure is unknown or highly heterogeneous.

From a Bayesian perspective, the VCF map estimate is the joint maximum a posteriori (MAP) estimate under a noise model (Laplace or Gaussian) and a prior over variances/precisions (inverse-Wishart), justifying the adaptive, context-sensitive weighting.

5. Algorithmic Schemes and Implementation Details

Implementation of variance-corrected fusion varies by domain:

• Diffusion models: Each diffusion step accumulates per-pixel sums of weighted samples and means from overlapping patches. Correction factors $\sigma_{k,i}^2$1 (defined above) restore the prescribed variance. Guidance weights (e.g., from precomputed maps decaying from patch centers) can optionally be incorporated (Sun et al., 2024). Style Alignment is orthogonal to VCF.
• Phase/depth fusion: Calibration of per-channel noise parameters $\sigma_{k,i}^2$2 is essential, typically via capture of multiple known intensities and empirical variance fitting. During fusion, channels or frequencies with deviant phase estimates (outside a noise-derived confidence band) are excluded by inflating their variance to infinity, ensuring robustness to outliers (Oh et al., 11 Mar 2026).
• C-TGV/variational fusion: Alternating convex search (ACS) iterates between closed-form $\sigma_{k,i}^2$3 updates and inner convex field optimization. PDHG accelerates convergence of non-smooth convex subproblems. Empirical convergence is rapid, requiring $\sigma_{k,i}^2$4 outer and $\sigma_{k,i}^2$5 inner iterations (Ntouskos et al., 2016).

Run-time and computational complexity are dominated by per-pixel statistical computations and convex solvers, but all steps are parallelizable and deterministic, enabling practical application in large-scale or real-time settings.

6. Empirical Performance and Application Domains

Variance-corrected fusion systematically improves accuracy and realism in domains with heterogeneous noise and redundancy.

• Structured Light 3D Reconstruction: LCAMV reduced mean squared planar error by up to $\sigma_{k,i}^2$6 over Mean-RGB, Y′UV, and Green-channel baselines, and outperformed methods omitting either minimum-variance fusion or LCA correction. Qualitative depth maps display spatial uniformity and ablation studies show catastrophic failure if either component is omitted (Oh et al., 11 Mar 2026).
• Patch-based Image Generation: On $\sigma_{k,i}^2$7 panoramic images, DDPM with VCF reduced FID from $\sigma_{k,i}^2$8 (DDIM-MD baseline) to $\sigma_{k,i}^2$9, with further improvements (down to $$x_i = \frac{\sum_k w_{k,i}\,d_{k,i}}{\sum_k w_{k,i}}, \qquad\text{where}\quad w_{k,i}=1/\sigma_{k,i}^2$$0) from guided weights and style alignment. Visual inspection confirms that VCF eliminates blur and seam artifacts induced by naive fusion (Sun et al., 2024).
• Depth Image Fusion and Scene Reconstruction: C-TGV with learned confidence maps (using ACS) achieves lower RMSE than fixed-weight TGV fusion and robustly reduces outlier rates in real datasets such as KITTI, yielding sharper edges and smoother surfaces than conventional approaches (Ntouskos et al., 2016).

A table summarizing the main VCF methodologies in distinct domains:

Domain	Fusion Rule/Formulation	Noise/Uncertainty Handling
Structured Light 3D	$$x_i = \frac{\sum_k w_{k,i}\,d_{k,i}}{\sum_k w_{k,i}}, \qquad\text{where}\quad w_{k,i}=1/\sigma_{k,i}^2$$1	Poisson–Gaussian, per-pixel variance, outlier exclusion (Oh et al., 11 Mar 2026)
Patch-based Diffusion	$$x_i = \frac{\sum_k w_{k,i}\,d_{k,i}}{\sum_k w_{k,i}}, \qquad\text{where}\quad w_{k,i}=1/\sigma_{k,i}^2$$2	Prescribed model variance, correction per-overlap (Sun et al., 2024)
C-TGV Fusion	Joint minimization of $$x_i = \frac{\sum_k w_{k,i}\,d_{k,i}}{\sum_k w_{k,i}}, \qquad\text{where}\quad w_{k,i}=1/\sigma_{k,i}^2$$3	Adaptive precision estimation, inverse-Wishart prior (Ntouskos et al., 2016)

7. Broader Applicability and Generalization

Variance-corrected fusion, as a generic strategy for minimum-variance estimation, generalizes to domains such as:

• Multi-frequency or multi-wavelength fringe analysis,
• Combination of disparate sensors (stereo vision, time-of-flight, etc.),
• High-dynamic-range (HDR) fusion from multiple exposures,
• Any scenario involving redundant, heterogeneously-noisy measurements.

The only essential requirements are the ability to calibrate or estimate local variances and the assumption of independence or, at minimum, uncorrelated noise among sources. The fusion formula

$$x_i = \frac{\sum_k w_{k,i}\,d_{k,i}}{\sum_k w_{k,i}}, \qquad\text{where}\quad w_{k,i}=1/\sigma_{k,i}^2$$4

achieves minimum variance among all linear unbiased estimators whenever these conditions hold.

A plausible implication is that any further advancements in high-fidelity physical measurement, stochastic generative modeling, or sensor-based perception can benefit from explicit VCF frameworks, particularly where traditional averaging is insufficient to preserve uncertainty or spatial structure.