Multi-Scale Disparity Fusion

• Multi-scale disparity fusion is a method that integrates fine and coarse disparity cues across spatial resolutions to improve depth estimation in ambiguous regions.
• It employs diverse network architectures, including multi-branch, transformer-based, and evidential fusion, to reconcile heterogeneous depth information.
• Practical applications span stereo matching, depth completion, and sensor fusion, delivering enhanced accuracy and uncertainty-aware performance.

Multi-scale disparity fusion refers to a collection of algorithmic and network-based strategies that aggregate, reconcile, and regularize disparity (depth) information across spatial scales, feature resolutions, network hierarchies, or sensor modalities. By leveraging scene structure at multiple scales—ranging from fine edges to broad contextual cues—these methods address the non-uniform accuracy and robustness of single-scale, single-source, or single-path depth estimates. Multi-scale fusion is fundamental to modern stereo matching, light field image synthesis, and disparity refinement tasks, providing a principled framework to combine heterogeneous depth cues, exploit uncertainty, and optimize both local fidelity and global consistency.

1. Foundational Principles of Multi-Scale Disparity Fusion

Multi-scale disparity fusion emerged as a response to the limitations of single-scale stereo matching and the ambiguous, ill-posed regions such as textureless surfaces, occlusions, and fine object structures. The core principle is that disparity evidence at different spatial or contextual resolutions is complementary: coarse-scale estimates stabilize weakly-textured or defocused regions while fine-scale estimates capture edge detail and thin structures.

Formally, multi-scale fusion typically involves:

• Construction of cost or feature pyramids at distinct spatial resolutions (Zhu et al., 2019, Rao et al., 2019).
• Parallel or cascaded neural branches with varying receptive fields or kernel sizes (Zhu et al., 2019, Rao et al., 2019, Zhang et al., 2019).
• Explicit mechanisms to combine disparities or confidence weights from each scale, such as concatenation, learnable attention, or Bayesian evidence aggregation (Lou et al., 2023).

The coherence of scale-wise disparity cues is further ensured via supervision at multiple pyramid levels, attention-guided residual fusion, or explicit geometric consistency constraints (Xing et al., 2021, Zhang et al., 2019).

2. Algorithmic and Network Architectures

Multi-Branch and Pyramid Modules

Networks such as CFP-Net employ a Cross-Form Spatial Pyramid (CFSPP) module to aggregate dilated and pooled contexts at several scales, providing multi-resolution cost volumes (Zhu et al., 2019). MSDC-Net supplements DenseNet-based multi-scale unary feature fusion with multi-level residual 3D convolutions, fusing geometry context at increasingly global scales (Rao et al., 2019). Both designs fuse features or match costs across levels via channel concatenation followed by learnable convolution, allowing the network to adaptively decide the contribution of each scale per spatial location and disparity.

Self-Attention and Transformer Architectures

The Multi-scale Disparity Transformer (MDT) isolates and disentangles disparity reasoning via multiple self-attention branches, each operating on selected angular/light field sub-aperture images and channel subsets tailored to a specific disparity band (Hu et al., 2024). The outputs of all branches are concatenated along the channel axis to construct a complete, multi-scale disparity embedding, subsequently refined via convolution and angular attention. This architecture sharply reduces complexity while providing explicit disparity-range specialization.

Evidential and Uncertainty-Aware Fusion

ELFNet generalizes multi-scale fusion to an evidential framework. After multi-scale cost aggregation, separate regression heads output Normal-Inverse-Gamma (NIG) parameters at each pyramid level (Lou et al., 2023). The mixture-of-NIG (MoNIG) operator fuses low- and high-level evidence into a unified NIG posterior per pixel, supporting consistent disparity point estimates and pixelwise uncertainty quantification. This paradigm supports both intra-scale fusion (across cost pyramid levels) and inter-head fusion (combining cost-volume and transformer branches), leading to trustworthy, confidence-aware disparity maps.

Continuous and Prior-Guided Fusion

The continuous cost aggregation (CCA) framework for Dual-Pixel imagery exploits a parabola-based, continuous-disparity SGM variant with an inherent multi-scale hierarchy (Monin et al., 2023). At each finer scale, upsampled, weighted priors (parabola coefficients) from coarser levels bias the cost aggregation and disparity minimization, reconciling fine-grained local cues against robust, blur-averaged coarse evidence. Cross-scale fusion is performed with an explicit prior-addition step on quadratic coefficients, resulting in efficient, closed-form inference and monotonic performance improvement as scales are added.

3. Mathematical Formulations and Fusion Operators

Learnable and Explicit Fusion

Most deep methods employ channel-wise concatenation followed by 1×1 or 3×3 convolution to combine multi-scale features, enabling the network to synthesize spatial and semantic context adaptively (Zhu et al., 2019, Rao et al., 2019, Zhang et al., 2019). Element-wise addition and feature concatenation are both used—e.g., in MSFNet, fine- and coarse-scale features are fused via addition and concatenation, followed by a 1×1 bottleneck convolution to produce a unified prior (Zhang et al., 2019).

In transformer architectures, the concatenation of per-branch disparity self-attention outputs after Q-K-V computation delivers the multi-scale fusion, with subsequent convolutions learning higher-order interactions (Hu et al., 2024).

Bayesian/Evidential Fusion

ELFNet's MoNIG operation fuses NIG parameters from each scale as follows. For two NIGs with parameters $(\delta_1, \gamma_1, \alpha_1, \beta_1)$ and $(\delta_2, \gamma_2, \alpha_2, \beta_2)$: $$\delta_{\text{fused}} = \frac{\gamma_1 \delta_1 + \gamma_2 \delta_2}{\gamma_1 + \gamma_2}$$

$\gamma_{\text{fused}} = \gamma_1 + \gamma_2$

$$\alpha_{\text{fused}} = \alpha_1 + \alpha_2 + \frac12$$

$$\beta_{\text{fused}} = \beta_1 + \beta_2 + \frac{(\delta_1 - \delta_2)^2 \gamma_1 \gamma_2}{2(\gamma_1 + \gamma_2)}$$

This operator propagates both mean and uncertainty, enabling downstream risk-aware applications (Lou et al., 2023).

Attention-Guided Fusion

In scale-consistent fusion for light field synthesis, an attention-guided multi-scale residual module fuses disparity and RGB features at each pyramid level using learned per-pixel attention masks. The final residual map at scale $i$ is given by

$$\hat D^i_{t_0} = W^i_{\rm disp} \odot R^i_{\rm disp} + W^i_{\rm rgb} \odot R^i_{\rm rgb} + \Gamma_{\rm up} [\hat D^{i-1}_{t_0}]$$

combining scale-wise residuals in a coarse-to-fine manner (Xing et al., 2021).

Parabola-Based Prior Fusion

The CCA framework fuses upsampled coarse-scale priors via coefficient addition: $$\alpha_{p,s-1} \gets \alpha_{p,s-1} + w\,A^{\mathrm{prior}}_{p,s-1}$$

$$\beta_{p,s-1} \gets \beta_{p,s-1} + w\,B^{\mathrm{prior}}_{p,s-1}$$

where $w$ is an empirically chosen scale-fusion parameter (Monin et al., 2023).

4. Applications, Performance, and Comparative Results

Multi-scale disparity fusion is central to state-of-the-art stereo matching, depth completion, and light field synthesis.

• CFP-Net: Achieved Out-All (>3 px) error rates of 1.83% on KITTI 2012 and D1-all = 2.31% on KITTI 2015, improving over SPP/ASPP-based competing architectures primarily in ill-posed and textureless regions (Zhu et al., 2019).
• MSDC-Net: On Scene Flow, combined 2D and 3D multi-scale fusion reduced >3 px error from 18.2% (single-scale) to 8.7% (multi-scale), halving MAE and RMS (Rao et al., 2019). On KITTI 2015, D1-all = 2.26% was achieved.
• ELFNet: Incorporation of intra-evidential (multi-scale) and inter-evidential (cost-volume/transformer) fusion produced state-of-the-art accuracy and robust uncertainty estimation (Lou et al., 2023).
• CCA: For Dual-Pixel sensors, the multi-scale prior drastically reduced error as scales increased (from 0.069 to 0.055 geometric mean across metrics), enabling robust DP disparity even in challenging PSF and focus conditions with O(WH) space and O(WH(D+R)) time complexity (Monin et al., 2023).
• MDT: By explicit branch-based disparity disentanglement, MDT achieves lower complexity (≈33% of projection cost), absence of QK-FFN cost, and up to +0.41 dB PSNR on 4× super-resolution tasks relative to prior LFSR backbones (Hu et al., 2024).
• SDF-MAN: Multi-scale patch-based discriminator fusion, combined with a U-Net refiner and MRF prior, achieved lower absolute disparity errors (e.g., 3.10 px → 2.84 px with semi-supervision on Synthetic Garden) and generalizes fusion to arbitrary sensor combinations (Pu et al., 2018).

Ablation studies consistently demonstrate that multi-scale fusion mechanisms outperform single-scale variants and provide the largest relative improvements in ambiguous or ill-posed regions.

5. Heterogeneous Modalities, Sensor Fusion, and Disparity Regularization

Multi-scale fusion is not limited to spatial resolution. SDF-MAN applies these principles across algorithmic and sensor boundaries—simultaneously fusing stereo, monocular, and ToF disparity cues together with image gradients, using a fully convolutional refiner and a patch-level multi-scale adversarial discriminator (Pu et al., 2018). This architecture can ingest any stack of raw disparity maps plus appearance cues, supervising with a mixture of data, MRF, and adversarial losses at multiple scales.

Attention-guided, coarse-to-fine fusion modules as in scale-consistent light field synthesis (Xing et al., 2021) further combine RGB and intermediate disparity features, using per-scale residual blocks with learnable attention weights, in both spatial and angular domains (SAA), enforcing globally consistent structure in the final 4D disparity field.

6. Quantitative Impact and Implementation Considerations

Empirical studies report monotonic accuracy gains as the number of scales increases, subject to computational and memory constraints. For example, CCA error rates improve with each scale up to three (0.069 → 0.067 → 0.055), with only O(WH) workspace required due to compact parabola parameterization (Monin et al., 2023). In deep architectures, parameter efficiency is retained: MSDC-Net achieves or outperforms much larger single-scale nets with ~4.6M parameters, while MDT reduces transformer compute by ≈70% without degrading accuracy (Hu et al., 2024, Rao et al., 2019).

Hyperparameters such as scale-fusion weights, pyramid factors, path directions (in CCA/SGM-like algorithms), and attention module depths are dataset- and modality-dependent, with optimal settings reported globally in each cited work.

7. Challenges and Ongoing Research

Open challenges include handling scale inconsistency across heterogeneous capture settings (addressed via scale-consistent volume rescaling in (Xing et al., 2021)), reliable uncertainty quantification (see evidential fusion in (Lou et al., 2023)), and the design of efficient, learnable cross-scale attention or transformer modules for high-dimensional inputs (Hu et al., 2024).

A plausible implication is the broader deployment of multi-scale, attention-guided, and evidential disparity fusion in applications requiring dense depth in the presence of strong ambiguity, such as autonomous navigation, immersive rendering, low-light imaging, and sensor fusion across hybrid depth modalities.