Gradient Consistency Model for Disparity Estimation

• Gradient Consistency Model (GCM) is a self-scheduling weighting framework that leverages gradient consistency to validate local linearised brightness constancy in multi-view disparity estimation.
• The method penalizes view pairs with mismatched spatial gradients by incorporating gradient and scale inconsistency in the variational energy, thereby optimizing the data term dynamically.
• Empirical results demonstrate that GCM achieves improved accuracy (lower RMSE) and faster convergence compared to traditional coarse-to-fine schemes, especially at object boundaries.

Gradient Consistency Model (GCM) denotes an analytically inspired weighting model for multi-view variational disparity estimation in which gradient consistency information is used to assess the validity of the linearised brightness constancy constraint. In the formulation introduced in "Multi-view Disparity Estimation Using a Novel Gradient Consistency Model" (Gray et al., 2024), the model penalises the data term for view pairs that have a mismatch between the spatial gradients in the source view and the spatial gradients in the target view, and it is described as self-scheduling because the weights evolve as the algorithm progresses rather than following a tuned or learned schedule.

1. Problem setting and motivation

The model is posed for multi-view disparity estimation in a calibrated camera array. A reference view $I_r$ is compared with multiple target views $I_t$, and the goal is to estimate a scalar disparity-like quantity $w(\mathbf{s})$ over image coordinates $\mathbf{s}\in\Omega$. In the planar, equally oriented camera setting considered in the paper, the residual displacement to target $t$ is coupled to a residual reciprocal depth or normalized disparity $\delta w(\mathbf{s})$ through the baseline vector $\mathbf{B}_t$: $$\delta \mathbf{v}_t(\mathbf{s}) = \mathbf{B}_t \cdot \delta w(\mathbf{s}).$$

Variational disparity estimation begins from the nonlinear brightness constancy relation

$$I_r(\mathbf{s}) - I_t(\mathbf{s} + \delta \mathbf{v}_t(\mathbf{s})) = 0,$$

which is usually replaced by a local first-order linearisation,

$$\langle \nabla I_{r,t}(\mathbf{s}), \delta \mathbf{v}_t(\mathbf{s}) \rangle + \delta I_{r,t}(\mathbf{s}) = 0, \qquad \nabla I_{r,t}(\mathbf{s}) = \frac{\nabla I_r(\mathbf{s}) + \nabla I_t(\mathbf{s})}{2}.$$

This linearisation is only valid when the residual displacement is small and the image surface is sufficiently smooth. The paper identifies two standard failure modes: large displacements and high-frequency or non-smooth regions such as sharp edges, object boundaries, and textured regions. Standard remedies therefore use schedules, chiefly coarse-to-fine processing or progressive inclusion of views. Coarse-to-fine suppresses high-frequency content early by blurring and downsampling; progressive inclusion of views initially uses nearby views with smaller baselines. GCM is proposed as a replacement for such hand-designed schedules by estimating, from the data itself, where the local linearisation is credible (Gray et al., 2024).

2. Variational formulation

The unweighted variational energy is written as

$I_t$0

with data term

$I_t$1

GCM replaces this by a weighted data term,

$I_t$2

so that each target view, pixel, and eventually each scale contributes according to an estimate of linearisation validity.

For the multi-view disparity-specific setting, at scale $I_t$3 the brightness constancy equation becomes

$I_t$4

with linearised form

$I_t$5

The scale-dependent image difference and gradient are

$I_t$6

and

$I_t$7

with

$I_t$8

A central clarification in the paper is that gradient consistency does not mean a gradient constancy observation model of the Brox type. Here it means that the gradients observed in the source and target views should be mutually compatible with the assumption that the local linearised brightness constancy equation is valid. If the reference and target gradients at corresponding locations differ substantially, the local image surface cannot be approximated well by a single local linear function over the displacement interval (Gray et al., 2024).

3. Equivalent-noise weighting and the self-scheduling mechanism

The analytical backbone of GCM is an inverse-noise weighting argument. Ignoring regularisation for the derivation, the Euler–Lagrange equation of the weighted multi-view multi-scale data term is

$I_t$9

which yields the estimate

$w(\mathbf{s})$0

If perturbations $w(\mathbf{s})$1 affect each $w(\mathbf{s})$2, the perturbed solution is

$w(\mathbf{s})$3

Appendix A of the paper shows that minimizing expected squared error gives

$w(\mathbf{s})$4

with arbitrary $w(\mathbf{s})$5.

The model then interprets gradient inconsistency, scale inconsistency, and image acquisition noise as sources of equivalent noise: $w(\mathbf{s})$6 The first term is the gradient inconsistency contribution,

$w(\mathbf{s})$7

This quantity uses first-order spatial gradients only; it is based on the difference of gradients between target and reference, projected onto the baseline direction.

The gradient mismatch is multiplied by an estimate of the current disparity error magnitude,

$w(\mathbf{s})$8

where

$w(\mathbf{s})$9

The second term, $\mathbf{s}\in\Omega$0, is a scale inconsistency upper bound derived from the mismatch between two coarse-scale linearised forms: $\mathbf{s}\in\Omega$1 and

$\mathbf{s}\in\Omega$2

The resulting upper bound is

$\mathbf{s}\in\Omega$3

with finest-scale surrogate

$\mathbf{s}\in\Omega$4

The third term,

$\mathbf{s}\in\Omega$5

comes from assuming additive white Gaussian image noise filtered by a Gaussian of width $\mathbf{s}\in\Omega$6. The paper sets

$\mathbf{s}\in\Omega$7

Within a common baseline direction, the paper additionally imposes a monotonicity constraint. The baseline vector space is divided into 8 angular sectors $\mathbf{s}\in\Omega$8, and for view $\mathbf{s}\in\Omega$9 in sector $t$0,

$t$1

This ensures that a farther view is not trusted more than an intermediate one in the same direction if gradient consistency worsens. Because all these quantities depend on the current images, scale, and disparity state, the weights evolve automatically during optimization; this is the precise meaning of self-scheduling in the method (Gray et al., 2024).

4. Optimization procedure

The regulariser is total variation,

$t$2

with $t$3 the $t$4 norm. The operational data term is

$t$5

Using standard gradient-based linearisation and IRLS, this becomes

$t$6

where $t$7 are the IRLS weights.

The implementation uses warping of target images according to the current disparity estimate, linearisation of the data term, IRLS to realize the $t$8 robust norm, and conjugate gradient to solve each least-squares system. A practical iteration therefore consists of warping each target view to the reference using the current disparity, computing $t$9, $\delta w(\mathbf{s})$0, and $\delta w(\mathbf{s})$1, recomputing the GCM weights $\delta w(\mathbf{s})$2, recomputing IRLS weights $\delta w(\mathbf{s})$3, solving for $\delta w(\mathbf{s})$4, and updating $\delta w(\mathbf{s})$5. The paper states that warping is performed inside the IRLS scheme, prior to each reweighting stage (Gray et al., 2024).

Several implementation details are material. The method uses a 5×5 median filtering step after each stage; invalid warp regions, where displacements leave the image, are filled by copying data from the reference view into the warped target; and baselines are normalized so nearest views satisfy $\delta w(\mathbf{s})$6. GCM can use multiple scales simultaneously. For the 4D light-field experiments the implementation uses 3 scales. For Middlebury 2006 it uses 6 scales, but with a sliding window of 3 active scales at a time for practical memory and working-set reasons. The paper presents this as a self-scheduling alternative to coarse-to-fine and progressive inclusion, since all views and scales can feed the optimization concurrently while the weights determine their current influence (Gray et al., 2024).

5. Empirical performance and ablations

The experiments use the synthetic 4D Lightfield Dataset of Honauer et al. 2016 and Middlebury 2006. On the 4D light-field benchmark, 24 scenes are used from stratified, additional, and training sets, with 17 of the 81 views arranged in a cross-hair shape. On Middlebury 2006, half-size images and 7 views in a collinear arrangement are used. The paper reports final average RMSE and also tracks progress by the number of linear system solves $\delta w(\mathbf{s})$7 (Gray et al., 2024).

Benchmark	Method	Final average RMSE
4D Lightfield, single-scale	Naive equal weighting	0.3260942820416670
4D Lightfield, single-scale	Gradient Consistency Model	0.22723282625
4D Lightfield, single-scale	Progressive inclusion of views	0.2519847182916670
4D Lightfield, multi-scale	Coarse-to-fine	0.26925784950000000
4D Lightfield, multi-scale	Gradient Consistency Model	0.224522
4D Lightfield, multi-scale	GCM with $\delta w(\mathbf{s})$8	0.3194290569583330
4D Lightfield, multi-scale	GCM with $\delta w(\mathbf{s})$9	0.288265
Middlebury 2006	Coarse-to-fine	2.826845383666770
Middlebury 2006	Gradient Consistency Model	2.1100248087789457

The reported convergence behavior is also central. On the 4D Lightfield multiscale benchmark, GCM reaches near-final accuracy after roughly 30–40 solves, whereas coarse-to-fine may require 100–200 solves depending on the scene. The paper states that GCM outperforms standard coarse-to-fine schemes and the recently proposed progressive inclusion of views approach in both rate of convergence and accuracy.

The ablations isolate the two main consistency terms. Setting $\mathbf{B}_t$0 makes convergence much slower: the method over-favors fine-scale data and underweights coarse-scale data, making large initial corrections difficult. Setting $\mathbf{B}_t$1 leaves convergence rate similar but degrades final accuracy: the method overweights coarse-scale data and underweights fine-scale data near boundaries, producing smeared or misshapen discontinuities. The paper therefore interprets $\mathbf{B}_t$2 as primarily aiding convergence speed and $\mathbf{B}_t$3 as primarily aiding accuracy, especially boundary localization. The strongest qualitative gains are reported at object boundaries, sharp disparity discontinuities, and scenes such as backgammon and pens. The method is also reported to remain robust when simulating slight barrel distortion and small baseline measurement noise (Gray et al., 2024).

6. Scope, interpretation, and acronymic ambiguity

A common misconception is to equate this GCM with a gradient constancy observation model. The paper explicitly rejects that reading. In its formulation, gradient consistency is a reliability signal for the local linearised brightness constancy equations, not a replacement observation model. The distinction from ordinary robust estimation is equally explicit: standard robust norms downweight large residuals, whereas GCM downweights observations whose linearisation should not be trusted in the first place. The method is therefore an adaptive, analytically motivated confidence model that modulates the data term over views and scales according to local linearisation validity (Gray et al., 2024).

The scope is also specific. The paper develops GCM concretely for multi-view disparity estimation with coplanar cameras, identical camera orientation, and linear coupling of per-view displacement to a scalar disparity field. It states that the idea can generalize, but $\mathbf{B}_t$4, $\mathbf{B}_t$5, and $\mathbf{B}_t$6 must be adapted to the application. Additional limitations given in the paper are that the derivation is only analytically inspired rather than a full exact probabilistic model of all nonlinear effects; the treatment of gradient inconsistency is approximate; gradient consistency may fail as an indicator where gradients are weak, ambiguous, or altered by strong non-Lambertian effects; brightness constancy plus first-order linearisation remain the underlying model; and simultaneous multiscale use can create practical memory pressure, which is why the Middlebury experiments use a sliding window of active scales rather than all scales at once (Gray et al., 2024).

The acronym GCM is not unique in arXiv usage. In adversarial robustness it can denote Gradient Concealment Module (Pei et al., 2022); in mathematical relativity it denotes generally covariant modulated (Klainerman et al., 2019); and in atmospheric modeling it can denote general circulation model (Belochitski et al., 2021). Within the disparity-estimation literature considered here, however, Gradient Consistency Model has the specific meaning introduced above: a self-scheduling weighting model that uses gradient mismatch, scale inconsistency, and acquisition noise to determine how strongly each view and scale should contribute to a variational disparity solver.