Multiresolution Horn–Schunck Framework

• The framework is a coarse-to-fine optical flow estimation method that uses a pyramid-based image decomposition and iterative Horn–Schunck updates.
• It employs bilinear interpolation and warping to propagate and refine flow estimates, effectively handling large displacements and noise.
• Empirical evaluations on the MPI-Sintel benchmark show approximately 25% reduction in endpoint error and 23% decrease in angular error versus the classic approach.

The multiresolution Horn–Schunck (MR-HS) framework is a coarse-to-fine optical flow estimation methodology for sequential image analysis. It extends the classical global variational Horn–Schunck approach by integrating a pyramid-based scale-space decomposition and bilinear interpolation for inter-level flow propagation. This structure enables robust, accurate motion estimation—even in scenarios with large displacements or challenging image conditions—by progressively refining flow fields from coarse to fine spatial resolutions (Ziani, 20 Nov 2025).

1. Pyramid-Based Image Decomposition

The MR-HS framework relies on constructing a Gaussian (or equivalently Laplacian) image pyramid for both input frames $I_t$ and $I_{t+1}$. For images of size $W \times H$, a sequence of $L+1$ images $\{I^0, I^1, \dots, I^L\}$ is built, where

• $I^0$ is the original (finest) image,
• Each subsequent level ($\ell=1,\dots,L$) is formed by Gaussian convolution with kernel $g$ (e.g., $3 \times 3$, σ ≈ 1) followed by decimation (downsampling by 2).

This is implemented by the operation: $$I^\ell(x,y) = \sum_{i=-1}^{+1}\sum_{j=-1}^{+1} g(i,j)\, I^{\ell-1}(2x+i, 2y+j)$$ Alternatively, smoothing and downsampling are often combined via 2×2 bilinear interpolation: $$I^\ell(x, y) = \sum_{m=0}^1\sum_{n=0}^1 w_{mn}(u, v)\, I^{\ell-1}(2x+m, 2y+n)$$ where $u = 2x - \lfloor 2x \rfloor$, $v = 2y - \lfloor 2y \rfloor$, and $w_{mn}$ are classic bilinear weights.

At the coarsest level ($L$), the optical flow fields $u^L$, $v^L$ are initialized to zero.

2. Coarse-to-Fine Flow Estimation Strategy

Optical flow refinement proceeds hierarchically from the coarsest (smallest) resolution to the finest (full) resolution:

1. Prolongation: The estimated flow field $(u^\ell, v^\ell)$ at level $\ell$ is upsampled to level $\ell-1$ using bilinear interpolation. The magnitude is doubled to account for the spatial scaling:

$$u^{\ell-1}_0(x, y) = 2 \sum_{m=0}^1\sum_{n=0}^1 w_{mn}(\tfrac{x}{2} - i, \tfrac{y}{2} - j)\, u^\ell(i + m, j + n)$$

with analogous treatment for $v$, where $$(i, j) = (\lfloor x/2 \rfloor, \lfloor y/2 \rfloor)$$.

1. Warping: The finer-level $I^{\ell-1}_{t+1}$ is warped towards $I^{\ell-1}_t$ using the upsampled flow to account for currently estimated motion, reducing bias from large displacements.
2. Initialization and Refinement: The upsampled flow initializes the Horn–Schunck solver at the next finer level. The energy functional minimized per level is:

$$E_\ell(u, v) = \iint \left(I_x u + I_y v + I_t\right)^2 + \alpha^2\left(\|\nabla u\|^2 + \|\nabla v\|^2\right) \, dx\, dy$$

Spatial derivatives ($I_x, I_y$) and temporal difference ($I_t = I^{\ell-1}_{t+1} - I^{\ell-1}_t$) are taken on the warped images.

3. Iterative Horn–Schunck Solver and Numerical Updates

At each level and refinement iteration, the MR-HS algorithm applies Gauss–Seidel (or Jacobi/SOR) updates based on local averages:

$$\bar u_{i, j} = \frac{1}{5}\left(u^k_{i, j} + u^k_{i-1, j} + u^k_{i+1, j} + u^k_{i, j-1} + u^k_{i, j+1}\right)$$

$$\bar v_{i, j} = \frac{1}{5}\left(v^k_{i, j} + v^k_{i-1, j} + v^k_{i+1, j} + v^k_{i, j-1} + v^k_{i, j+1}\right)$$

Given these, the flow is updated elementwise: $$u^{k+1}_{i, j} = \bar u_{i, j} - \frac{I_x(\bar u_{i, j} I_x + \bar v_{i, j} I_y + I_t)}{\alpha^2 + I_x^2 + I_y^2 + \varepsilon}$$

$$v^{k+1}_{i, j} = \bar v_{i, j} - \frac{I_y(\bar u_{i, j} I_x + \bar v_{i, j} I_y + I_t)}{\alpha^2 + I_x^2 + I_y^2 + \varepsilon}$$

with $\varepsilon \sim 10^{-5}$ for regularization. Iterations are performed until a prescribed convergence criterion ($\|\Delta u\|_2 < \tau$) or a set maximum number of steps.

4. Bilinear Interpolation and Boundary Considerations

All resampling—both image downsampling and flow upsampling—uses the standard 2×2 bilinear kernel: $$f(x, y) = \sum_{m=0}^1\sum_{n=0}^1 w_{mn}(a, b)\,f(i+m, j+n)$$ where $a = x-\lfloor x \rfloor$, $b = y - \lfloor y \rfloor$, and the weights $w_{mn}$ are as previously defined.

Boundary handling is implemented with either index clamping (border replication) or zero padding; practical implementation uses libraries such as OpenCV's INTER_LINEAR interpolant with cv2.BORDER_REPLICATE.

A key detail is that prolongation uses the same bilinear weights as downsampling but includes the scaling factor of 2 to correct flow magnitudes at each resolution.

5. Parameter Selection and Convergence Behavior

Critical parameters influencing MR-HS performance include:

• Smoothness weight $\alpha$: Tunes the balance of data fidelity and smoothness, typical range $[0.5, 5]$; for Sintel data, $\alpha \approx 1$ is effective.
• Pyramid depth $L$: Chosen such that the coarsest grid contains at least $20 \times 20$ pixels; for $640 \times 384$ frames, values $L = 3$ or $4$ are common.
• Iterations per level: Fewer (50–100) for coarse levels, more (200–500) for finer resolutions; typical convergence occurs after a few hundred iterations monitored via the $\ell_2$ norm of flow updates.

Convergence guarantees exist under small-displacement and convexity conditions (see Mitiche & Mansouri, 2004). Empirically, the coarse-to-fine initialization mitigates poor local minima by handling large displacements at successively finer scales.

6. Computational Complexity and Quantitative Performance

The iteration cost per level is $\mathcal{O}(N_\ell)$, with $N_\ell = (W/2^\ell) \times (H/2^\ell)$. Total computational effort is approximately: $$\sum_{\ell=0}^L I_\ell N_\ell \approx \frac{4}{3} I N$$ assuming a constant iteration count $I$ per level. Inclusion of warping and gradient computations maintains the overall scaling at $\mathcal{O}(N \times L \times I)$, effectively linear in the number of pixels with a small pyramid factor.

Empirical evaluation on the MPI-Sintel benchmark ("final" pass) demonstrates:

Scene	HS (AAE/EPE)	MR-HS (AAE/EPE)	Levels
Alley_1	12.46°, 2.62	6.61°, 1.81	4
Bamboo_2	10.83°, 1.68	8.81°, 1.17	4
Market_2	19.08°, 0.47	15.31°, 0.41	3
Mountain_1	17.13°, 3.90	15.28°, 2.78	4
Average	14.88°, 2.17	11.50°, 1.54	—

MR-HS provides approximately 25% reduction in endpoint error (EPE) and 23% reduction in angular error (AAE) relative to classic single-scale Horn–Schunck, with negligible added computational cost.

7. Algorithmic Workflow

The following pseudocode synthesizes the end-to-end MR-HS procedure, capturing all core algorithmic steps:

Input: I_t, I_{t+1}, levels L, smoothness α, iters_per_level Output: flow u^0,v^0 at full resolution // 1. Build image pyramids I^0_t = I_t; I^0_{t+1} = I_{t+1} for ℓ = 1 to L do I^ℓ_t = downsample( I^{ℓ-1}_t ) I^ℓ_{t+1} = downsample( I^{ℓ-1}_{t+1} ) end for // 2. Initialize flow at coarsest level u^L ← zeros( size(I^L) ) v^L ← zeros( size(I^L) ) // 3. Coarse-to-fine for ℓ = L down to 0 do if ℓ < L then // prolongate previous level’s flow (u^ℓ, v^ℓ) = bilinear_upsample( u^{ℓ+1}, v^{ℓ+1} ) × 2 end if // warp I_{t+1} toward I_t under current flow I^ℓ_{t+1,w} = warp( I^ℓ_{t+1}, u^ℓ, v^ℓ ) // compute image derivatives on (I^ℓ_t, I^ℓ_{t+1,w}) [I_x, I_y] = spatial_gradients( I^ℓ_t ) I_t = I^ℓ_{t+1,w} – I^ℓ_t // refine flow by Horn–Schunck iterations for k = 1 to iters_per_level[ℓ] do for each pixel (i,j) in scan order do compute local averages ȳu, ȳv update u^ℓ_{i,j}, v^ℓ_{i,j} by the HS formula above end for if convergence then break end for end for return (u^0, v^0)

This MR-HS pipeline combines spatial multiresolution, variational regularization, and bilinear prolongation, yielding a principled, scalable approach for optical flow computation suitable for diverse applications in computer vision (Ziani, 20 Nov 2025).