---
title: Multiresolution Horn–Schunck Framework
url: https://www.emergentmind.com/topics/multiresolution-horn-schunck-framework
type: topic
---

# Multiresolution Horn–Schunck Framework

The multiresolution Horn–Schunck (MR-HS) framework is a coarse-to-fine optical flow estimation methodology for sequential image analysis. It extends the classical global variational Horn–Schunck approach by integrating a pyramid-based scale-space decomposition and bilinear interpolation for inter-level flow propagation. This structure enables robust, accurate motion estimation—even in scenarios with large displacements or challenging image conditions—by progressively refining flow fields from coarse to fine spatial resolutions [2511.16535].

## 1. Pyramid-Based Image Decomposition

The MR-HS framework relies on constructing a Gaussian (or equivalently Laplacian) image pyramid for both input frames \(I_t\) and \(I_{t+1}\). For images of size \(W \times H\), a sequence of \(L+1\) images \(\{I^0, I^1, \dots, I^L\}\) is built, where

- \(I^0\) is the original (finest) image,
- Each subsequent level (\(\ell=1,\dots,L\)) is formed by Gaussian convolution with kernel \(g\) (e.g., \(3 \times 3\), σ ≈ 1) followed by decimation (downsampling by 2).

This is implemented by the operation:
\[
I^\ell(x,y) = \sum_{i=-1}^{+1}\sum_{j=-1}^{+1} g(i,j)\, I^{\ell-1}(2x+i, 2y+j)
\]
Alternatively, smoothing and downsampling are often combined via 2×2 bilinear interpolation:
\[
I^\ell(x, y) = \sum_{m=0}^1\sum_{n=0}^1 w_{mn}(u, v)\, I^{\ell-1}(2x+m, 2y+n)
\]
where \(u = 2x - \lfloor 2x \rfloor\), \(v = 2y - \lfloor 2y \rfloor\), and \(w_{mn}\) are classic bilinear weights.

At the coarsest level (\(L\)), the optical flow fields \(u^L\), \(v^L\) are initialized to zero.

## 2. Coarse-to-Fine Flow Estimation Strategy

Optical flow refinement proceeds hierarchically from the coarsest (smallest) resolution to the finest (full) resolution:

1. **Prolongation:** The estimated flow field \((u^\ell, v^\ell)\) at level \(\ell\) is upsampled to level \(\ell-1\) using bilinear interpolation. The magnitude is doubled to account for the spatial scaling:
   \[
   u^{\ell-1}_0(x, y) = 2 \sum_{m=0}^1\sum_{n=0}^1 w_{mn}(\tfrac{x}{2} - i, \tfrac{y}{2} - j)\, u^\ell(i + m, j + n)
   \]
   with analogous treatment for \(v\), where \((i, j) = (\lfloor x/2 \rfloor, \lfloor y/2 \rfloor)\).

2. **Warping:** The finer-level \(I^{\ell-1}_{t+1}\) is warped towards \(I^{\ell-1}_t\) using the upsampled flow to account for currently estimated motion, reducing bias from large displacements.

3. **Initialization and Refinement:** The upsampled flow initializes the Horn–Schunck solver at the next finer level. The energy functional minimized per level is:
   \[
   E_\ell(u, v) = \iint \left(I_x u + I_y v + I_t\right)^2 + \alpha^2\left(\|\nabla u\|^2 + \|\nabla v\|^2\right) \, dx\, dy
   \]
   Spatial derivatives (\(I_x, I_y\)) and temporal difference (\(I_t = I^{\ell-1}_{t+1} - I^{\ell-1}_t\)) are taken on the warped images.

## 3. Iterative Horn–Schunck Solver and Numerical Updates

At each level and refinement iteration, the MR-HS algorithm applies Gauss–Seidel (or Jacobi/SOR) updates based on local averages:

\[
\bar u_{i, j} = \frac{1}{5}\left(u^k_{i, j} + u^k_{i-1, j} + u^k_{i+1, j} + u^k_{i, j-1} + u^k_{i, j+1}\right)
\]
\[
\bar v_{i, j} = \frac{1}{5}\left(v^k_{i, j} + v^k_{i-1, j} + v^k_{i+1, j} + v^k_{i, j-1} + v^k_{i, j+1}\right)
\]
Given these, the flow is updated elementwise:
\[
u^{k+1}_{i, j} = \bar u_{i, j} - \frac{I_x(\bar u_{i, j} I_x + \bar v_{i, j} I_y + I_t)}{\alpha^2 + I_x^2 + I_y^2 + \varepsilon}
\]
\[
v^{k+1}_{i, j} = \bar v_{i, j} - \frac{I_y(\bar u_{i, j} I_x + \bar v_{i, j} I_y + I_t)}{\alpha^2 + I_x^2 + I_y^2 + \varepsilon}
\]
with \(\varepsilon \sim 10^{-5}\) for regularization. Iterations are performed until a prescribed convergence criterion (\(\|\Delta u\|_2 < \tau\)) or a set maximum number of steps.

## 4. Bilinear Interpolation and Boundary Considerations

All resampling—both image downsampling and flow upsampling—uses the standard 2×2 bilinear kernel:
\[
f(x, y) = \sum_{m=0}^1\sum_{n=0}^1 w_{mn}(a, b)\,f(i+m, j+n)
\]
where \(a = x-\lfloor x \rfloor\), \(b = y - \lfloor y \rfloor\), and the weights \(w_{mn}\) are as previously defined.

Boundary handling is implemented with either index clamping (border replication) or zero padding; practical implementation uses libraries such as OpenCV's INTER_LINEAR interpolant with cv2.BORDER_REPLICATE.

A key detail is that prolongation uses the same bilinear weights as downsampling but includes the scaling factor of 2 to correct flow magnitudes at each resolution.

## 5. Parameter Selection and Convergence Behavior

Critical parameters influencing MR-HS performance include:

- **Smoothness weight \(\alpha\):** Tunes the balance of data fidelity and smoothness, typical range \([0.5, 5]\); for Sintel data, \(\alpha \approx 1\) is effective.
- **Pyramid depth \(L\):** Chosen such that the coarsest grid contains at least \(20 \times 20\) pixels; for \(640 \times 384\) frames, values \(L = 3\) or \(4\) are common.
- **Iterations per level:** Fewer (50–100) for coarse levels, more (200–500) for finer resolutions; typical convergence occurs after a few hundred iterations monitored via the \(\ell_2\) norm of flow updates.

Convergence guarantees exist under small-displacement and convexity conditions (see Mitiche & Mansouri, 2004). Empirically, the coarse-to-fine initialization mitigates poor local minima by handling large displacements at successively finer scales.

## 6. Computational Complexity and Quantitative Performance

The iteration cost per level is \(\mathcal{O}(N_\ell)\), with \(N_\ell = (W/2^\ell) \times (H/2^\ell)\). Total computational effort is approximately:
\[
\sum_{\ell=0}^L I_\ell N_\ell \approx \frac{4}{3} I N
\]
assuming a constant iteration count \(I\) per level. Inclusion of warping and gradient computations maintains the overall scaling at \(\mathcal{O}(N \times L \times I)\), effectively linear in the number of pixels with a small pyramid factor.

Empirical evaluation on the MPI-Sintel benchmark ("final" pass) demonstrates:

| Scene       | HS (AAE/EPE)   | MR-HS (AAE/EPE) | Levels |
|-------------|----------------|-----------------|--------|
| Alley\_1    | 12.46°, 2.62   | 6.61°, 1.81     | 4      |
| Bamboo\_2   | 10.83°, 1.68   | 8.81°, 1.17     | 4      |
| Market\_2   | 19.08°, 0.47   | 15.31°, 0.41    | 3      |
| Mountain\_1 | 17.13°, 3.90   | 15.28°, 2.78    | 4      |
| **Average** | 14.88°, 2.17   | 11.50°, 1.54    | —      |

MR-HS provides approximately 25% reduction in endpoint error (EPE) and 23% reduction in angular error (AAE) relative to classic single-scale Horn–Schunck, with negligible added computational cost.

## 7. Algorithmic Workflow

The following pseudocode synthesizes the end-to-end MR-HS procedure, capturing all core algorithmic steps:

```python
Input: I_t, I_{t+1}, levels L, smoothness α, iters_per_level
Output: flow u^0,v^0 at full resolution

// 1. Build image pyramids
I^0_t = I_t;   I^0_{t+1} = I_{t+1}
for ℓ = 1 to L do
  I^ℓ_t     = downsample( I^{ℓ-1}_t )
  I^ℓ_{t+1} = downsample( I^{ℓ-1}_{t+1} )
end for

// 2. Initialize flow at coarsest level
u^L ← zeros( size(I^L) )
v^L ← zeros( size(I^L) )

// 3. Coarse-to-fine  
for ℓ = L down to 0 do
  if ℓ < L then
    // prolongate previous level’s flow
    (u^ℓ, v^ℓ) = bilinear_upsample( u^{ℓ+1}, v^{ℓ+1} ) × 2
  end if

  // warp I_{t+1} toward I_t under current flow
  I^ℓ_{t+1,w} = warp( I^ℓ_{t+1}, u^ℓ, v^ℓ )

  // compute image derivatives on (I^ℓ_t, I^ℓ_{t+1,w})
  [I_x, I_y] = spatial_gradients( I^ℓ_t )
  I_t = I^ℓ_{t+1,w} – I^ℓ_t

  // refine flow by Horn–Schunck iterations
  for k = 1 to iters_per_level[ℓ] do
    for each pixel (i,j) in scan order do
      compute local averages ȳu, ȳv
      update u^ℓ_{i,j}, v^ℓ_{i,j} by the HS formula above
    end for
    if convergence then break
  end for
end for

return (u^0, v^0)
```

This MR-HS pipeline combines spatial multiresolution, variational regularization, and bilinear prolongation, yielding a principled, scalable approach for optical flow computation suitable for diverse applications in computer vision [2511.16535].

Source: https://www.emergentmind.com/topics/multiresolution-horn-schunck-framework