MR-HS Optical Flow Estimation

• The method extends the classical Horn–Schunck technique with a multiresolution pyramid to effectively handle large displacements and avoid local minima.
• It employs a variational formulation with Gauss–Seidel iterations and bilinear interpolation for subpixel optical flow refinement.
• Empirical evaluations show significant improvements, reducing average angular error by ~23% and end-point error by ~29% over standard methods.

The Multiresolution Horn–Schunck (MR-HS) method is an extension of the classical Horn–Schunck framework for optical flow estimation, incorporating a coarse-to-fine multiresolution pyramid and bilinear interpolation strategies to enhance accuracy and convergence. MR-HS targets the recovery of dense motion fields between consecutive image frames by minimizing a global energy functional, where the multiresolution design addresses challenges posed by large displacements and local minima, especially under varying image conditions (Ziani, 20 Nov 2025).

1. Horn–Schunck Variational Formulation

The foundational principle of MR-HS is the original Horn–Schunck variational method, which seeks a dense optical flow field $(u(x,y), v(x,y))$ that minimizes the following energy:

$$E(u,v) = \int_\Omega (I_x\,u + I_y\,v + I_t)^2 + \alpha \left( |\nabla u|^2 + |\nabla v|^2 \right)\,dx\,dy,$$

where $I_x$, $I_y$, and $I_t$ are the spatial and temporal derivatives of the image intensity, and $\alpha > 0$ is a regularization parameter controlling the balance between data fidelity and smoothness. The Euler–Lagrange equations for this minimization yield the coupled elliptic PDEs:

$$I_x (I_x u + I_y v + I_t) - \alpha\,\nabla^2 u = 0,$$

$$I_y (I_x u + I_y v + I_t) - \alpha\,\nabla^2 v = 0.$$

Spatial gradients are discretized using 3×3 Sobel masks, while the Laplacian $\nabla^2 u$ is approximated by subtracting $u$ from its 5×5 Gaussian-smoothed local average.

2. Iterative Update and Single-Level Scheme

At each iteration and pixel $$E(u,v) = \int_\Omega (I_x\,u + I_y\,v + I_t)^2 + \alpha \left( |\nabla u|^2 + |\nabla v|^2 \right)\,dx\,dy,$$0, local Gaussian averages $$E(u,v) = \int_\Omega (I_x\,u + I_y\,v + I_t)^2 + \alpha \left( |\nabla u|^2 + |\nabla v|^2 \right)\,dx\,dy,$$1, $$E(u,v) = \int_\Omega (I_x\,u + I_y\,v + I_t)^2 + \alpha \left( |\nabla u|^2 + |\nabla v|^2 \right)\,dx\,dy,$$2 are computed. The updates proceed via the Gauss–Seidel method:

$$E(u,v) = \int_\Omega (I_x\,u + I_y\,v + I_t)^2 + \alpha \left( |\nabla u|^2 + |\nabla v|^2 \right)\,dx\,dy,$$3

$$E(u,v) = \int_\Omega (I_x\,u + I_y\,v + I_t)^2 + \alpha \left( |\nabla u|^2 + |\nabla v|^2 \right)\,dx\,dy,$$4

where $$E(u,v) = \int_\Omega (I_x\,u + I_y\,v + I_t)^2 + \alpha \left( |\nabla u|^2 + |\nabla v|^2 \right)\,dx\,dy,$$5 regularizes small denominators. Over-relaxation is optional, but the reference implementation uses straightforward Gauss–Seidel updates. Convergence is monitored via the sum of L2-norms $$E(u,v) = \int_\Omega (I_x\,u + I_y\,v + I_t)^2 + \alpha \left( |\nabla u|^2 + |\nabla v|^2 \right)\,dx\,dy,$$6 compared to a threshold $$E(u,v) = \int_\Omega (I_x\,u + I_y\,v + I_t)^2 + \alpha \left( |\nabla u|^2 + |\nabla v|^2 \right)\,dx\,dy,$$7.

3. Multiresolution Framework

MR-HS utilizes a Gaussian pyramid for both input images. Let $$E(u,v) = \int_\Omega (I_x\,u + I_y\,v + I_t)^2 + \alpha \left( |\nabla u|^2 + |\nabla v|^2 \right)\,dx\,dy,$$8 be the number of levels. Pyramidal construction is performed by iteratively applying Gaussian filtering and decimation by a factor of two:

$$E(u,v) = \int_\Omega (I_x\,u + I_y\,v + I_t)^2 + \alpha \left( |\nabla u|^2 + |\nabla v|^2 \right)\,dx\,dy,$$9

At each level, the optical flow is initialized and refined. Flow fields estimated at coarser levels are prolongated to finer levels via bilinear interpolation:

$I_x$0

where $I_x$1 is the integer part and $I_x$2 the fractional part of $I_x$3. Prolongated flows are scaled by 2 to account for increased resolution.

4. Level Integration and Algorithmic Structure

The MR-HS algorithm integrates estimates across resolution levels:

1. Construct pyramids for $I_x$4 and $I_x$5.
2. Initialize the flow $I_x$6 at the coarsest level ($I_x$7).
3. For levels $I_x$8:
   • If $I_x$9, upsample flow from level $I_y$0.
   • Warp $I_y$1 toward $I_y$2 using upsampled flow.
   • Recompute $I_y$3 at level $I_y$4.
   • Run single-level Horn–Schunck for up to the maximum iterations or until convergence (change below $I_y$5).
4. Output full-resolution flow $I_y$6.

Warping is performed by mapping pixels in $I_y$7 according to the upsampled optical flow, with bilinear interpolation used to sample subpixel positions.

5. Parameter Choices and Empirical Performance

Parameter settings reported include:

• Regularization $I_y$8, typical values chosen empirically.
• Number of pyramid levels: $I_y$9 or 4 ($I_t$0 for Alley_1, Bamboo_2, Mountain_1; $I_t$1 for Market_2).
• Maximum iterations per level: up to 5,000 (typically 100–300 suffice at each level, fewer in coarser levels due to rapid convergence).
• Convergence threshold $I_t$2.
• Small denominator buffer $I_t$3.

The MR-HS method demonstrates a reduction in both average angular error (AAE) and end-point error (EPE) compared to the baseline Horn–Schunck, attaining $I_t$4 AAE and 1.54 px EPE on the MPI-Sintel “final” benchmark, versus $I_t$5 and 2.17 px for the standard method—a drop of approximately 23% and 29%, respectively.

Method	Average Angular Error	End-Point Error
Horn–Schunck	$I_t$6	2.17 px
MR-HS	$I_t$7	1.54 px

6. Implementation Details and Computational Complexity

• Boundary conditions: Edge value replication is used for both convolution and interpolation at image borders.
• Iteration halting: The process stops if the L2-norm of the flow change drops below $I_t$8 or the maximum iterations are reached.
• Computational expense: Each iteration is $I_t$9 for $\alpha > 0$0 pixels, resulting in overall complexity $\alpha > 0$1, with $\alpha > 0$2 the number of pyramid levels and $\alpha > 0$3 the iterations per level.

Multiresolution processing mitigates the risk of suboptimal minima and expedites convergence in practice by initially estimating large-scale motion at the lowest resolution and incrementally refining at higher resolutions.

7. Context and Significance

MR-HS synthesizes global variational regularization with hierarchical, coarse-to-fine computation, bridging classical global techniques and robust multiscale approaches. The integration of Gaussian smoothing for gradient computation, bilinear prolongation for subpixel flow transfer, and pyramidal image warping results in improved precision over single-scale Horn–Schunck, particularly for complex sequences with large motions or challenging lighting. Boundary handling, convergence scheduling, and complexity management are directly addressed, enabling application to realistic datasets under diverse conditions. The empirical results on challenging synthetic data underscore the practical value of multiresolution strategies within variational optical flow frameworks (Ziani, 20 Nov 2025).