EpicFlow: Edge-Preserving Interpolation of Correspondences for Optical Flow (1501.02565v2)

Abstract: We propose a novel approach for optical flow estimation , targeted at large displacements with significant oc-clusions. It consists of two steps: i) dense matching by edge-preserving interpolation from a sparse set of matches; ii) variational energy minimization initialized with the dense matches. The sparse-to-dense interpolation relies on an appropriate choice of the distance, namely an edge-aware geodesic distance. This distance is tailored to handle occlusions and motion boundaries -- two common and difficult issues for optical flow computation. We also propose an approximation scheme for the geodesic distance to allow fast computation without loss of performance. Subsequent to the dense interpolation step, standard one-level variational energy minimization is carried out on the dense matches to obtain the final flow estimation. The proposed approach, called Edge-Preserving Interpolation of Correspondences (EpicFlow) is fast and robust to large displacements. It significantly outperforms the state of the art on MPI-Sintel and performs on par on Kitti and Middlebury.

• The paper introduces EpicFlow, which significantly improves optical flow estimation by leveraging edge-aware interpolation to handle occlusions and large displacements.
• It employs a two-step process that first generates sparse correspondences with advanced matching algorithms and then refines dense flow using variational energy minimization.
• Experimental results on datasets like MPI-Sintel, Kitti, and Middlebury demonstrate its superior performance, achieving lower endpoint errors and efficient run times.

EpicFlow: Edge-Preserving Interpolation of Correspondences for Optical Flow

The paper presents a novel approach to optical flow estimation relying on edge-preserving interpolation, termed EpicFlow. This method targets large displacements and significant occlusions commonly encountered in real-world videos. EpicFlow operates through a two-step process: interpolating dense optical flow from a sparse set of matches using an edge-aware geodesic distance and refining the flow using variational energy minimization. The proposed techniques demonstrate significant improvements over traditional coarse-to-fine methods, especially in handling occlusions and motion boundaries.

Methodology

EpicFlow's methodology is structured around the utilization of edge-aware geodesic distance to interpolate dense optical flow. The key innovations can be summarized as follows:

1. Sparse Matching: Initial correspondences are generated using state-of-the-art matching algorithms such as DeepMatching. These matches are pruned through saliency checks and consistency verifications to remove outliers.
2. Edge-Aware Geodesic Distance: To handle occlusions and maintain motion boundaries, the Euclidean distance is replaced with an edge-aware geodesic distance. This distance ensures that interpolation respects image edges, which often coincide with motion boundaries.
3. Fast Approximation: The paper introduces an efficient approximation of the geodesic distance, enabling rapid computation without significant performance loss. This approximation leverages a geodesic Voronoi diagram to limit the computational load.
4. Variational Refinement: Once an initial dense flow field is obtained through interpolation, the method applies a one-level variational energy minimization to refine the flow. This process avoids the error propagation issues observed in coarse-to-fine schemes.

Results and Implications

The experimental results show that EpicFlow significantly outperforms existing methods on challenging datasets such as MPI-Sintel, Kitti, and Middlebury. Key performance metrics include:

• Performance on MPI-Sintel: EpicFlow achieves an average endpoint error (AEE) of 6.285, surpassing other leading methods by approximately 0.5 to 1 pixel.
• Performance on Kitti: It achieves the lowest AEE on non-occluded areas (1.5) and is competitive across other metrics.
• Performance on Middlebury: On this dataset with relatively small displacements, EpicFlow performs competitively with an AEE below 0.4 pixels.

The improvement is particularly notable in occluded regions and large displacement scenarios. The use of an edge-aware distance crucially contributes to maintaining motion boundaries and improving accuracy.

Theoretical and Practical Implications

From a theoretical perspective, the paper challenges the reliance on the coarse-to-fine framework, demonstrating that a robust interpolation strategy combined with variational refinement can yield superior results. The edge-aware geodesic distance plays a pivotal role in addressing the inherent weaknesses of coarse-to-fine methods in handling motion discontinuities and large displacements.

Practically, EpicFlow demonstrates faster computation times compared to other high-performing methods, which enhances its applicability in real-world scenarios demanding timely results.

Future Directions

Future research could focus on further improving the quality and density of initial matches and enhancing contour detection methods. Integrating real-time edge detection and matching updates could also provide more dynamic and robust optical flow estimations in evolving scenes. Moreover, exploring the integration of EpicFlow into broader computer vision systems, such as autonomous driving frameworks or real-time video processing, could underscore its utility and inspire additional optimizations.

Conclusion

EpicFlow presents a substantial advancement in optical flow estimation by leveraging edge-preserving interpolation guided by geodesic distances. Its robust handling of large displacements and occlusions, combined with significant improvements in accuracy and computation efficiency, establishes it as a high-performing alternative to traditional methods. The presented approach encourages a reevaluation of coarse-to-fine paradigms and opens new avenues for high-accuracy, real-world optical flow applications.