Flow Fields: Dense Correspondence Fields for Highly Accurate Large Displacement Optical Flow Estimation (1508.05151v2)

Abstract: Modern large displacement optical flow algorithms usually use an initialization by either sparse descriptor matching techniques or dense approximate nearest neighbor fields. While the latter have the advantage of being dense, they have the major disadvantage of being very outlier prone as they are not designed to find the optical flow, but the visually most similar correspondence. In this paper we present a dense correspondence field approach that is much less outlier prone and thus much better suited for optical flow estimation than approximate nearest neighbor fields. Our approach is conceptually novel as it does not require explicit regularization, smoothing (like median filtering) or a new data term, but solely our novel purely data based search strategy that finds most inliers (even for small objects), while it effectively avoids finding outliers. Moreover, we present novel enhancements for outlier filtering. We show that our approach is better suited for large displacement optical flow estimation than state-of-the-art descriptor matching techniques. We do so by initializing EpicFlow (so far the best method on MPI-Sintel) with our Flow Fields instead of their originally used state-of-the-art descriptor matching technique. We significantly outperform the original EpicFlow on MPI-Sintel, KITTI and Middlebury.

• The paper presents a novel data-based search strategy that minimizes outliers without needing explicit regularization.
• It employs a hierarchical matching approach to reliably capture large displacements while preserving intricate image details.
• Experimental results on MPI-Sintel, KITTI, and Middlebury show significantly lower endpoint errors compared to existing methods.

Overview of "Flow Fields: Dense Correspondence Fields for Highly Accurate Large Displacement Optical Flow Estimation"

The paper presents a dense correspondence field approach optimized for large displacement optical flow estimation, addressing the limitations of existing methods like approximate nearest neighbor fields (ANNFs) and descriptor matching techniques. The proposed Flow Fields technique significantly reduces outliers without explicit regularization or smoothing, relying instead on a novel data-based search strategy. This advancement results in more accurate optical flow estimation, demonstrated through superior performance on benchmark datasets such as MPI-Sintel, KITTI, and Middlebury.

Contributions and Methodology

1. Data-Based Search Strategy: The paper introduces a unique search strategy that finds the most accurate inliers while efficiently avoiding the detection of outliers. Unlike traditional methods, it does not require explicit data term modifications or the application of smoothing filters.
2. Hierarchical Matching Approach: Flow Fields employ a hierarchical correspondence field search strategy. This approach is characterized by non-locality in the image space, which allows for higher fidelity in preserving small objects and intricate details, addressing a common shortcoming of coarse-to-fine strategies used in optical flow estimation.
3. Enhanced Outlier Filtering: The paper presents advanced techniques for filtering out outliers, including a two-way consistency check and region-based filtering methods, to refine the accuracy of optical flow predictions.
4. Robustness and Computational Efficiency: Flow Fields are designed to be computationally efficient, applicable with standard optical flow algorithms like EpicFlow, resulting in significant performance improvements while maintaining efficiency.

Experimental Results

The paper highlights the performance of Flow Fields relative to state-of-the-art methods. In the tests conducted on challenging datasets like MPI-Sintel and KITTI, Flow Fields result in lower endpoint errors (EPE) and better handling of large displacements compared to ANNF and other descriptor-based techniques. The results underscore:

• MPI-Sintel: Flow Fields demonstrate a marked reduction in EPE, particularly for large motion regions, outperforming other methods in both the clean and final datasets.
• KITTI: The approach ranks among the top-performing methods for estimating optical flow in real-world urban scenes, particularly excelling in the detection of large displacements and maintaining good performance under occlusions.
• Middlebury: The capabilities of Flow Fields are evident in improving results over traditional single-scale methods, demonstrating notable robustness in dense optical flow tasks.

Implications and Future Directions

The implications of this research are significant for the development of improved optical flow estimation techniques, particularly in scenarios where large motions and complex textures pose challenges. The approach's superiority in detail preservation and outlier management makes it a valuable tool for applications in video analysis, autonomous vehicles, and advanced vision systems requiring precise motion detection.

Future research could explore integration with more advanced data terms beyond traditional descriptors like SIFT, potentially incorporating machine learning approaches for feature extraction and matching. Moreover, the scalability of Flow Fields could be examined for real-time applications and extended to other related domains, such as stereo matching and scene reconstruction.

Overall, the paper provides substantial evidence that a data-based search strategy, coupled with a sophisticated outlier filtering mechanism, can significantly enhance the accuracy and reliability of optical flow estimation, setting a new standard for future research in this domain.