Learning Optical Flow from a Few Matches (2104.02166v1)

Abstract: State-of-the-art neural network models for optical flow estimation require a dense correlation volume at high resolutions for representing per-pixel displacement. Although the dense correlation volume is informative for accurate estimation, its heavy computation and memory usage hinders the efficient training and deployment of the models. In this paper, we show that the dense correlation volume representation is redundant and accurate flow estimation can be achieved with only a fraction of elements in it. Based on this observation, we propose an alternative displacement representation, named Sparse Correlation Volume, which is constructed directly by computing the k closest matches in one feature map for each feature vector in the other feature map and stored in a sparse data structure. Experiments show that our method can reduce computational cost and memory use significantly, while maintaining high accuracy compared to previous approaches with dense correlation volumes. Code is available at https://github.com/zacjiang/scv .

• The paper introduces the Sparse Correlation Volume to significantly reduce computational complexity and memory usage in optical flow estimation.
• It employs a k-nearest neighbors search on feature maps to retain only top-k matches, reducing the correlation volume from O(N²) to O(Nk).
• Experimental results confirm that the SCV maintains competitive accuracy, enabling efficient optical flow estimation on high-resolution images and in resource-constrained scenarios.

Learning Optical Flow from a Few Matches: An Overview

The paper "Learning Optical Flow from a Few Matches" provides a novel approach to optical flow estimation, which traditionally requires dense and computationally expensive correlation volumes. The authors, Jiang et al., present an alternative representation, known as the Sparse Correlation Volume (SCV), which reduces computational costs without compromising accuracy.

Optical flow estimation is a fundamental problem in computer vision, which involves determining the pixel-by-pixel motion between two images. Traditional methods often employ dense correlation volumes to capture per-pixel displacements, but these are computationally intensive and memory-demanding, inhibiting efficient model deployment and training on high-resolution images.

Sparse Correlation Volume

The central contribution of this work is the Sparse Correlation Volume, which relies on the observation that dense correlation volumes hold significant redundancy. By selecting the top-k closest matches in one feature map for each feature vector in another feature map, the authors construct a sparse data structure. This reduces the spatial complexity of the correlation volume from $O(N^2)$ to $O(Nk)$, where $N$ represents the number of pixels, and $k$ is a constant denoting top matches to retain. Experimental results indicate that the SCV can maintain high accuracy levels even when a limited subset of correlations is preserved.

Methodology

The proposed methodology involves using a feature extraction network to derive feature maps from paired images, followed by the application of a k-nearest neighbors (kNN) search to identify the top-k correlation scores for each feature vector. The established sparse correlation volume is then leveraged with algorithmic blocks designed to iteratively update and refine displacement vectors predicting residual flow. The network's architecture and the iterative refinement approach parallel those found in recent advancements, such as RAFT, but with a reduced memory footprint due to SCV's efficiency.

Implications and Future Directions

The introduction of SCV has several significant implications for the field of optical flow estimation and computer vision at large. By drastically reducing memory consumption while retaining performance, the SCV enables models to handle higher-resolution inputs, capturing finer image details that were previously unmanageable due to resource constraints. These advancements make it viable to deploy optical flow solutions in more resource-constrained environments, broadening the applicability of such methods.

This research opens up several avenues for future exploration. One area is optimizing the sparse correlation volume for different types of image sequences, potentially incorporating dynamic k-values or adaptive sparse representations based on scene complexity. Another prospect is integrating SCV with other vision tasks beyond optical flow, such as stereoscopic depth estimation or object tracking, where similar dense correlation grounds could benefit from sparsification.

In summary, the approach presented in this paper demonstrates a substantial improvement in the efficiency of optical flow estimation while maintaining comparable accuracy to existing dense methods, effectively balancing the trade-off between computation and precision. This balance is crucial for evolving how optical flow models are applied in practical scenarios, highlighting the importance of pursuing storage-efficient and scalable solutions in computer vision. Future research may well leverage these insights to extend sparse correlation techniques across broader domains, further democratizing access to state-of-the-art computer vision capabilities.