FlowNet: Learning Optical Flow with Convolutional Networks (1504.06852v2)

Abstract: Convolutional neural networks (CNNs) have recently been very successful in a variety of computer vision tasks, especially on those linked to recognition. Optical flow estimation has not been among the tasks where CNNs were successful. In this paper we construct appropriate CNNs which are capable of solving the optical flow estimation problem as a supervised learning task. We propose and compare two architectures: a generic architecture and another one including a layer that correlates feature vectors at different image locations. Since existing ground truth data sets are not sufficiently large to train a CNN, we generate a synthetic Flying Chairs dataset. We show that networks trained on this unrealistic data still generalize very well to existing datasets such as Sintel and KITTI, achieving competitive accuracy at frame rates of 5 to 10 fps.

• The paper introduces two novel CNN architectures—FlowNetSimple and FlowNetCorr—that directly estimate optical flow from image pairs.
• It leverages a large synthetic dataset, Flying Chairs, to overcome training data limitations and outperforms state-of-the-art methods.
• Post-processing with variational refinement further improves accuracy, illustrating the adaptability of CNNs to real-world scenarios.

FlowNet: Learning Optical Flow with Convolutional Networks

The paper “FlowNet: Learning Optical Flow with Convolutional Networks” authored by Philipp Fischer, Alexey Dosovitskiy, Eddy Ilg, Philip Häusser, Caner Hazırbaş, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox, explores the viability of Convolutional Neural Networks (CNNs) in estimating the optical flow from pairs of images. Optical flow represents the pattern of apparent motion of objects, surfaces, and edges in a visual scene, caused by the relative movement between an observer and the scene, and is typically a challenging task requiring precise per-pixel localization and correspondence matching between images.

Methodology

The authors propose two CNN architectures designed for optical flow estimation: FlowNetSimple and FlowNetCorr. These architectures differ primarily in how they handle the task of matching image features.

• FlowNetSimple: This architecture directly stacks the input image pair and processes them through generic convolutional layers.
• FlowNetCorr: This architecture introduces a correlation layer which explicitly computes matching costs between the feature maps of the two images, aiding in correspondence matching.

Both architectures heavily utilize convolutional layers with the following characteristics:

1. Nine convolutional layers with a ReLU activation function following each convolution.
2. Decreasing filter sizes towards deeper layers, starting with $7 \times 7$ and ending with $3 \times 3$.
3. Increased number of feature maps in deeper layers, approximately doubling after each layer with a stride of 2.

Synthetic Data for Training

A critical challenge in training CNNs for optical flow is the availability of sufficiently large labeled datasets. Existing datasets such as Middlebury, KITTI, and MPI Sintel, although useful, are limited in size for training deep networks effectively. To address this, a large synthetic dataset named Flying Chairs was introduced with 22,872 image pairs. This dataset comprises random background images with overlaid rendered chairs, transformed with complex affine motions to simulate varying displacements akin to real-world scenarios. The use of synthetic data is justified by the CNNs’ ability to generalize well to existing real-world datasets without aggressive overfitting.

Results

The networks’ performances were evaluated on several optical flow datasets, observing the following:

1. Flying Chairs: Both FlowNetSimple and FlowNetCorr showed excellent performance, even outperforming state-of-the-art methods such as EpicFlow and DeepFlow.
2. Sintel and KITTI: FlowNetS demonstrated superior generalization on Sintel Final, while FlowNetCorr excelled on Sintel Clean. Fine-tuning on Sintel data improved performance significantly on KITTI datasets, highlighting the networks' adaptability to better quality training data.

Additional insights:

• Variational Refinement: A post-processing step was proposed to smooth and refine the outputs further using variational methods which improved the results in real world datasets.
• Generalization: Despite being trained on synthetic data, the networks generalized well to real-world datasets. This observation is particularly intriguing as it implies that the networks learned robust features transferable to real-world scenarios.

Implications and Future Developments

The results indicate that CNNs could revolutionize the way optical flow is estimated, leveraging their ability to learn complex representations directly from data. This research opens several avenues:

• Real-world Data Augmentation: Enhancing available datasets with more realistic synthetic data could further push the boundaries of what these networks can achieve.
• Network Architectures: Exploring alternative architectures that integrate domain-specific knowledge or employ attention mechanisms could lead to even better performance.
• Application Domains: The fast execution times make these networks viable for real-time applications in robotics, autonomous driving, and video processing.

Conclusions

"FlowNet: Learning Optical Flow with Convolutional Networks" provides a significant contribution by demonstrating that CNNs can effectively predict optical flow and do so with competitive accuracy and speed. The introduction of new architectures and synthetic datasets underlines the potential of neural networks to handle traditionally challenging computer vision tasks efficiently. Future research can build on these findings to develop even more sophisticated models and methods that enhance both the theoretical understanding and practical applications in the field of optical flow estimation.