Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness (1608.05842v1)

Abstract: Recently, convolutional networks (convnets) have proven useful for predicting optical flow. Much of this success is predicated on the availability of large datasets that require expensive and involved data acquisition and laborious la- beling. To bypass these challenges, we propose an unsuper- vised approach (i.e., without leveraging groundtruth flow) to train a convnet end-to-end for predicting optical flow be- tween two images. We use a loss function that combines a data term that measures photometric constancy over time with a spatial term that models the expected variation of flow across the image. Together these losses form a proxy measure for losses based on the groundtruth flow. Empiri- cally, we show that a strong convnet baseline trained with the proposed unsupervised approach outperforms the same network trained with supervision on the KITTI dataset.

• The paper introduces an unsupervised framework combining photometric and smoothness loss functions to predict optical flow without reliance on groundtruth annotations.
• It leverages a convolutional encoder-decoder with skip connections to capture hierarchical flow features and preserve spatial details, achieving competitive performance on KITTI.
• The method addresses data scarcity in real-world scenarios, offering efficient and real-time optical flow estimation beneficial for applications like autonomous driving.

Overview of "Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness"

The paper "Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness", authored by Jason J. Yu, Adam W. Harley, and Konstantinos G. Derpanis, proposes an unsupervised approach to train convolutional neural networks for predicting optical flow between image pairs. The emphasis of this research is the elimination of the need for large datasets with groundtruth flow annotations, which are typically difficult to obtain, thereby addressing a key bottleneck in optical flow estimation in real-world scenarios.

Core Contributions and Methodology

1. Unsupervised Learning Framework: The research introduces an unsupervised learning framework leveraging a combination of photometric and smoothness loss functions. The photometric loss evaluates the consistency of pixel values across images, while the smoothness loss ensures coherency of motion in the predicted flow field. This methodology bypasses the requirement of groundtruth flow data that a supervised approach would depend on.
2. Optimization Objective: By using RGB image pairs, the model predicts optical flow in the form of horizontal and vertical displacements for each pixel. The loss function comprises a photometric term (ensuring brightness constancy) and a smoothness term. The photometric term is calculated by warping one image to align with the other using predicted flow and measuring discrepancies. Meanwhile, the smoothness term measures disparity in flow predictions between spatially neighboring pixels.
3. Network Architecture: The architecture consists of a contractive encoder and an expansive decoder, which employs a "skip-layer" mechanism to incorporate detailed information from the encoder into the decoding phase. This design enables effective learning of hierarchical flow features while maintaining high spatial resolution.

Empirical Results

The unsupervised method demonstrated competitive performance on the KITTI dataset compared to supervised methods. Notably, the approach achieved improvements on non-occluded regions relative to the baseline supervised models. This indicates the model's ability to generalize from raw observational data without requiring groundtruth annotations during training, particularly in practical scenarios such as automotive applications where large annotated datasets are scarce.

The evaluations include tests against datasets like the synthetic "Flying Chairs" and real-world "KITTI 2012", supplemented by several robust data augmentation techniques to improve model generalization. Interestingly, even though the fully supervised FlowNet performed better on the highly controlled "Flying Chairs" dataset, the unsupervised method showed superior or comparable performance in real-world scenarios with KITTI data.

Implications and Future Directions

The unsupervised methodology proposed in this research presents several significant implications for the field of computer vision. It can potentially enable broader applicability of optical flow estimation in domains with limited annotated data. The proposed network is also efficient, operating in real time on GPUs, which is advantageous for applications requiring low-latency processing, such as autonomous driving and video analysis.

Future work could delve into augmenting the loss functions with additional spatial and temporal constraints, or integrating this framework with other unsupervised learning approaches for related tasks such as depth perception or stereo vision. There is the prospect of utilizing more sophisticated network architectures or loss formulations to push the boundaries of performance further, especially in scenes with complex motion and lighting conditions. Overall, this paper lays the groundwork for significant advancements in unsupervised learning for optical flow and related visual motion estimation tasks.