RealFlow: EM-based Realistic Optical Flow Dataset Generation from Videos (2207.11075v1)

Abstract: Obtaining the ground truth labels from a video is challenging since the manual annotation of pixel-wise flow labels is prohibitively expensive and laborious. Besides, existing approaches try to adapt the trained model on synthetic datasets to authentic videos, which inevitably suffers from domain discrepancy and hinders the performance for real-world applications. To solve these problems, we propose RealFlow, an Expectation-Maximization based framework that can create large-scale optical flow datasets directly from any unlabeled realistic videos. Specifically, we first estimate optical flow between a pair of video frames, and then synthesize a new image from this pair based on the predicted flow. Thus the new image pairs and their corresponding flows can be regarded as a new training set. Besides, we design a Realistic Image Pair Rendering (RIPR) module that adopts softmax splatting and bi-directional hole filling techniques to alleviate the artifacts of the image synthesis. In the E-step, RIPR renders new images to create a large quantity of training data. In the M-step, we utilize the generated training data to train an optical flow network, which can be used to estimate optical flows in the next E-step. During the iterative learning steps, the capability of the flow network is gradually improved, so is the accuracy of the flow, as well as the quality of the synthesized dataset. Experimental results show that RealFlow outperforms previous dataset generation methods by a considerably large margin. Moreover, based on the generated dataset, our approach achieves state-of-the-art performance on two standard benchmarks compared with both supervised and unsupervised optical flow methods. Our code and dataset are available at https://github.com/megvii-research/RealFlow

• The paper introduces an iterative EM-based framework that alternates between dataset generation using Realistic Image Pair Rendering and model training to improve optical flow estimation.
• The experimental results demonstrate that networks like RAFT achieve state-of-the-art performance on benchmarks such as KITTI and Sintel when trained on RealFlow-generated data.
• Ablation studies validate the necessity of components like depth maps and advanced splatting techniques, confirming their role in mitigating image synthesis artifacts.

A Critical Analysis of "RealFlow: EM-based Realistic Optical Flow Dataset Generation from Videos"

The paper "RealFlow: EM-based Realistic Optical Flow Dataset Generation from Videos" by Han et al. introduces an innovative Expectation-Maximization (EM)-based framework aimed at autonomously generating large-scale optical flow datasets directly from unlabeled real-world videos. This paper addresses the significant issue of domain adaptation in optical flow estimation by proposing a method that synthesizes optical flow training data from real-world video streams, thereby seeking to bridge the domain gap between synthetic training datasets and real-world application scenarios.

At the core of RealFlow is the iterative EM-based framework that alternates between two steps: dataset generation (E-step) and model training (M-step). During the E-step, the authors employ their Realistic Image Pair Rendering (RIPR) technique to synthesize new training pairs from video frames, leveraging estimated optical flow and depth maps. RIPR incorporates techniques such as softmax splatting and bi-directional hole filling to mitigate artifacts commonly associated with image synthesis. The synthesized data are subsequently used in the M-step to iteratively update the optical flow network, thereby enhancing its accuracy and performance.

The experimental results highlight the superiority of RealFlow compared to existing dataset generation methods like Depthstillation, which relies on single image-based generation and random transformations. RealFlow-generated datasets reportedly allow supervised networks, such as RAFT, to achieve state-of-the-art performance on established benchmarks like KITTI and Sintel, clearly illustrating the practical benefits of dataset realism in terms of motion representation and scene authenticity. Furthermore, the paper compares RealFlow against unsupervised optical flow estimation methods, revealing that networks trained on RealFlow-generated data surpass the performance of state-of-the-art unsupervised approaches.

Crucially, the robustness of RealFlow is also validated by training various optical flow networks on datasets generated by this framework, indicating its versatility and potential to significantly enhance supervised optical flow estimation across different architectures. The ablation studies substantiate the framework’s architectural choices and parameter settings, affirming the necessity of each component, such as the use of depth maps and specific splatting techniques.

In terms of implications, RealFlow sets a precedent for utilizing real-world videos to produce labeled datasets that can significantly improve the robustness and accuracy of optical flow models. This advancement reduces dependence on synthetic datasets, which traditionally suffer from the inability to capture the intricacies of real-world motion dynamics comprehensively.

The potential future trajectories suggested by this research involve expanding RealFlow applications beyond optical flow to other computer vision tasks requiring extensive labeled data. Furthermore, the exploration of additional video sources and the integration of advanced depth estimation techniques may further enhance dataset quality.

In summary, RealFlow presents a sophisticated blend of dataset generation and model learning, promising to mitigate the domain gap in optical flow estimation. By effectively harnessing real-world video data, this framework advances the field towards more authentic representations of motion, potentially catalyzing progress in a variety of computer vision applications reliant on accurate motion analysis.