Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences (2311.17099v1)

Published 28 Nov 2023 in cs.CV and cs.AI

Abstract: Occlusions between consecutive frames have long posed a significant challenge in optical flow estimation. The inherent ambiguity introduced by occlusions directly violates the brightness constancy constraint and considerably hinders pixel-to-pixel matching. To address this issue, multi-frame optical flow methods leverage adjacent frames to mitigate the local ambiguity. Nevertheless, prior multi-frame methods predominantly adopt recursive flow estimation, resulting in a considerable computational overlap. In contrast, we propose a streamlined in-batch framework that eliminates the need for extensive redundant recursive computations while concurrently developing effective spatio-temporal modeling approaches under in-batch estimation constraints. Specifically, we present a Streamlined In-batch Multi-frame (SIM) pipeline tailored to video input, attaining a similar level of time efficiency to two-frame networks. Furthermore, we introduce an efficient Integrative Spatio-temporal Coherence (ISC) modeling method for effective spatio-temporal modeling during the encoding phase, which introduces no additional parameter overhead. Additionally, we devise a Global Temporal Regressor (GTR) that effectively explores temporal relations during decoding. Benefiting from the efficient SIM pipeline and effective modules, StreamFlow not only excels in terms of performance on the challenging KITTI and Sintel datasets, with particular improvement in occluded areas but also attains a remarkable $63.82\%$ enhancement in speed compared with previous multi-frame methods. The code will be available soon at https://github.com/littlespray/StreamFlow.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. Starflow: A script-centric data analysis environment. In Provenance and Annotation of Data and Processes: Third International Provenance and Annotation Workshop, IPAW 2010, Troy, NY, USA, June 15-16, 2010. Revised Selected Papers 3, pages 236–250. Springer, 2010.
  2. Is space-time attention all you need for video understanding? In ICML, page 4, 2021.
  3. A naturalistic open source movie for optical flow evaluation. In European conference on computer vision, pages 611–625. Springer, 2012.
  4. Optical flow based visual potential field for autonomous driving. In 2020 IEEE Intelligent Vehicles Symposium (IV), pages 885–891. IEEE, 2020.
  5. Mfcflow: A motion feature compensated multi-frame recurrent network for optical flow estimation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5068–5077, 2023.
  6. Moving object tracking based on sparse optical flow with moving window and target estimator. Sensors, 22(8):2878, 2022.
  7. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural Information Processing Systems, 34:9355–9366, 2021.
  8. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015.
  9. Flowformer: A transformer architecture for optical flow. arXiv preprint arXiv:2203.16194, 2022.
  10. Liteflownet: A lightweight convolutional neural network for optical flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8981–8989, 2018.
  11. A lightweight optical flow cnn—revisiting data fidelity and regularization. IEEE transactions on pattern analysis and machine intelligence, 43(8):2555–2569, 2020.
  12. Iterative residual refinement for joint optical flow and occlusion estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5754–5763, 2019.
  13. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  14. Learning to estimate hidden motions with global motion aggregation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9772–9781, 2021.
  15. Optical flow estimation via motion feature recovery. In 2021 IEEE International Conference on Image Processing (ICIP), pages 2558–2562. IEEE, 2021.
  16. Moving object tracking using optical flow and motion vector estimation. In 2015 4th international conference on reliability, infocom technologies and optimization (ICRITO)(trends and future directions), pages 1–6. IEEE, 2015.
  17. The hci benchmark suite: Stereo and flow ground truth with uncertainties for urban autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 19–28, 2016.
  18. Deep contextual video compression. Advances in Neural Information Processing Systems, 34:18114–18125, 2021.
  19. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
  20. Dvc: An end-to-end deep video compression framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11006–11015, 2019.
  21. Transflow: Transformer as flow learner. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18063–18073, 2023.
  22. Learning optical flow with adaptive graph reasoning. arXiv preprint arXiv:2202.03857, 2022.
  23. Gaflow: Incorporating gaussian attention into optical flow. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9642–9651, 2023.
  24. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2016. arXiv:1512.02134.
  25. Joint 3d estimation of vehicles and scene flow. In ISPRS Workshop on Image Sequence Analysis (ISA), 2015.
  26. Continual occlusion and optical flow estimation. In Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part IV 14, pages 159–174. Springer, 2019.
  27. Automatic differentiation in pytorch. 2017.
  28. The 2017 davis challenge on video object segmentation. arXiv:1704.00675, 2017.
  29. Learning spatio-temporal representation with pseudo-3d residual networks. In proceedings of the IEEE International Conference on Computer Vision, pages 5533–5541, 2017.
  30. A fusion approach for multi-frame optical flow estimation. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 2077–2086. IEEE, 2019.
  31. Csflow: Learning optical flow via cross strip correlation for autonomous driving. In 2022 IEEE Intelligent Vehicles Symposium (IV), pages 1851–1858. IEEE, 2022.
  32. Videoflow: Exploiting temporal cues for multi-frame optical flow estimation. arXiv preprint arXiv:2303.08340, 2023a.
  33. Flowformer++: Masked cost volume autoencoding for pretraining optical flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1599–1610, 2023b.
  34. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi-domain operations applications, page 1100612. International Society for Optics and Photonics, 2019.
  35. Craft: Cross-attentional flow transformer for robust optical flow. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 17602–17611, 2022.
  36. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8934–8943, 2018.
  37. Models matter, so does training: An empirical study of cnns for optical flow estimation. IEEE transactions on pattern analysis and machine intelligence, 42(6):1408–1423, 2019.
  38. Skflow: Learning optical flow with super kernels. Advances in Neural Information Processing Systems, 35:11313–11326, 2022.
  39. Raft: Recurrent all-pairs field transforms for optical flow. In European conference on computer vision, pages 402–419. Springer, 2020.
  40. Splatflow: Learning multi-frame optical flow via splatting. arXiv preprint arXiv:2306.08887, 2023.
  41. Gmflow: Learning optical flow via global matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8121–8130, 2022.
  42. Volumetric correspondence networks for optical flow. Advances in neural information processing systems, 32, 2019.
  43. Hierarchical discrete distribution decomposition for match density estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6044–6053, 2019.
  44. Separable flow: Learning motion cost volumes for optical flow estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10807–10817, 2021.
  45. Maskflownet: Asymmetric feature matching with learnable occlusion mask. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6278–6287, 2020.
  46. Global matching with overlapping attention for optical flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17592–17601, 2022.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now