Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Rethinking RAFT for Efficient Optical Flow (2401.00833v1)

Published 1 Jan 2024 in cs.CV

Abstract: Despite significant progress in deep learning-based optical flow methods, accurately estimating large displacements and repetitive patterns remains a challenge. The limitations of local features and similarity search patterns used in these algorithms contribute to this issue. Additionally, some existing methods suffer from slow runtime and excessive graphic memory consumption. To address these problems, this paper proposes a novel approach based on the RAFT framework. The proposed Attention-based Feature Localization (AFL) approach incorporates the attention mechanism to handle global feature extraction and address repetitive patterns. It introduces an operator for matching pixels with corresponding counterparts in the second frame and assigning accurate flow values. Furthermore, an Amorphous Lookup Operator (ALO) is proposed to enhance convergence speed and improve RAFTs ability to handle large displacements by reducing data redundancy in its search operator and expanding the search space for similarity extraction. The proposed method, Efficient RAFT (Ef-RAFT),achieves significant improvements of 10% on the Sintel dataset and 5% on the KITTI dataset over RAFT. Remarkably, these enhancements are attained with a modest 33% reduction in speed and a mere 13% increase in memory usage. The code is available at: https://github.com/n3slami/Ef-RAFT

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. M. Vihlman and A. Visala, “Optical flow in deep visual tracking,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 07, 2020, pp. 12 112–12 119.
  2. C. Yang, H. Lamdouar, E. Lu, A. Zisserman, and W. Xie, “Self-supervised video object segmentation by motion grouping,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 7177–7188.
  3. G. C. de Croon, C. De Wagter, and T. Seidl, “Enhancing optical-flow-based control by learning visual appearance cues for flying robots,” Nature Machine Intelligence, vol. 3, no. 1, pp. 33–41, 2021.
  4. S. S. Beauchemin and J. L. Barron, “The computation of optical flow,” ACM computing surveys (CSUR), vol. 27, no. 3, pp. 433–466, 1995.
  5. B. K. Horn and B. G. Schunck, “Determining optical flow,” Artificial intelligence, vol. 17, no. 1-3, pp. 185–203, 1981.
  6. A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. Van Der Smagt, D. Cremers, and T. Brox, “Flownet: Learning optical flow with convolutional networks,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 2758–2766.
  7. A. Ranjan and M. J. Black, “Optical flow estimation using a spatial pyramid network,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4161–4170.
  8. D. Sun, X. Yang, M.-Y. Liu, and J. Kautz, “Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8934–8943.
  9. Z. Teed and J. Deng, “Raft: Recurrent all-pairs field transforms for optical flow,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16.   Springer, 2020, pp. 402–419.
  10. S. Jiang, D. Campbell, Y. Lu, H. Li, and R. Hartley, “Learning to estimate hidden motions with global motion aggregation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9772–9781.
  11. S. Zhao, L. Zhao, Z. Zhang, E. Zhou, and D. Metaxas, “Global matching with overlapping attention for optical flow estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 17 592–17 601.
  12. A. Luo, F. Yang, K. Luo, X. Li, H. Fan, and S. Liu, “Learning optical flow with adaptive graph reasoning,” in Proceedings of the AAAI conference on artificial intelligence, vol. 36, no. 2, 2022, pp. 1890–1898.
  13. Z. Huang, X. Shi, C. Zhang, Q. Wang, K. C. Cheung, H. Qin, J. Dai, and H. Li, “Flowformer: A transformer architecture for optical flow,” in European Conference on Computer Vision.   Springer, 2022, pp. 668–685.
  14. C. Zach, T. Pock, and H. Bischof, “A duality based approach for realtime tv-l 1 optical flow,” in Pattern Recognition: 29th DAGM Symposium, Heidelberg, Germany, September 12-14, 2007. Proceedings 29.   Springer, 2007, pp. 214–223.
  15. M. J. Black and P. Anandan, “A framework for the robust estimation of optical flow,” in 1993 (4th) International Conference on Computer Vision.   IEEE, 1993, pp. 231–236.
  16. T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, “High accuracy optical flow estimation based on a theory for warping,” in Computer Vision-ECCV 2004: 8th European Conference on Computer Vision, Prague, Czech Republic, May 11-14, 2004. Proceedings, Part IV 8.   Springer, 2004, pp. 25–36.
  17. C. Bailer, B. Taetz, and D. Stricker, “Flow fields: Dense correspondence fields for highly accurate large displacement optical flow estimation,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 4015–4023.
  18. M. Menze, C. Heipke, and A. Geiger, “Discrete optimization for optical flow,” in Pattern Recognition: 37th German Conference, GCPR 2015, Aachen, Germany, October 7-10, 2015, Proceedings 37.   Springer, 2015, pp. 16–28.
  19. P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid, “Deepflow: Large displacement optical flow with deep matching,” in Proceedings of the IEEE international conference on computer vision, 2013, pp. 1385–1392.
  20. T. Brox and J. Malik, “Large displacement optical flow: descriptor matching in variational motion estimation,” IEEE transactions on pattern analysis and machine intelligence, vol. 33, no. 3, pp. 500–513, 2010.
  21. J. Revaud, P. Weinzaepfel, Z. Harchaoui, and C. Schmid, “Epicflow: Edge-preserving interpolation of correspondences for optical flow,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1164–1172.
  22. E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox, “Flownet 2.0: Evolution of optical flow estimation with deep networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2462–2470.
  23. T.-W. Hui, X. Tang, and C. C. Loy, “Liteflownet: A lightweight convolutional neural network for optical flow estimation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8981–8989.
  24. S. Zhao, Y. Sheng, Y. Dong, E. I. Chang, Y. Xu et al., “Maskflownet: Asymmetric feature matching with learnable occlusion mask,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 6278–6287.
  25. J. Hur and S. Roth, “Iterative residual refinement for joint optical flow and occlusion estimation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 5754–5763.
  26. G. Yang and D. Ramanan, “Volumetric correspondence networks for optical flow,” Advances in neural information processing systems, vol. 32, 2019.
  27. C. Deng, A. Luo, H. Huang, S. Ma, J. Liu, and S. Liu, “Explicit motion disentangling for efficient optical flow estimation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 9521–9530.
  28. F. Zhang, O. J. Woodford, V. A. Prisacariu, and P. H. Torr, “Separable flow: Learning motion cost volumes for optical flow estimation,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 807–10 817.
  29. Y. Lu, Q. Wang, S. Ma, T. Geng, Y. V. Chen, H. Chen, and D. Liu, “Transflow: Transformer as flow learner,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 18 063–18 073.
  30. X. Shi, Z. Huang, D. Li, M. Zhang, K. C. Cheung, S. See, H. Qin, J. Dai, and H. Li, “Flowformer++: Masked cost volume autoencoding for pretraining optical flow estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 1599–1610.
  31. H. Xu, J. Yang, J. Cai, J. Zhang, and X. Tong, “High-resolution optical flow from 1d attention and correlation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10 498–10 507.
  32. X. Sui, S. Li, X. Geng, Y. Wu, X. Xu, Y. Liu, R. Goh, and H. Zhu, “Craft: Cross-attentional flow transformer for robust optical flow,” in Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, 2022, pp. 17 602–17 611.
  33. J. Park, S. Woo, J.-Y. Lee, and I. S. Kweon, “Bam: Bottleneck attention module,” arXiv preprint arXiv:1807.06514, 2018.
  34. S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “Cbam: Convolutional block attention module,” in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 3–19.
  35. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  36. N. Mayer, E. Ilg, P. Hausser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox, “A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 4040–4048.
  37. A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The kitti dataset,” The International Journal of Robotics Research, vol. 32, no. 11, pp. 1231–1237, 2013.
  38. D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black, “A naturalistic open source movie for optical flow evaluation,” in Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part VI 12.   Springer, 2012, pp. 611–625.
  39. T.-W. Hui, X. Tang, and C. C. Loy, “A lightweight optical flow cnn—revisiting data fidelity and regularization,” IEEE transactions on pattern analysis and machine intelligence, vol. 43, no. 8, pp. 2555–2569, 2020.
  40. J. Wang, Y. Zhong, Y. Dai, K. Zhang, P. Ji, and H. Li, “Displacement-invariant matching cost learning for accurate optical flow estimation,” Advances in Neural Information Processing Systems, vol. 33, pp. 15 220–15 231, 2020.
  41. S. Jiang, Y. Lu, H. Li, and R. Hartley, “Learning optical flow from a few matches,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 16 592–16 600.
  42. A. Luo, F. Yang, X. Li, and S. Liu, “Learning optical flow with kernel patch attention,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 8906–8915.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - n3slami/Ef-RAFT: Ef-RAFT: Rethinking RAFT for Efficient Optical Flow (MVIP 2024) (31 stars)