Papers
Topics
Authors
Recent
2000 character limit reached

ESC: Evolutionary Stitched Camera Calibration in the Wild (2404.12694v1)

Published 19 Apr 2024 in cs.CV and cs.NE

Abstract: This work introduces a novel end-to-end approach for estimating extrinsic parameters of cameras in multi-camera setups on real-life sports fields. We identify the source of significant calibration errors in multi-camera environments and address the limitations of existing calibration methods, particularly the disparity between theoretical models and actual sports field characteristics. We propose the Evolutionary Stitched Camera calibration (ESC) algorithm to bridge this gap. It consists of image segmentation followed by evolutionary optimization of a novel loss function, providing a unified and accurate multi-camera calibration solution with high visual fidelity. The outcome allows the creation of virtual stitched views from multiple video sources, being as important for practical applications as numerical accuracy. We demonstrate the superior performance of our approach compared to state-of-the-art methods across diverse real-life football fields with varying physical characteristics.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (40)
  1. ChyronHego TRACAB Technologies, 2021. [Online]. Available: https://tracab.com
  2. Sagsport, 2023. [Online]. Available: https://sagsport.com
  3. Veo, 2023. [Online]. Available: https://www.veo.co/product/camera
  4. W. Jiang, J. C. G. Higuera, B. Angles, W. Sun, M. Javan, and K. M. Yi, “Optimizing through learned errors for accurate sports field registration,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020, pp. 201–210.
  5. J. Chen and J. J. Little, “Sports camera calibration via synthetic data,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, 2019, pp. 0–0.
  6. G. Rypeść, G. Kurzejamski, and J. Komorowski, “Sports camera pose refinement using an evolution strategy,” in 2022 IEEE Congress on Evolutionary Computation (CEC), 2022, pp. 1–8.
  7. R. Szeliski, “Video mosaics for virtual environments,” IEEE computer Graphics and Applications, vol. 16, no. 2, pp. 22–30, 1996.
  8. J. Gao, S. J. Kim, and M. S. Brown, “Constructing image panoramas using dual-homography warping,” in CVPR 2011.   IEEE, 2011, pp. 49–56.
  9. K. M. Yi, E. Trulls, V. Lepetit, and P. Fua, “Lift: Learned invariant feature transform,” in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14.   Springer, 2016, pp. 467–483.
  10. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
  11. G. Koch, “Siamese neural networks for one-shot image recognition,” Ph.D. dissertation, University of Toronto, 2015.
  12. T. Lindeberg, “Scale invariant feature transform,” Scholarpedia, vol. 7, no. 5, p. 10491, 2012.
  13. H. Le, F. Liu, S. Zhang, and A. Agarwala, “Deep homography estimation for dynamic scenes,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 7652–7661.
  14. L. Nie, C. Lin, K. Liao, S. Liu, and Y. Zhao, “Unsupervised deep image stitching: Reconstructing stitched features to images,” IEEE Transactions on Image Processing, vol. 30, pp. 6184–6197, 2021.
  15. F. Wang, L. Sun, B. Yang, and S. Yang, “Fast arc detection algorithm for play field registration in soccer video mining,” in 2006 IEEE International Conference on Systems, Man and Cybernetics, vol. 6.   IEEE, 2006, pp. 4932–4936.
  16. A. Yamada, Y. Shirai, and J. Miura, “Tracking players and a ball in video image sequence and estimating camera parameters for 3d interpretation of soccer games,” in Object recognition supported by user interaction for service robots, vol. 1.   IEEE, 2002, pp. 303–306.
  17. H. Kim and K. S. Hong, “Soccer video mosaicing using self-calibration and line tracking,” in Proceedings 15th International Conference on Pattern Recognition. ICPR-2000, vol. 1.   IEEE, 2000, pp. 592–595.
  18. D. Farin, S. Krabbe, W. Effelsberg et al., “Robust camera calibration for sport videos using court models,” in Storage and Retrieval Methods and Applications for Multimedia 2004, vol. 5307.   International Society for Optics and Photonics, 2003, pp. 80–91.
  19. R. O. Duda and P. E. Hart, “Use of the hough transformation to detect lines and curves in pictures,” Communications of the ACM, vol. 15, no. 1, pp. 11–15, 1972.
  20. J. Puwein, R. Ziegler, J. Vogel, and M. Pollefeys, “Robust multi-view camera calibration for wide-baseline camera networks,” in 2011 IEEE Workshop on Applications of Computer Vision (WACV).   IEEE, 2011, pp. 321–328.
  21. B. Yang, “Sports image classification and application based on sift algorithm,” in 2022 Second International Conference on Advanced Technologies in Intelligent Control, Environment, Computing & Communication Engineering (ICATIECE).   IEEE, 2022, pp. 1–5.
  22. D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” International journal of computer vision, vol. 60, pp. 91–110, 2004.
  23. J. Matas, O. Chum, M. Urban, and T. Pajdla, “Robust wide-baseline stereo from maximally stable extremal regions,” Image and vision computing, vol. 22, no. 10, pp. 761–767, 2004.
  24. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  25. O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted intervention.   Springer, 2015, pp. 234–241.
  26. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” Communications of the ACM, vol. 63, no. 11, pp. 139–144, 2020.
  27. S. Minaee, Y. Boykov, F. Porikli, A. Plaza, N. Kehtarnavaz, and D. Terzopoulos, “Image segmentation using deep learning: A survey,” IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 7, pp. 3523–3542, 2021.
  28. J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3431–3440.
  29. W. Liu, A. Rabinovich, and A. C. Berg, “Parsenet: Looking wider to see better,” arXiv preprint arXiv:1506.04579, 2015.
  30. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.
  31. S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. H. Torr et al., “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 6881–6890.
  32. R. Strudel, R. Garcia, I. Laptev, and C. Schmid, “Segmenter: Transformer for semantic segmentation,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 7262–7272.
  33. Z. Zhang, “A flexible new technique for camera calibration,” IEEE Transactions on pattern analysis and machine intelligence, vol. 22, no. 11, pp. 1330–1334, 2000.
  34. G. Gallego and A. Yezzi, “A compact formula for the derivative of a 3-d rotation in exponential coordinates,” Journal of Mathematical Imaging and Vision, vol. 51, no. 3, pp. 378–384, 2015.
  35. T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2980–2988.
  36. G. Bradski, “The opencv library.” Dr. Dobb’s Journal: Software Tools for the Professional Programmer, vol. 25, no. 11, pp. 120–123, 2000.
  37. S.-Y. Cao, J. Hu, Z. Sheng, and H.-L. Shen, “Iterative deep homography estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 1879–1888.
  38. R. A. Sharma, B. Bhat, V. Gandhi, and C. Jawahar, “Automated top view registration of broadcast football videos,” in 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).   IEEE, 2018, pp. 305–313.
  39. A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” 2017.
  40. D. DeTone, T. Malisiewicz, and A. Rabinovich, “Deep image homography estimation,” arXiv preprint arXiv:1606.03798, 2016.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Slide Deck Streamline Icon: https://streamlinehq.com

Whiteboard

Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up