Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

UniScene: Multi-Camera Unified Pre-training via 3D Scene Reconstruction for Autonomous Driving (2305.18829v5)

Published 30 May 2023 in cs.CV, cs.MM, and cs.RO

Abstract: Multi-camera 3D perception has emerged as a prominent research field in autonomous driving, offering a viable and cost-effective alternative to LiDAR-based solutions. The existing multi-camera algorithms primarily rely on monocular 2D pre-training. However, the monocular 2D pre-training overlooks the spatial and temporal correlations among the multi-camera system. To address this limitation, we propose the first multi-camera unified pre-training framework, called UniScene, which involves initially reconstructing the 3D scene as the foundational stage and subsequently fine-tuning the model on downstream tasks. Specifically, we employ Occupancy as the general representation for the 3D scene, enabling the model to grasp geometric priors of the surrounding world through pre-training. A significant benefit of UniScene is its capability to utilize a considerable volume of unlabeled image-LiDAR pairs for pre-training purposes. The proposed multi-camera unified pre-training framework demonstrates promising results in key tasks such as multi-camera 3D object detection and surrounding semantic scene completion. When compared to monocular pre-training methods on the nuScenes dataset, UniScene shows a significant improvement of about 2.0% in mAP and 2.0% in NDS for multi-camera 3D object detection, as well as a 3% increase in mIoU for surrounding semantic scene completion. By adopting our unified pre-training method, a 25% reduction in 3D training annotation costs can be achieved, offering significant practical value for the implementation of real-world autonomous driving. Codes are publicly available at https://github.com/chaytonmin/UniScene.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. Y. Ma, T. Wang, X. Bai, H. Yang, Y. Hou, Y. Wang, Y. Qiao, R. Yang, D. Manocha, and X. Zhu, “Vision-centric bev perception: A survey,” arXiv preprint arXiv:2208.02797, 2022.
  2. H. Li, C. Sima, J. Dai, W. Wang, L. Lu, H. Wang, E. Xie, Z. Li, H. Deng, H. Tian et al., “Delving into the devils of bird’s-eye-view perception: A review, evaluation and recipe,” arXiv preprint arXiv:2209.05324, 2022.
  3. T. Liang, H. Xie, K. Yu, Z. Xia, Z. Lin, Y. Wang, T. Tang, B. Wang, and Z. Tang, “Bevfusion: A simple and robust lidar-camera fusion framework,” arXiv preprint arXiv:2205.13790, 2022.
  4. X. Chen, T. Zhang, Y. Wang, Y. Wang, and H. Zhao, “Futr3d: A unified sensor fusion framework for 3d detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 172–181.
  5. Y. Li, Y. Chen, X. Qi, Z. Li, J. Sun, and J. Jia, “Unifying voxel-based representation with transformer for 3d object detection,” arXiv preprint arXiv:2206.00630, 2022.
  6. Y. Wang, V. C. Guizilini, T. Zhang, Y. Wang, H. Zhao, and J. Solomon, “Detr3d: 3d object detection from multi-view images via 3d-to-2d queries,” in Conference on Robot Learning.   PMLR, 2022, pp. 180–191.
  7. Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Q. Yu, and J. Dai, “Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers,” arXiv preprint arXiv:2203.17270, 2022.
  8. J. Huang, G. Huang, Z. Zhu, and D. Du, “Bevdet: High-performance multi-camera 3d object detection in bird-eye-view,” arXiv preprint arXiv:2112.11790, 2021.
  9. Y. Li, Z. Ge, G. Yu, J. Yang, Z. Wang, Y. Shi, J. Sun, and Z. Li, “Bevdepth: Acquisition of reliable depth for multi-view 3d object detection,” arXiv preprint arXiv:2206.10092, 2022.
  10. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition.   Ieee, 2009, pp. 248–255.
  11. D. Park, R. Ambrus, V. Guizilini, J. Li, and A. Gaidon, “Is pseudo-lidar needed for monocular 3d object detection?” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 3142–3152.
  12. W. Tong, C. Sima, T. Wang, S. Wu, H. Deng, L. Chen, Y. Gu, L. Lu, P. Luo, D. Lin et al., “Scene as occupancy,” arXiv preprint arXiv:2306.02851, 2023.
  13. Y. Shi, K. Jiang, J. Li, J. Wen, Z. Qian, M. Yang, K. Wang, and D. Yang, “Grid-centric traffic scenario perception for autonomous driving: A comprehensive review,” arXiv preprint arXiv:2303.01212, 2023.
  14. Y. Li, Z. Yu, C. Choy, C. Xiao, J. M. Alvarez, S. Fidler, C. Feng, and A. Anandkumar, “Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 9087–9098.
  15. J. Philion and S. Fidler, “Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d,” in European Conference on Computer Vision.   Springer, 2020, pp. 194–210.
  16. H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 11 621–11 631.
  17. Y. Zhang, Z. Zhu, W. Zheng, J. Huang, G. Huang, J. Zhou, and J. Lu, “Beverse: Unified perception and prediction in birds-eye-view for vision-centric autonomous driving,” arXiv preprint arXiv:2205.09743, 2022.
  18. C. Han, J. Sun, Z. Ge, J. Yang, R. Dong, H. Zhou, W. Mao, Y. Peng, and X. Zhang, “Exploring recurrent long-term temporal fusion for multi-view 3d perception,” arXiv preprint arXiv:2303.05970, 2023.
  19. Z. Wang, C. Min, Z. Ge, Y. Li, Z. Li, H. Yang, and D. Huang, “Sts: Surround-view temporal stereo for multi-view 3d detection,” arXiv preprint arXiv:2208.10145, 2022.
  20. J. Park, C. Xu, S. Yang, K. Keutzer, K. Kitani, M. Tomizuka, and W. Zhan, “Time will tell: New outlooks and a baseline for temporal multi-view 3d object detection,” arXiv preprint arXiv:2210.02443, 2022.
  21. Y. Li, H. Bao, Z. Ge, J. Yang, J. Sun, and Z. Li, “Bevstereo: Enhancing depth estimation in multi-view 3d object detection with dynamic temporal stereo,” arXiv preprint arXiv:2209.10248, 2022.
  22. Y. Liu, T. Wang, X. Zhang, and J. Sun, “Petr: Position embedding transformation for multi-view 3d object detection,” arXiv preprint arXiv:2203.05625, 2022.
  23. Y. Hu, J. Yang, L. Chen, K. Li, C. Sima, X. Zhu, S. Chai, S. Du, T. Lin, W. Wang, L. Lu, X. Jia, Q. Liu, J. Dai, Y. Qiao, and H. Li, “Planning-oriented autonomous driving,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  24. M. Caron, P. Bojanowski, A. Joulin, and M. Douze, “Deep clustering for unsupervised learning of visual features,” in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 132–149.
  25. M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” Advances in Neural Information Processing Systems, vol. 33, pp. 9912–9924, 2020.
  26. K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 9729–9738.
  27. J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar et al., “Bootstrap your own latent-a new approach to self-supervised learning,” Advances in neural information processing systems, vol. 33, pp. 21 271–21 284, 2020.
  28. K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 16 000–16 009.
  29. H. Bao, L. Dong, S. Piao, and F. Wei, “Beit: Bert pre-training of image transformers,” arXiv preprint arXiv:2106.08254, 2021.
  30. C. Min, D. Zhao, L. Xiao, Y. Nie, and B. Dai, “Voxel-mae: Masked autoencoders for pre-training large-scale point clouds,” arXiv preprint arXiv:2206.09900, 2022.
  31. A. Boulch, C. Sautier, B. Michele, G. Puy, and R. Marlet, “Also: Automotive lidar self-supervision by occupancy estimation,” arXiv preprint arXiv:2212.05867, 2022.
  32. Z. Zhang, M. Bai, and E. Li, “Implicit surface contrastive clustering for lidar point clouds,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 21 716–21 725.
  33. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  34. Y. Yan, Y. Mao, and B. Li, “Second: Sparsely embedded convolutional detection,” Sensors, vol. 18, no. 10, p. 3337, 2018.
  35. S. Shi, C. Guo, L. Jiang, Z. Wang, J. Shi, X. Wang, and H. Li, “Pv-rcnn: Point-voxel feature set abstraction for 3d object detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 10 529–10 538.
  36. T. Yin, X. Zhou, and P. Krahenbuhl, “Center-based 3d object detection and tracking,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 11 784–11 793.
  37. X. Zhu, H. Zhou, T. Wang, F. Hong, Y. Ma, W. Li, H. Li, and D. Lin, “Cylindrical and asymmetrical 3d convolution networks for lidar segmentation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 9939–9948.
  38. Y. Huang, W. Zheng, Y. Zhang, J. Zhou, and J. Lu, “Tri-perspective view for vision-based 3d semantic occupancy prediction,” arXiv preprint arXiv:2302.07817, 2023.
  39. X. Wang, Z. Zhu, W. Xu, Y. Zhang, Y. Wei, X. Chi, Y. Ye, D. Du, J. Lu, and X. Wang, “Openoccupancy: A large scale benchmark for surrounding semantic occupancy perception,” arXiv preprint arXiv:2303.03991, 2023.
  40. X. Tian, T. Jiang, L. Yun, Y. Wang, Y. Wang, and H. Zhao, “Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving,” arXiv preprint arXiv:2304.14365, 2023.
  41. W. Gan, N. Mo, H. Xu, and N. Yokoya, “A simple attempt for 3d occupancy estimation in autonomous driving,” arXiv preprint arXiv:2303.10076, 2023.
  42. Z. Chen, Z. Li, S. Zhang, L. Fang, Q. Jiang, and F. Zhao, “Bevdistill: Cross-modal bev distillation for multi-view 3d object detection,” arXiv preprint arXiv:2211.09386, 2022.
  43. P. Huang, L. Liu, R. Zhang, S. Zhang, X. Xu, B. Wang, and G. Liu, “Tig-bev: Multi-view bev 3d object detection via target inner-geometry learning,” arXiv preprint arXiv:2212.13979, 2022.
  44. J. Liu, T. Wang, B. Liu, Q. Zhang, Y. Liu, and H. Li, “Towards better 3d knowledge transfer via masked image modeling for multi-view 3d understanding,” arXiv preprint arXiv:2303.11325, 2023.
  45. B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Communications of the ACM, vol. 65, no. 1, pp. 99–106, 2021.
  46. J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan, “Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 5855–5864.
  47. Q. Zhu, C. Min, Z. Wei, Y. Chen, and G. Wang, “Deep learning for multi-view stereo via plane sweep: A survey,” arXiv preprint arXiv:2106.15328, 2021.
  48. Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan, “Mvsnet: Depth inference for unstructured multi-view stereo,” in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 767–783.
  49. Z. Wei, Q. Zhu, C. Min, Y. Chen, and G. Wang, “Aa-rmvsnet: Adaptive aggregation recurrent multi-view stereo network,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 6187–6196.
  50. Z. Wei, Q. Zhu, C. Min, and G. Wang, “Bidirectional hybrid lstm based recurrent neural network for multi-view stereo,” IEEE Transactions on Visualization and Computer Graphics, 2022.
  51. . T. A. Day, “[online],” http://https://www.youtube.com/watch?v=jPCV4GKX9Dw.
  52. T. Wang, X. Zhu, J. Pang, and D. Lin, “FCOS3D: Fully convolutional one-stage monocular 3d object detection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2021.
  53. “Cvpr 2023 occupancy prediction challenge,” 2023. [Online]. Available: https://github.com/CVPR2023-3D-Occupancy-Prediction/CVPR2023-3D-Occupancy-Prediction
  54. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022.
  55. P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine et al., “Scalability in perception for autonomous driving: Waymo open dataset,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 2446–2454.
  56. A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in 2012 IEEE conference on computer vision and pattern recognition.   IEEE, 2012, pp. 3354–3361.
  57. D. Ha and J. Schmidhuber, “World models,” arXiv preprint arXiv:1803.10122, 2018.
  58. D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba, “Mastering atari with discrete world models,” arXiv preprint arXiv:2010.02193, 2020.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Chen Min (17 papers)
  2. Liang Xiao (80 papers)
  3. Dawei Zhao (22 papers)
  4. Yiming Nie (9 papers)
  5. Bin Dai (60 papers)
Citations (14)
View on Semantic Scholar

Summary

Multi-Camera Unified Pre-Training via 3D Scene Reconstruction

The research paper introduces a novel approach to multi-camera 3D perception in autonomous driving, which has increasingly become a focal point in the field, as it provides a more cost-efficient alternative to LiDAR-based solutions. The current state-of-the-art methodologies rely heavily on monocular 2D pre-training, which often neglects the spatial and temporal correlations inherent in multi-camera systems. To bridge this gap, the authors propose a unified pre-training framework named UniScene, which emphasizes 3D scene reconstruction as a foundational pre-training phase.

The essence of the UniScene framework lies in its ability to harness 3D geometric occupancy as a primary representation for the pre-training models. The approach enables the incorporation of geometric priors from the surrounding environment, a critical evolution from the current methods that focus mainly on monocular depth estimation. UniScene's methodology leverages unlabeled image-LiDAR pairs for pre-training, which offers a significant advantage in terms of reducing the manual annotation costs associated with 3D training, with a reported reduction by 25%.

The experimental evaluation within the paper utilizes the nuScenes dataset and exhibits a marked performance improvement over monocular pre-training methods. Specifically, the UniScene model demonstrates a 2.0% improvement in mAP and NDS metrics for multi-camera 3D object detection and a 3.0% increase in mIoU for surrounding semantic scene completion tasks. This enhanced performance can be attributed to the model’s capacity to effectively process the rich spatial and temporal information from the image-LiDAR data. The capability of reconstructing entire 3D occupancy grids rather than solely relying on depth estimation from monocular views is a pivotal advancement.

Methodologically, UniScene incorporates a multi-camera unified pre-training process that constructs 3D scene reconstructions. A noteworthy decision within this process is the multi-frame fusion approach in which LiDAR point clouds from several frames are combined to produce occupancy labels, enhancing the richness and accuracy of the data representation. The training process employs a focal loss approach to address the inherent imbalance in the occupancy prediction task, focusing on ensuring accurate predictions of occupied voxels.

Besides providing empirical evidence of improved accuracy and precision in 3D perception, the authors speculate on the potential applications and extensions of their framework. The UniScene model lays groundwork that future research can build on, particularly towards enhancing the perception of autonomous systems through the integration of more sophisticated 3D reconstruction methodologies, potentially involving NeRF or MVS techniques.

The implications of this research are substantial both theoretically and practically. From a theoretical standpoint, this work challenges the traditional reliance on 2D single-view pre-training in multi-camera systems by presenting a robust alternative that captures more comprehensive spatial and temporal dynamics. Practically, the adoption of such frameworks in real-world autonomous driving scenarios can catalyze advancements in system performance while simultaneously reducing data annotation overheads.

In conclusion, UniScene’s paradigm shift towards 3D scene reconstruction for multi-camera systems sets a new direction for autonomous vehicle perception systems. This methodology not only exhibits superior performance in benchmark tests but also offers a scalable, cost-effective approach for future developments in AI-driven perception systems. The framework’s advancement could play a pivotal role in fulfilling the increasing demands for efficient, precise, and scalable autonomous systems.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown