Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Zero-BEV: Zero-shot Projection of Any First-Person Modality to BEV Maps (2402.13848v2)

Published 21 Feb 2024 in cs.CV and cs.RO

Abstract: Bird's-eye view (BEV) maps are an important geometrically structured representation widely used in robotics, in particular self-driving vehicles and terrestrial robots. Existing algorithms either require depth information for the geometric projection, which is not always reliably available, or are trained end-to-end in a fully supervised way to map visual first-person observations to BEV representation, and are therefore restricted to the output modality they have been trained for. In contrast, we propose a new model capable of performing zero-shot projections of any modality available in a first person view to the corresponding BEV map. This is achieved by disentangling the geometric inverse perspective projection from the modality transformation, eg. RGB to occupancy. The method is general and we showcase experiments projecting to BEV three different modalities: semantic segmentation, motion vectors and object bounding boxes detected in first person. We experimentally show that the model outperforms competing methods, in particular the widely used baseline resorting to monocular depth estimation.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. Mohamed Aly. Real time detection of lane markers in urban streets. In 2008 IEEE intelligent vehicles symposium, pages 7–12. IEEE, 2008.
  2. LaRa: Latents and rays for multi-camera bird’s-eye-view semantic segmentation. In Conference on Robot Learning (CoRL), pages 1663–1672, 2023.
  3. EgoMap: Projective mapping and structured egocentric memory for Deep RL. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD), pages 525–540. Springer, 2020.
  4. Structured bird’s-eye-view traffic scene understanding from onboard images. In International Conference on Computer Vision (ICCV), pages 15641–15650, 2021.
  5. Matterport3d: Learning from rgb-d data in indoor environments. In International Conference on 3D Vision (3DV), 2018.
  6. Goat: Go to any thing. arXiv preprint arXiv:2311.06430, 2023.
  7. Learning To Explore Using Active Neural SLAM. In International Conference on Learning Representations (ICLR), 2019.
  8. Object goal navigation using goal-oriented semantic exploration. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
  9. Describing textures in the wild. In Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
  10. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In International Conference on Computer Vision (ICCV), pages 10786–10796, 2021.
  11. Semantic grid estimation with a hybrid bayesian and deep neural network approach. In International Conference on Intelligent Robots and Systems (IROS), 2018.
  12. Navigating to objects in the real world. Science Robotics, 8(79), 2023.
  13. Gitnet: Geometric prior-based transformation for birds-eye-view segmentation. In European Conference on Computer Vision (ECCV), pages 396–411, 2022.
  14. SkyEye: Self-supervised bird’s-eye-view semantic mapping using monocular frontal view images. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 14901–14910, 2023.
  15. Improving Grid-based SLAM with Rao-Blackwellized Particle Filters by Adaptive Proposals and Selective Resampling. In International Conference on Robotics and Automation (ICRA), 2005.
  16. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  17. Mapnet: An allocentric spatial memory for mapping environments. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 8476–8484, 2018.
  18. Fiery: Future instance prediction in bird’s-eye view from surround monocular cameras. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 15273–15282, 2021.
  19. 3D common corruptions and data augmentation. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 18963–18974, 2022.
  20. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
  21. Rtab-map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation. Journal of Field Robotics, 36(2):416–446, 2019.
  22. Delving into the devils of bird’s-eye-view perception: A review, evaluation and recipe. arXiv preprint, abs/2209.05324, 2022a.
  23. Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. In European Conference Computer Vision (ECCV), pages 1–18, 2022b.
  24. Feature pyramid networks for object detection. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 2117–2125, 2017.
  25. Understanding road layout from videos as a whole. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 4413–4422, 2020.
  26. End-to-end multi-task learning with attention. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 1871–1880, 2019.
  27. Petrv2: A unified framework for 3d perception from multi-camera images. arXiv preprint, abs/2206.01256, 2022.
  28. Monocular semantic occupancy grid mapping with convolutional variational encoder–decoder networks. IEEE Robotics and Automation Letters, 4(2):445–452, 2019.
  29. Vision-centric bev perception: A survey. arXiv preprint, abs/2208.02797, 2023.
  30. Inverse perspective mapping simplifies optical flow computation and obstacle detection. Biological Cybernetics, 64:177–185, 1991.
  31. Teaching agents how to map: Spatial reasoning for multi-object navigation. In International Conference on Intelligent Robots and Systems (IROS), 2022.
  32. Weakly supervised 3d object detection from lidar point cloud. In European Conference on computer vision (ECCV), pages 515–531, 2020.
  33. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In International Conference on 3D Vision (3DV), pages 565–571. Ieee, 2016.
  34. Cross-view semantic segmentation for sensing surroundings. IEEE Robotics and Automation Letters, 5(3):4867–4873, 2020.
  35. Bevsegformer: Bird’s eye view semantic segmentation from arbitrary camera rigs. In Winter Conference on Applications of Computer Vision, pages 5935–5943, 2023.
  36. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In European Conference on Computer Vision (ECCV), 2020.
  37. Habitat-matterport 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AI. In Conference on Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track (Round 2), 2021.
  38. Poni: Potential functions for objectgoal navigation with interaction-free learning. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 18890–18900, 2022.
  39. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence, 2020.
  40. You only look once: Unified, real-time object detection. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  41. Predicting semantic map representations from images using pyramid occupancy networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 11138–11147, 2020.
  42. Conditional Monte Carlo Dense Occupancy Tracker. In International Conference on Intelligent Transportation Systems, 2015.
  43. Enabling spatio-temporal aggregation in birds-eye-view vehicle estimation. In International Conference on Robotics and Automation (ICRA), pages 5133–5139. IEEE, 2021.
  44. Translating images into maps. In International Conference on Robotics and Automation (ICRA). IEEE, 2022.
  45. Habitat: A platform for embodied ai research. In International Conference on Computer Vision (ICCV), 2019.
  46. Learning to look around objects for top-view representations of outdoor scenes. In European Conference Computer Vision (ECCV), pages 815–831. Springer, 2018.
  47. Automatic dense visual semantic mapping from street-level imagery. In International Conference on Intelligent Robots and Systems (IROS), pages 857–862, 2012.
  48. Richard Szeliski. Computer vision algorithms and applications. Springer Verlag, 2011.
  49. Raft: Recurrent all-pairs field transforms for optical flow. In European Conference Computer Vision (ECCV), pages 402–419. Springer, 2020.
  50. Probabilistic robotics, vol. 1. MIT Press Cambridge, 2005.
  51. Attention is all you need. In Conference on Neural Information Processing Systems (NeurIPS), 2017.
  52. Probabilistic and geometric depth: Detecting objects in perspective. In Conference on Robot Learning (CoRL), pages 1475–1485, 2021.
  53. M22{}^{\mbox{2}}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPTBEV: Multi-camera joint 3d detection and segmentation with unified birds-eye view representation. arXiv preprint, abs/2204.05088, 2022.
  54. Habitat-matterport 3D semantics dataset. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 4927–4936, 2023.
  55. Woodscape: A multi-task, multi-camera fisheye dataset for autonomous driving. In International Conference on Computer Vision (ICCV), 2019.
  56. Robust learning through cross-task consistency. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 11197–11206, 2020.
  57. Cross-view transformers for real-time map-view semantic segmentation. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 13760–13769, 2022.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now