Papers
Topics
Authors
Recent
Search
2000 character limit reached

A Survey on Monocular Re-Localization: From the Perspective of Scene Map Representation

Published 27 Nov 2023 in cs.RO | (2311.15643v3)

Abstract: Monocular Re-Localization (MRL) is a critical component in autonomous applications, estimating 6 degree-of-freedom ego poses w.r.t. the scene map based on monocular images. In recent decades, significant progress has been made in the development of MRL techniques. Numerous algorithms have accomplished extraordinary success in terms of localization accuracy and robustness. In MRL, scene maps are represented in various forms, and they determine how MRL methods work and how MRL methods perform. However, to the best of our knowledge, existing surveys do not provide systematic reviews about the relationship between MRL solutions and their used scene map representation. This survey fills the gap by comprehensively reviewing MRL methods from such a perspective, promoting further research. 1) We commence by delving into the problem definition of MRL, exploring current challenges, and comparing ours with existing surveys. 2) Many well-known MRL methods are categorized and reviewed into five classes according to the representation forms of utilized map, i.e., geo-tagged frames, visual landmarks, point clouds, vectorized semantic map, and neural network-based map. 3) To quantitatively and fairly compare MRL methods with various map, we introduce some public datasets and provide the performances of some state-of-the-art MRL methods. The strengths and weakness of MRL methods with different map are analyzed. 4) We finally introduce some topics of interest in this field and give personal opinions. This survey can serve as a valuable referenced materials for MRL, and a continuously updated summary of this survey is publicly available to the community at: https://github.com/jinyummiao/map-in-mono-reloc.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (222)
  1. doi:https://doi.org/10.1016/j.patcog.2017.09.013. URL https://www.sciencedirect.com/science/article/pii/S0031320317303448
  2. doi:10.1109/TITS.2022.3175656.
  3. doi:10.1109/TVCG.2014.27.
  4. arXiv:https://doi.org/10.1177/0278364914561101, doi:10.1177/0278364914561101. URL https://doi.org/10.1177/0278364914561101
  5. doi:10.1109/TITS.2022.3176914.
  6. doi:10.3390/s20071870. URL https://www.mdpi.com/1424-8220/20/7/1870
  7. doi:10.1109/ICRA.2012.6224623.
  8. doi:10.1109/IROS.2016.7759304.
  9. doi:10.1109/ITSC.2019.8917470.
  10. doi:10.1109/TPAMI.2017.2711011.
  11. doi:10.1109/LRA.2020.2969917.
  12. doi:10.1109/CVPR.2019.01300.
  13. doi:10.1109/ICCV.2011.6126544.
  14. doi:10.1109/CVPRW.2018.00060.
  15. doi:10.1145/3065386. URL https://doi.org/10.1145/3065386
  16. doi:10.1109/TPAMI.2016.2577031.
  17. doi:10.1109/ICCV.2015.336.
  18. doi:10.1109/TPAMI.2017.2667665.
  19. doi:10.1145/3394171.3413896. URL https://doi.org/10.1145/3394171.3413896
  20. doi:10.1007/s10514-009-9138-7. URL https://doi.org/10.1007/s10514-009-9138-7
  21. doi:10.1109/TII.2020.3010580.
  22. doi:10.1109/TPAMI.2019.2952114.
  23. doi:10.1109/TCSVT.2021.3061265.
  24. doi:10.1109/ICRA.2018.8461051. URL https://doi.org/10.1109/ICRA.2018.8461051
  25. doi:10.1109/LRA.2018.2801879.
  26. doi:10.1109/IROS51168.2021.9636248.
  27. doi:10.1109/ICRA40945.2020.9196638.
  28. doi:10.1109/ICCV.2019.00013.
  29. doi:10.1109/TRO.2015.2496823.
  30. doi:10.24963/ijcai.2021/603. URL https://doi.org/10.24963/ijcai.2021/603
  31. doi:https://doi.org/10.1016/j.patcog.2020.107760. URL https://www.sciencedirect.com/science/article/pii/S003132032030563X
  32. doi:10.1007/s10462-012-9365-8. URL https://doi.org/10.1007/s10462-012-9365-8
  33. doi:10.1109/ICRA.2017.7989671.
  34. doi:10.1109/CVPR.2010.5540039.
  35. doi:10.1023/B:VISI.0000029664.99615.94.
  36. doi:10.1109/TRO.2009.2022424.
  37. doi:10.1109/ROBOT.2000.844734.
  38. doi:10.1109/TPAMI.2018.2846566.
  39. doi:10.1109/CVPR52688.2022.01328.
  40. doi:10.1109/WACV56688.2023.00301.
  41. doi:https://doi.org/10.1016/j.patrec.2021.11.014. URL https://www.sciencedirect.com/science/article/pii/S0167865521004050
  42. doi:10.1016/j.cviu.2007.09.014.
  43. doi:10.1007/s10514-017-9684-3. URL https://doi.org/10.1007/s10514-017-9684-3
  44. doi:10.1109/IROS.2015.7353986.
  45. doi:10.1109/CVPR.2013.207.
  46. doi:10.1109/IROS.2011.6094885.
  47. doi:10.1109/TRO.2012.2197158.
  48. doi:10.1109/IROS40897.2019.8967726.
  49. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/rob.22088, doi:https://doi.org/10.1002/rob.22088. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/rob.22088
  50. doi:10.1109/TITS.2021.3074520.
  51. doi:10.1109/TRO.2012.2192013.
  52. doi:10.1177/0278364908090961.
  53. doi:10.1177/0278364910385483.
  54. doi:10.1109/ICRA.2015.7139959.
  55. doi:10.1109/ICCV.2011.6126542.
  56. doi:10.1109/LRA.2018.2849609.
  57. doi:10.1109/ICRA.2018.8461146.
  58. doi:10.1109/LRA.2019.2897151.
  59. doi:10.1109/CVPR46437.2021.01392.
  60. doi:10.1109/TRO.2017.2704598.
  61. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/rob.22060, doi:https://doi.org/10.1002/rob.22060. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/rob.22060
  62. doi:10.1109/LRA.2021.3067633.
  63. doi:10.1109/LRA.2020.3005627.
  64. doi:10.1109/LRA.2022.3194310.
  65. doi:10.1145/2623330.2623732. URL https://doi.org/10.1145/2623330.2623732
  66. doi:10.1017/CBO9780511811685.
  67. doi:10.1109/TPAMI.2004.17.
  68. doi:10.1109/TIM.2023.3271000.
  69. doi:10.1109/CVPR42600.2020.00499.
  70. doi:10.1109/CVPR46437.2021.00881.
  71. doi:10.1109/CVPR.2018.00282.
  72. doi:10.1109/CVPR.2019.01044.
  73. doi:10.1109/CVPR42600.2020.00138.
  74. doi:10.1109/ICCV.2019.00442.
  75. doi:10.1109/CVPR42600.2020.01130.
  76. doi:10.1109/ICRA40945.2020.9196607.
  77. doi:10.1109/ICRA48506.2021.9560872. URL https://doi.org/10.1109/ICRA48506.2021.9560872
  78. doi:10.1109/ICCV48922.2021.01196.
  79. doi:10.1109/CVPR.2019.00458.
  80. doi:10.1109/ICCVW.2017.113.
  81. doi:10.1109/IROS51168.2021.9635870. URL https://doi.org/10.1109/IROS51168.2021.9635870
  82. doi:10.1109/CVPR46437.2021.01433.
  83. doi:10.1109/CVPR46437.2021.00327.
  84. doi:10.1007/s001380050048. URL https://doi.org/10.1007/s001380050048
  85. doi:10.1109/CVPR.2016.445.
  86. doi:10.1109/TRO.2021.3075644.
  87. doi:10.1109/CVPR.2017.523.
  88. doi:10.1109/CVPR.2017.649.
  89. doi:10.1109/CVPR.2019.01127.
  90. doi:10.1109/CVPR.2019.00263.
  91. doi:10.1109/CVPR.2019.00828.
  92. doi:10.1109/CVPR42600.2020.00662.
  93. doi:10.1109/ICCV.2017.374.
  94. doi:10.1109/ICCV48922.2021.01122.
  95. doi:10.1109/CVPR.2017.410.
  96. doi:10.1109/TPAMI.2019.2915233.
  97. doi:10.1109/ICIEA54703.2022.10006314.
  98. doi:10.1109/TPAMI.2014.2321376.
  99. doi:https://doi.org/10.1016/j.patrec.2011.08.021. URL https://www.sciencedirect.com/science/article/pii/S0167865511002765
  100. doi:10.1109/TIP.2014.2307478.
  101. doi:10.1007/s11263-018-1117-z. URL https://doi.org/10.1007/s11263-018-1117-z
  102. doi:10.1109/TGRS.2018.2820040.
  103. doi:10.1109/TPAMI.2017.2652468.
  104. doi:10.1109/CVPR.2017.302.
  105. doi:10.1109/TIP.2015.2496305.
  106. doi:10.1007/s11263-010-0318-x. URL https://doi.org/10.1007/s11263-010-0318-x
  107. doi:10.1145/2602142. URL https://doi.org/10.1145/2602142
  108. doi:10.1109/TVCG.2015.2410272.
  109. doi:10.1109/TCSVT.2020.3023781.
  110. doi:10.1109/TCSVT.2017.2718225.
  111. doi:10.1109/TCSVT.2014.2339591.
  112. doi:10.1109/TPAMI.2016.2611662.
  113. doi:10.1109/CVPR.2017.628.
  114. doi:10.1109/TCSVT.2019.2935838.
  115. doi:10.1109/CVPR.2017.16.
  116. doi:10.1109/CVPR.2019.00030.
  117. doi:10.1109/TPAMI.2020.3048013.
  118. doi:10.1109/TCSVT.2021.3068761.
  119. doi:10.1109/TPAMI.2010.147.
  120. doi:10.1109/TPAMI.2020.3016711.
  121. doi:10.1109/ICCV48922.2021.00615.
  122. doi:10.1109/ICCV.1999.791231.
  123. doi:10.1109/CVPR.2008.4587793.
  124. doi:10.1109/34.166625.
  125. doi:10.1109/34.784291.
  126. doi:10.1109/34.908965.
  127. doi:10.1109/TPAMI.2012.41.
  128. doi:10.1109/ICCV.2011.6126266.
  129. doi:10.1109/ICCV.2013.291.
  130. doi:https://doi.org/10.1016/j.patrec.2018.02.028. URL https://www.sciencedirect.com/science/article/pii/S0167865518300692
  131. doi:10.1109/IROS40897.2019.8968482.
  132. doi:10.1109/34.862199.
  133. doi:10.1109/ICRA.2017.7989233. URL https://doi.org/10.1109/ICRA.2017.7989233
  134. doi:10.1109/ICCV48922.2021.00600.
  135. doi:10.1109/LRA.2021.3111760.
  136. doi:10.1109/ICCV.2019.00447.
  137. doi:10.1109/CVPR52688.2022.00808.
  138. doi:10.1109/ICCV48922.2021.01567.
  139. doi:10.1109/CVPR.2012.6248018.
  140. doi:10.1109/IROS.2018.8594362.
  141. doi:10.1109/IROS45743.2020.9341690.
  142. doi:10.1109/IROS.2014.6942558.
  143. doi:10.1109/IROS40897.2019.8968033.
  144. doi:10.1109/ICRA40945.2020.9197022.
  145. doi:10.1109/TRO.2015.2463671.
  146. doi:10.1109/TRO.2017.2705103.
  147. doi:10.1109/ICRA48506.2021.9561078. URL https://doi.org/10.1109/ICRA48506.2021.9561078
  148. doi:10.1109/CVPR.2018.00931.
  149. doi:10.1109/ICRA48506.2021.9561864.
  150. doi:10.1109/ICRA.2019.8794415.
  151. doi:10.1109/ICCVW.2009.5457637.
  152. doi:10.1109/CVPR46437.2021.01570.
  153. doi:10.1109/TCSVT.2022.3208859.
  154. doi:10.1109/LRA.2022.3183899.
  155. doi:https://doi.org/10.1016/j.isprsjprs.2022.10.009. URL https://www.sciencedirect.com/science/article/pii/S0924271622002775
  156. doi:10.1109/CVPR46437.2021.00048.
  157. doi:10.1017/S0373463319000638.
  158. doi:10.1109/CVPRW.2008.4563135.
  159. doi:10.1109/IROS.2013.6696383.
  160. doi:10.1109/IROS51168.2021.9635923.
  161. doi:10.1109/TITS.2017.2752461.
  162. doi:10.1109/JSEN.2019.2929135.
  163. doi:10.1109/JAS.2021.1004293.
  164. doi:10.1109/ITSC.2018.8569274.
  165. doi:10.1109/IV47402.2020.9304659.
  166. doi:10.1109/ICRA48506.2021.9561459.
  167. doi:10.1109/IVS.2017.7995762.
  168. doi:10.1109/IROS45743.2020.9341003.
  169. doi:10.1109/TRO.2018.2853729.
  170. doi:10.1109/IVS.2014.6856560.
  171. doi:10.1007/s42154-021-00173-x.
  172. doi:10.1109/CVPR.2015.7298594.
  173. doi:10.1109/IROS.2017.8205957.
  174. doi:10.1109/ICRA.2016.7487679. URL https://doi.org/10.1109/ICRA.2016.7487679
  175. doi:10.1109/ICCV.2017.75.
  176. doi:10.1109/CVPR.2017.694.
  177. doi:10.1109/CVPR.2018.00277.
  178. doi:10.1109/ICASSP39728.2021.9414939.
  179. doi:10.1109/ICCV.2019.00293.
  180. doi:10.1109/CVPRW50498.2020.00027.
  181. doi:10.1109/ICCV48922.2021.00273.
  182. doi:10.1109/ISMAR.2013.6671777.
  183. doi:10.1109/CVPR.2013.377.
  184. doi:10.1109/CVPR.2014.146.
  185. doi:10.1109/CVPR.2015.7299069.
  186. doi:10.1109/CVPR.2017.267.
  187. doi:10.1109/TPAMI.2021.3070754.
  188. doi:10.1109/CVPR42600.2020.01200.
  189. doi:10.1109/3DV57658.2022.00051.
  190. doi:10.1109/CVPR46437.2021.00187.
  191. doi:10.1109/CVPR42600.2020.00497.
  192. doi:10.1109/CVPR.2018.00489.
  193. doi:10.1109/ICCV.2019.00762.
  194. doi:10.1109/CVPR52729.2023.00405.
  195. doi:10.1109/3DV53792.2021.00125.
  196. doi:10.1109/CVPR46437.2021.00713.
  197. doi:10.1109/LRA.2018.2859916.
  198. doi:10.1109/TPAMI.2015.2409868.
  199. doi:10.1109/CVPR42600.2020.00270.
  200. doi:10.1109/ROBOT.2010.5509547.
  201. doi:10.1109/CRV.2016.38.
  202. doi:10.1109/TITS.2020.3001228.
  203. doi:10.1109/CVPR.2016.352.
  204. doi:10.1177/0278364919843996.
  205. doi:10.1109/TNNLS.2019.2908982.
  206. doi:https://doi.org/10.1016/j.patcog.2021.107952. URL https://www.sciencedirect.com/science/article/pii/S0031320321001394
  207. doi:10.1109/CVPR.2017.598.
  208. doi:10.1177/0278364916679498.
  209. doi:https://doi.org/10.1016/j.neucom.2022.09.127. URL https://www.sciencedirect.com/science/article/pii/S0925231222012188
  210. doi:10.1109/CVPR.2016.90.
  211. doi:10.1109/TRO.2019.2926475.
  212. doi:https://doi.org/10.1002/rob.21992.
  213. doi:10.1109/IROS40897.2019.8968043. URL https://doi.org/10.1109/IROS40897.2019.8968043
  214. doi:10.1109/ICRA48506.2021.9561126.
  215. doi:10.1177/0278364917740639. URL https://doi.org/10.1177/0278364917740639
  216. doi:10.1109/CVPR46437.2021.00464.
  217. doi:10.1109/ACCESS.2019.2947287.
  218. doi:10.1109/TIV.2022.3173662.
  219. doi:10.1109/ICCV.2019.00296.
  220. doi:10.1109/TPAMI.2019.2941876.
  221. doi:10.1109/LRA.2022.3185367.
  222. doi:10.1109/LRA.2023.3326697.
Citations (2)
View on Semantic Scholar

Summary

  • The paper systematically reviews and compares various map representations used to enhance 6-DOF pose estimation.
  • It evaluates methodologies from geo-tagged frame matching to neural network-based implicit mapping with detailed benchmark results.
  • The survey highlights future trends such as end-to-end optimization and sensor fusion to advance autonomous navigation.

Understanding MRL Techniques: A Review of Monocular Re-Localization Methods

Background of MRL

Monocular Re-Localization (MRL) is an essential process in autonomous applications such as robotics navigation and autonomous driving. The goal of MRL is to determine the 6 degrees of freedom (DOF) pose, which includes both orientation and position, relative to a scene map using a single camera image. Over the years, substantial progress has been made, leading to the development of various techniques characterized by their robustness and precision.

Variety of Map Representations

Central to MRL research is how the scene map is represented. The map plays a critical role, affecting both the method's approach to localization and its performance. Traditionally, MRL has used geo-tagged frames, visual landmarks, point clouds, or vectorized semantic maps. Recently, methods have emerged that leverage neural networks to represent the map implicitly, showcasing the potential of deep learning in the field.

Geo-Tagged Frame Map

Geo-tagged frames, where individual images are annotated with their geographical positions, underpin methods like Visual Place Recognition (VPR) and Relative Pose Regression (RPR). VPR involves retrieving the most visually similar reference frame to a query image, using techniques such as image descriptors and attention mechanisms for image pair matching. RPR, on the other hand, aims to estimate the relative pose between a query image and a reference image and has been improved through deep learning networks that directly predict the pose.

Visual Landmark Map

Visual landmarks—salient 3D points associated with 2D image descriptors—serve as the basis for the Hierarchical Localization (HLoc) framework. Featured as a two-step process, HLoc uses local feature extraction and matching, followed by pose solving via Perspective-n-Point (PnP) algorithms. Recent advances in local feature extraction and matching, leveraging deep neural networks, have aimed at enhancing the accuracy and efficiency of local feature matching in visual landmark-based MRL.

Point Cloud and HD Maps

For LiDAR-generated point cloud maps, MRL methods focus on image-to-point cloud registration—a process that aligns a monocular camera image with a 3D point cloud map. On the other hand, HD maps are compact representations with semantic map elements that are favorable in the context of autonomous driving due to their light storage requirements. Learning-based and tightly-coupled VO methods are leading towards more accurate and real-time HD map-based MRL.

NN-based Implicit Map

Neural network-based methods such as APR and SCR represent the scene map implicitly, embedded within the network parameters. APR aims to learn direct pose prediction from images while SCR predicts 3D coordinates for each pixel in the image. NeRF, representing scenes as neural radiance fields, has been a groundbreaking approach, enabling view synthesis and high-precision localization.

Benchmarking MRL Performance

Evaluations across various benchmarks, including Aachen Day-Night, InLoc, KITTI, and 7Scenes datasets, have shown diversity in the performance of MRL methods. Methods are assessed based on retrieval accuracy for IPA tasks, pose estimation accuracy for FPE tasks, map size, and processing efficiency. The evaluation results showcase the effectiveness of each approach in achieving different levels of localization accuracy.

Future of Visual Localization

As MRL continues to evolve, discussion revolves around the future of the domain. Key trends include the shift towards end-to-end pipeline optimization, resource-friendly methods that consider computational and storage constraints, novel map representations that adapt to dynamic environments, and the integration of multiple sensor data to enhance robustness and accuracy. The evolving landscape of MRL is steering towards solutions that balance precision with practical demands, enabling broader adoption of autonomous systems in real-world applications.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up