Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 190 tok/s
Gemini 2.5 Pro 51 tok/s Pro
GPT-5 Medium 39 tok/s Pro
GPT-5 High 46 tok/s Pro
GPT-4o 130 tok/s Pro
Kimi K2 202 tok/s Pro
GPT OSS 120B 439 tok/s Pro
Claude Sonnet 4.5 37 tok/s Pro
2000 character limit reached

DeepIPC: Deeply Integrated Perception and Control for an Autonomous Vehicle in Real Environments (2207.09934v7)

Published 20 Jul 2022 in cs.RO, cs.AI, and cs.CV

Abstract: In this work, we introduce DeepIPC, a novel end-to-end model tailored for autonomous driving, which seamlessly integrates perception and control tasks. Unlike traditional models that handle these tasks separately, DeepIPC innovatively combines a perception module, which processes RGBD images for semantic segmentation and generates bird's eye view (BEV) mappings, with a controller module that utilizes these insights along with GNSS and angular speed measurements to accurately predict navigational waypoints. This integration allows DeepIPC to efficiently translate complex environmental data into actionable driving commands. Our comprehensive evaluation demonstrates DeepIPC's superior performance in terms of drivability and multi-task efficiency across diverse real-world scenarios, setting a new benchmark for end-to-end autonomous driving systems with a leaner model architecture. The experimental results underscore DeepIPC's potential to significantly enhance autonomous vehicular navigation, promising a step forward in the development of autonomous driving technologies. For further insights and replication, we will make our code and datasets available at https://github.com/oskarnatan/DeepIPC.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. O. Natan and J. Miura, “End-to-end autonomous driving with semantic depth cloud mapping and multi-agent,” IEEE Trans. Intell. Veh., vol. 8, no. 1, pp. 557–571, Jan. 2022.
  2. L. Le Mero, D. Yi, M. Dianati, and A. Mouzakitis, “A survey on imitation learning techniques for end-to-end autonomous vehicles,” IEEE Trans. Intell. Transp. Syst., vol. 23, no. 9, pp. 14 128–14 147, Sept. 2022.
  3. S. Moten, F. Celiberti, M. Grottoli, A. van der Heide, and Y. Lemmens, “X-in-the-loop advanced driving simulation platform for the design, development, testing and validation of ADAS,” in Proc. IEEE Intell. Veh. Symp. (IV), Changshu, China, June 2018, pp. 1–6.
  4. T. Wu, A. Luo, R. Huang, H. Cheng, and Y. Zhao, “End-to-end driving model for steering control of autonomous vehicles with future spatiotemporal features,” in Proc. IEEE/RSJ Inter. Conf. Intell. Robots and Syst. (IROS), Macau, China, Nov. 2019, pp. 950–955.
  5. D. Omeiza, H. Web, M. Jirotka, and L. Kunze, “Towards accountability: Providing intelligible explanations in autonomous driving,” in Proc. IEEE Intell. Veh. Symp. (IV), Nagoya, Japan, July 2021, pp. 231–237.
  6. D. Feng, C. Haase-Schütz, L. Rosenbaum, H. Hertlein, C. Gläser, F. Timm, W. Wiesbeck, and K. Dietmayer, “Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges,” IEEE Trans. Intell. Transp. Syst., vol. 22, no. 3, pp. 1341–1360, Mar. 2021.
  7. M. Teti, W. E. Hahn, S. Martin, C. Teti, and E. Barenholtz, “A controlled investigation of behaviorally-cloned deep neural network behaviors in an autonomous steering task,” Robotics and Autonomous Systems, vol. 142, p. 103780, Aug. 2021.
  8. A. Amini, I. Gilitschenski, J. Phillips, J. Moseyko, R. Banerjee, S. Karaman, and D. Rus, “Learning robust control policies for end-to-end autonomous driving from data-driven simulation,” IEEE Robot. and Autom. Lett., vol. 5, no. 2, pp. 1143–1150, Apr. 2020.
  9. A. Ngo, M. P. Bauer, and M. Resch, “A multi-layered approach for measuring the simulation-to-reality gap of radar perception for autonomous driving,” in Proc. IEEE Intell. Transp. Syst. Conf. (ITSC), Indianapolis, USA, Sept. 2021, pp. 4008–4014.
  10. J. Zhou, R. Wang, X. Liu, Y. Jiang, S. Jiang, J. Tao, J. Miao, and S. Song, “Exploring imitation learning for autonomous driving with feedback synthesizer and differentiable rasterization,” in Proc. IEEE/RSJ Inter. Conf. Intell. Robots and Syst. (IROS), Prague, Czech Republic, Sept. 2021, pp. 1450–1457.
  11. J. Hawke, R. Shen, C. Gurau, S. Sharma, D. Reda, N. Nikolov, P. Mazur, S. Micklethwaite, N. Griffiths, A. Shah, and A. Kndall, “Urban driving with conditional imitation learning,” in Proc. IEEE Inter. Conf. Robot. and Autom. (ICRA), Paris, France, Aug. 2020, pp. 251–257.
  12. B. D. Argall, S. Chernova, M. Veloso, and B. Browning, “A survey of robot learning from demonstration,” Robotics and Autonomous Systems, vol. 57, no. 5, pp. 469–483, May 2009.
  13. S. Teng, X. Hu, P. Deng, B. Li, Y. Li, Y. Ai, D. Yang, L. Li, Z. Xuanyuan, F. Zhu, and L. Chen, “Motion planning for autonomous driving: The state of the art and future perspectives,” IEEE Transactions on Intelligent Vehicles, vol. 8, no. 6, pp. 3692–3711, June 2023.
  14. Y. Xu, R. Zheng, S. Zhang, M. Liu, and J. Yu, “Uncertainty-aware autonomous robot exploration using confidence-rich localization and mapping,” IEEE Transactions on Automation Science and Engineering, pp. 1–15, Feb. 2024.
  15. D. Xu, Z. Ding, X. He, H. Zhao, M. Moze, F. Aioun, and F. Guillemard, “Learning from naturalistic driving data for human-like autonomous highway driving,” IEEE Trans. Intell. Transp. Syst., vol. 22, no. 12, pp. 7341–7354, Dec. 2021.
  16. H. Ma, Y. Wang, R. Xiong, S. Kodagoda, and L. Tang, “DeepGoal: Learning to drive with driving intention from human control demonstration,” Robotics and Autonomous Systems, vol. 127, p. 103477, May 2020.
  17. S. Matsuzaki, J. Miura, and H. Masuzawa, “Multi-source pseudo-label learning of semantic segmentation for the scene recognition of agricultural mobile robots,” Advanced Robotics, vol. 36, no. 19, pp. 1011–1029, Aug. 2022.
  18. Y. Liu and J. Miura, “RDS-SLAM: Real-time dynamic SLAM using semantic segmentation methods,” IEEE Access, vol. 9, pp. 23 772–23 785, Jan. 2021.
  19. S. Matsuzaki, H. Masuzawa, and J. Miura, “Image-based scene recognition for robot navigation considering traversable plants and its manual annotation-free training,” IEEE Access, vol. 10, pp. 5115–5128, 2022.
  20. K. Minami, K. Hayashi, and J. Miura, “Development of the pedestrian awareness model for mobile robots*,” in IEEE International Conf. Robot and Human Interactive Communication (RO-MAN), Busan, South Korea, Nov. 2023, pp. 1295–1301.
  21. M. Hahner, D. Dai, C. Sakaridis, J.-N. Zaech, and L. V. Gool, “Semantic understanding of foggy scenes with purely synthetic data,” in Proc. IEEE Intell. Transp. Syst. Conf. (ITSC), Auckland, New Zealand, Oct. 2019, pp. 3675–3681.
  22. R. N. Rajaram, E. Ohn-Bar, and M. M. Trivedi, “RefineNet: Refining object detectors for autonomous driving,” IEEE Trans. Intell. Veh., vol. 1, no. 4, pp. 358–368, Dec. 2016.
  23. J. Wang, X. Wang, T. Shen, Y. Wang, L. Li, Y. Tian, H. Yu, L. Chen, J. Xin, X. Wu, N. Zheng, and F.-Y. Wang, “Parallel vision for long-tail regularization: Initial results from IVFC autonomous driving testing,” IEEE Trans. Intell. Veh., vol. 7, no. 2, pp. 286–299, June 2022.
  24. B. Ranft and C. Stiller, “The role of machine vision for intelligent vehicles,” IEEE Trans. Intell. Veh., vol. 1, no. 1, pp. 8–19, Mar. 2016.
  25. J. Yoo and R. Langari, “A predictive perception model and control strategy for collision-free autonomous driving,” IEEE Trans. Intell. Transp. Syst., vol. 20, no. 11, pp. 4078–4091, Nov. 2019.
  26. K. Ishihara, A. Kanervisto, J. Miura, and V. Hautamaki, “Multi-task learning with attention for end-to-end autonomous driving,” in Proc. IEEE/CVF Conf. Comput. Vision and Pattern Recog. Workshops (CVPRW), Nashville, USA, June 2021, pp. 2896–2905.
  27. K. Chitta, A. Prakash, and A. Geiger, “NEAT: Neural attention fields for end-to-end autonomous driving,” in Proc. IEEE/CVF Inter. Conf. Comput. Vision (ICCV), Montreal, Canada, Oct. 2021, pp. 15 773–15 783.
  28. O. Natan and J. Miura, “Semantic segmentation and depth estimation with RGB and DVS sensor fusion for multi-view driving perception,” in Proc. Asian Conf. Pattern Recog. (ACPR), Jeju Island, South Korea, Nov. 2021, pp. 352–365.
  29. T. Suzuki, K. Ohno, S. Kojima, N. Miyamoto, T. Suzuki, T. Komatsu, Y. Shibata, K. Asano, and K. Nagatani, “Estimation of articulated angle in six-wheeled dump trucks using multiple gnss receivers for autonomous driving,” Advanced Robotics, vol. 35, no. 23, pp. 1376–1387, Sept. 2021.
  30. A. Prakash, K. Chitta, and A. Geiger, “Multi-modal fusion transformer for end-to-end autonomous driving,” in Proc. IEEE/CVF Conf. Comput. Vision and Pattern Recog. (CVPR), Nashville, USA, June 2021, pp. 7073–7083.
  31. K. Chitta, A. Prakash, B. Jaeger, Z. Yu, K. Renz, and A. Geiger, “TransFuser: Imitation with transformer-based sensor fusion for autonomous driving,” IEEE Trans. Pattern Anal. Mach. Intell., 2022.
  32. U. Niesen and J. Unnikrishnan, “Camera-radar fusion for 3-D depth reconstruction,” in Proc. IEEE Intell. Veh. Symp. (IV), Las Vegas, USA, Oct. 2020, pp. 265–271.
  33. M. Shan, Y. Zou, M. Guan, C. Wen, and C.-L. Ng, “A leader-following approach based on probabilistic trajectory estimation and virtual train model,” in Proc. IEEE Intell. Transp. Syst. Conf. (ITSC), Yokohama, Japan, Oct. 2017, pp. 1–6.
  34. Z. Huang, C. Lv, Y. Xing, and J. Wu, “Multi-modal sensor fusion-based deep neural network for end-to-end autonomous driving with scene understanding,” IEEE Sensors J., vol. 21, no. 10, pp. 11 781–11 790, May 2021.
  35. F. Sasaki, T. Yohira, and A. Kawaguchi, “Adversarial behavioral cloning,” Advanced Robotics, vol. 34, no. 9, pp. 592–598, Feb. 2020.
  36. H. Shen, W. Wan, and H. Wang, “Learning category-level generalizable object manipulation policy via generative adversarial self-imitation learning from demonstrations,” IEEE Robot. and Autom. Lett., vol. 7, no. 4, pp. 11 166–11 173, Oct. 2022.
  37. D.-T. Pham, T.-N. Tran, S. Alam, and V. N. Duong, “A generative adversarial imitation learning approach for realistic aircraft taxi-speed modeling,” IEEE Trans. Intell. Transp. Syst., vol. 23, no. 3, pp. 2509–2522, Mar. 2022.
  38. X. Fang, Q. Zhang, Y. Gao, and D. Zhao, “Offline reinforcement learning for autonomous driving with real world driving data,” in Proc. IEEE Intell. Transp. Syst. Conf. (ITSC), Macau, China, Oct. 2022, pp. 3417–3422.
  39. R. Bhattacharyya, B. Wulfe, D. J. Phillips, A. Kuefler, J. Morton, R. Senanayake, and M. J. Kochenderfer, “Modeling human driving behavior through generative adversarial imitation learning,” IEEE Trans. Intell. Transp. Syst., vol. 24, no. 3, pp. 2874–2887, Mar. 2023.
  40. P. Cai, H. Wang, H. Huang, Y. Liu, and M. Liu, “Vision-based autonomous car racing using deep imitative reinforcement learning,” IEEE Robot. and Autom. Lett., vol. 6, no. 4, pp. 7262–7269, Oct. 2021.
  41. A. Chatty, P. Gaussier, S. K. Hasnain, I. Kallel, and A. M. Alimi, “The effect of learning by imitation on a multi-robot system based on the coupling of low-level imitation strategy and online learning for cognitive map building,” Advanced Robotics, vol. 28, no. 11, pp. 731–743, Feb. 2014.
  42. S. Hoshino and K. Unuma, “End-to-end motion planners through multi-task learning for mobile robots with 2D lidar,” in Proc. IEEE/SICE Inter. Symp. Syst. Integration (SII), Atlanta, USA, Jan. 2023, pp. 1–6.
  43. S. Yan, Z. Wu, J. Wang, Y. Huang, M. Tan, and J. Yu, “Real-world learning control for autonomous exploration of a biomimetic robotic shark,” IEEE Trans. Ind. Electron., vol. 70, no. 4, pp. 3966–3974, Apr. 2023.
  44. O. Natan, D. U. K. Putri, and A. Dharmawan, “Deep learning-based weld spot segmentation using modified UNet with various convolutional blocks,” ICIC Express Lett. Part B: Applications, vol. 12, no. 12, pp. 1169–1176, Dec. 2021.
  45. R. Araki, T. Hirakawa, T. Yamashita, and H. Fujiyoshi, “MT-DSSD: Multi-task deconvolutional single shot detector for object detection, segmentation, and grasping detection,” Advanced Robotics, vol. 36, no. 8, pp. 373–387, Mar. 2022.
  46. H. Masuzawa and J. Miura, “Image-based recognition of green perilla leaves using a deep neural network for robotic harvest support,” Advanced Robotics, vol. 35, no. 6, pp. 359–367, Jan. 2021.
  47. M. Tan and Q. Le, “EfficientNet: Rethinking model scaling for convolutional neural networks,” in Proc. Inter. Conf. Machine Learning (ICML), Long Beach, USA, June 2019, pp. 6105–6114.
  48. S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in Proc. Inter. Conf. Machine Learning (ICML), Lille, France, July 2015, pp. 448–456.
  49. V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in Proc. Inter. Conf. Machine Learning (ICML), Haifa, Israel, June 2010, pp. 807–814.
  50. K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio, “On the properties of neural machine translation: Encoder-decoder approaches,” in Proc. Workshop Syntax, Semantics and Structure in Statistical Translation (SSST), Doha, Qatar, Oct. 2014, pp. 103–111.
  51. O. Natan and J. Miura, “Towards compact autonomous driving perception with balanced learning and multi-sensor fusion,” IEEE Trans. Intell. Transp. Syst., vol. 23, no. 9, pp. 16 249–16 266, Sept. 2022.
  52. A. O. Ly and M. Akhloufi, “Learning to drive by imitation: An overview of deep behavior cloning methods,” IEEE Trans. Intell. Veh., vol. 6, no. 2, pp. 195–209, June 2021.
  53. G. A. G. Ricardez, N. Koganti, P.-C. Yang, S. Okada, P. M. U. Eljuri, A. Yasuda, L. E. Hafi, M. Yamamoto, J. Takamatsu, and T. Ogasawara, “Adaptive motion generation using imitation learning and highly compliant end effector for autonomous cleaning,” Advanced Robotics, vol. 34, no. 3-4, pp. 189–201, Dec. 2020.
  54. H. Fujiishi, T. Kobayashi, and K. Sugimoto, “Safe and efficient imitation learning by clarification of experienced latent space,” Advanced Robotics, vol. 35, no. 16, pp. 1012–1027, July 2021.
  55. Y. Uzawa, S. Matsuzaki, H. Masuzawa, and J. Miura, “Dataset generation for deep visual navigation in unstructured environments,” in Proc. European Conf. Mobile Robots (ECMR), Coimbra, Portugal, Sept. 2023, pp. 1–6.
  56. E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “SegFormer: Simple and efficient design for semantic segmentation with transformers,” in Proc. Inter. Conf. Neural Information Processing Syst. (NIPS), Online, Dec. 2021, pp. 1–18.
  57. M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in Proc. IEEE/CVF Conference on Comput. Vision and Pattern Recog. (CVPR), Las Vegas, USA, June 2016, pp. 3213–3223.
  58. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “PyTorch: An imperative style, high performance deep learning library,” in Proc. Inter. Conf. Neural Information Processing Syst. (NIPS), Vancouver, Canada, Dec. 2019, pp. 8024–8035.
  59. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. Inter. Conf. Learning Representations (ICLR), San Diego, USA, May 2015, pp. 1–15.
  60. I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. Inter. Conf. Learning Representations (ICLR), New Orleans, USA, May 2019, pp. 1–10.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Authors (2)

  1. Oskar Natan
  2. Jun Miura
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - oskarnatan/DeepIPC (2 stars)
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube