Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Multimodal and Force-Matched Imitation Learning with a See-Through Visuotactile Sensor (2311.01248v4)

Published 2 Nov 2023 in cs.RO, cs.AI, and cs.LG

Abstract: Contact-rich tasks continue to present many challenges for robotic manipulation. In this work, we leverage a multimodal visuotactile sensor within the framework of imitation learning (IL) to perform contact-rich tasks that involve relative motion (e.g., slipping and sliding) between the end-effector and the manipulated object. We introduce two algorithmic contributions, tactile force matching and learned mode switching, as complimentary methods for improving IL. Tactile force matching enhances kinesthetic teaching by reading approximate forces during the demonstration and generating an adapted robot trajectory that recreates the recorded forces. Learned mode switching uses IL to couple visual and tactile sensor modes with the learned motion policy, simplifying the transition from reaching to contacting. We perform robotic manipulation experiments on four door-opening tasks with a variety of observation and algorithm configurations to study the utility of multimodal visuotactile sensing and our proposed improvements. Our results show that the inclusion of force matching raises average policy success rates by 62.5%, visuotactile mode switching by 30.3%, and visuotactile data as a policy input by 42.5%, emphasizing the value of see-through tactile sensing for IL, both for data collection to allow force matching, and for policy execution to enable accurate task feedback. Project site: https://papers.starslab.ca/sts-il .

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. C. Chi, X. Sun, N. Xue, T. Li, and C. Liu, “Recent Progress in Technologies for Tactile Sensors,” Sensors, vol. 18, no. 4, p. 948, Apr. 2018.
  2. W. Yuan, S. Dong, and E. H. Adelson, “GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force,” Sensors, vol. 17, no. 12, p. 2762, Dec. 2017.
  3. A. Padmanabha, F. Ebert, S. Tian, R. Calandra, C. Finn, and S. Levine, “OmniTact: A Multi-Directional High-Resolution Touch Sensor,” in 2020 IEEE International Conference on Robotics and Automation (ICRA), May 2020, pp. 618–624.
  4. D. Ma, E. Donlon, S. Dong, and A. Rodriguez, “Dense Tactile Force Estimation using GelSlim and inverse FEM,” in 2019 International Conference on Robotics and Automation (ICRA), May 2019, pp. 5418–5424.
  5. F. R. Hogan, M. Jenkin, S. Rezaei-Shoshtari, Y. Girdhar, D. Meger, and G. Dudek, “Seeing Through your Skin: Recognizing Objects with a Novel Visuotactile Sensor,” Dec. 2020.
  6. F. R. Hogan, J.-F. Tremblay, B. H. Baghi, M. Jenkin, K. Siddiqi, and G. Dudek, “Finger-STS: Combined Proximity and Tactile Sensing for Robotic Manipulation,” IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 10 865–10 872, Oct. 2022.
  7. A. G. Billard, S. Calinon, and R. Dillmann, “Learning from Humans,” in Springer Handbook of Robotics, B. Siciliano and O. Khatib, Eds.   Cham: Springer International Publishing, 2016, pp. 1995–2014.
  8. T. Ablett, Y. Zhai, and J. Kelly, “Seeing All the Angles: Learning Multiview Manipulation Policies for Contact-Rich Tasks from Demonstrations,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’21), Prague, Czech Republic, Sept. 2021.
  9. K. Li, D. Chappell, and N. Rojas, “Immersive Demonstrations are the Key to Imitation Learning,” Jan. 2023.
  10. A. Pervez, A. Ali, J.-H. Ryu, and D. Lee, “Novel learning from demonstration approach for repetitive teleoperation tasks,” in 2017 IEEE World Haptics Conference (WHC).   Munich, Germany: IEEE, June 2017, pp. 60–65.
  11. K. Fischer, et al., “A comparison of types of robot control for programming by Demonstration,” in 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI), Mar. 2016, pp. 213–220.
  12. B. Akgun and K. Subramanian, “Robot Learning from Demonstration: Kinesthetic Teaching vs. Teleoperation,” Technical Report, 2011.
  13. B. Siciliano, “Force Control,” in Robotics: Modelling, Planning and Control, ser. Advanced Textbooks in Control and Signal Processing, L. Sciavicco, L. Villani, and G. Oriolo, Eds.   London: Springer, 2009, pp. 363–405.
  14. M. H. Raibert and J. J. Craig, “Hybrid Position/Force Control of Manipulators,” Journal of Dynamic Systems, Measurement, and Control, vol. 103, no. 2, pp. 126–133, June 1981.
  15. N. Hogan, “Impedance Control: An Approach to Manipulation,” in 1984 American Control Conference, June 1984, pp. 304–313.
  16. A. Jilani, “Direct Shape from Touch Sensing,” Master’s Thesis [in Progress], McGill University, Jan. 2024.
  17. A. Yamaguchi and C. G. Atkeson, “Implementing tactile behaviors using FingerVision,” in 2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids), Nov. 2017, pp. 241–248.
  18. A. M. Okamura, “Haptic Feedback in Robot-Assisted Minimally Invasive Surgery,” Current opinion in urology, vol. 19, no. 1, pp. 102–107, Jan. 2009.
  19. O. Limoyo, T. Ablett, and J. Kelly, “Learning Sequential Latent Variable Models from Multimodal Time Series Data,” in Intelligent Autonomous Systems 17, ser. Lecture Notes in Networks and Systems, I. Petrovic, E. Menegatti, and I. Marković, Eds.   Cham: Springer Nature Switzerland, 2023, pp. 511–528.
  20. J. Hansen, F. Hogan, D. Rivkin, D. Meger, M. Jenkin, and G. Dudek, “Visuotactile-RL: Learning Multimodal Manipulation Policies with Deep Reinforcement Learning,” in 2022 International Conference on Robotics and Automation (ICRA), May 2022, pp. 8298–8304.
  21. H. Li, et al., “See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation,” in 6th Annual Conference on Robot Learning, Nov. 2022.
  22. Y. Lin, J. Lloyd, A. Church, and N. F. Lepora, “Tactile Gym 2.0: Sim-to-Real Deep Reinforcement Learning for Comparing Low-Cost High-Resolution Robot Touch,” IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 10 754–10 761, Oct. 2022.
  23. Y. She, S. Wang, S. Dong, N. Sunil, A. Rodriguez, and E. Adelson, “Cable manipulation with a tactile-reactive gripper,” The International Journal of Robotics Research, vol. 40, no. 12-14, pp. 1385–1401, Dec. 2021.
  24. L. Villani and J. De Schutter, “Force Control,” in Springer Handbook of Robotics, ser. Springer Handbooks, B. Siciliano and O. Khatib, Eds.   Cham: Springer International Publishing, 2016, pp. 195–220.
  25. W. Yuan, “Tactile Measurement with a GelSight Sensor,” Master’s thesis, Massachusetts Institute of Technology, 2014.
  26. W. Kim, W. D. Kim, J.-J. Kim, C.-H. Kim, and J. Kim, “UVtac: Switchable UV Marker-Based Tactile Sensing Finger for Effective Force Estimation and Object Localization,” IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 6036–6043, July 2022.
  27. M. Bain and C. Sammut, “A Framework for Behavioural Cloning,” in Machine Intelligence 15.   Oxford University Press, 1996, pp. 103–129.
  28. P. Abbeel and A. Y. Ng, “Apprenticeship learning via inverse reinforcement learning,” in International Conference on Machine Learning (ICML’04).   Banff, Alberta, Canada: ACM Press, 2004.
  29. A. Mandlekar, et al., “What Matters in Learning from Offline Human Demonstrations for Robot Manipulation,” in Conference on Robot Learning, Nov. 2021.
  30. T. Zhang, et al., “Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation,” in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA’18).   Brisbane, QLD, Australia: IEEE, May 2018, pp. 5628–5635.
  31. T. Ablett, B. Chan, and J. Kelly, “Learning From Guided Play: Improving Exploration for Adversarial Imitation Learning With Simple Auxiliary Tasks,” IEEE Robotics and Automation Letters, vol. 8, no. 3, pp. 1263–1270, Mar. 2023.
  32. W.-D. Chang, S. Fujimoto, D. Meger, and G. Dudek, “Imitation Learning from Observation through Optimal Transport,” Oct. 2023.
  33. M. Orsini, et al., “What Matters for Adversarial Imitation Learning?” in Conference on Neural Information Processing Systems, June 2021.
  34. P. Kormushev, S. Calinon, and D. G. Caldwell, “Imitation Learning of Positional and Force Skills Demonstrated via Kinesthetic Teaching and Haptic Input,” Advanced Robotics, vol. 25, no. 5, pp. 581–603, Jan. 2011.
  35. Y. Chebotar, O. Kroemer, and J. Peters, “Learning robot tactile sensing for object manipulation,” in 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, Sept. 2014, pp. 3368–3375.
  36. I. Huang and R. Bajcsy, “Robot Learning from Demonstration with Tactile Signals for Geometry-Dependent Tasks,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   Las Vegas, NV, USA: IEEE, Oct. 2020, pp. 8323–8328.
  37. S. Dasari, et al., “RB2: Robotic Manipulation Benchmarking with a Twist,” in Thirty-Fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), Oct. 2021.
  38. Y. Wang, N. Figueroa, S. Li, A. Shah, and J. Shah, “Temporal Logic Imitation: Learning Plan-Satisficing Motion Policies from Demonstrations,” in 6th Annual Conference on Robot Learning, Aug. 2022.
  39. N. Figueroa, “Easy-kinesthetic-recording,” https://github.com/nbfigueroa/easy-kinesthetic-recording, Oct. 2023.
  40. G. Bradski, “The OpenCV library,” Dr. Dobb’s Journal of Software Tools, 2000.
  41. R. Tsai and R. Lenz, “A new technique for fully autonomous and efficient 3D robotics hand/eye calibration,” IEEE Transactions on Robotics and Automation, vol. 5, no. 3, pp. 345–358, June 1989.
  42. O. Limoyo, T. Ablett, F. Marić, L. Volpatti, and J. Kelly, “Self-Calibration of Mobile Manipulator Kinematic and Sensor Extrinsic Parameters Through Contact-Based Interaction,” in 2018 IEEE International Conference on Robotics and Automation (ICRA), May 2018, pp. 1–8.
  43. A. Gupta, et al., “Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Intervention,” in Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA’21), Apr. 2021.
  44. Y. Lin, A. S. Wang, G. Sutanto, A. Rai, and F. Meier, “Polymetis,” https://facebookresearch.github.io/fairo/polymetis/, 2021.
  45. S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of deep visuomotor policies,” Journal of Machine Learning Research, vol. 17, no. 39, pp. 1–40, 2016.
  46. K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16).   Las Vegas, NV, USA: IEEE, June 2016, pp. 770–778.
  47. O. Limoyo, B. Chan, F. Marić, B. Wagstaff, A. R. Mahmood, and J. Kelly, “Heteroscedastic Uncertainty for Robust Generative Latent Dynamics,” IEEE Robotics and Automation Letters, vol. 5, no. 4, pp. 6654–6661, Oct. 2020.
  48. P. de Haan, D. Jayaraman, and S. Levine, “Causal Confusion in Imitation Learning,” arXiv:1905.11979 [cs, stat], Nov. 2019.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now