Continual Policy Distillation of Reinforcement Learning-based Controllers for Soft Robotic In-Hand Manipulation (2404.04219v1)
Abstract: Dexterous manipulation, often facilitated by multi-fingered robotic hands, holds solid impact for real-world applications. Soft robotic hands, due to their compliant nature, offer flexibility and adaptability during object grasping and manipulation. Yet, benefits come with challenges, particularly in the control development for finger coordination. Reinforcement Learning (RL) can be employed to train object-specific in-hand manipulation policies, but limiting adaptability and generalizability. We introduce a Continual Policy Distillation (CPD) framework to acquire a versatile controller for in-hand manipulation, to rotate different objects in shape and size within a four-fingered soft gripper. The framework leverages Policy Distillation (PD) to transfer knowledge from expert policies to a continually evolving student policy network. Exemplar-based rehearsal methods are then integrated to mitigate catastrophic forgetting and enhance generalization. The performance of the CPD framework over various replay strategies demonstrates its effectiveness in consolidating knowledge from multiple experts and achieving versatile and adaptive behaviours for in-hand manipulation tasks.
- “Trends and challenges in robot manipulation” In Science 364, 2019 DOI: 10.1126/science.aat8414
- “Grasping and Control of Multi-Fingered Hands” In Advanced Bimanual Manipulation 80, 2012 DOI: 10.1007/978-3-642-29041-1˙5
- “A Century of Robotic Hands” In Annual Review of Control, Robotics, and Autonomous Systems 2, 2019 DOI: 10.1146/annurev-control-060117-105003
- “Soft Robotic Grippers” In Advanced Materials 30, 2018 DOI: 10.1002/adma.201707035
- “A Slip Detection and Correction Strategy for Precision Robot Grasping” In IEEE/ASME Transactions on Mechatronics 21.5, 2016 DOI: 10.1109/TMECH.2016.2551557
- S. Puhlmann, J. Harris and O. Brock “RBO Hand 3: A Platform for Soft Dexterous Manipulation” In IEEE Transactions on Robotics 38.6, 2022 DOI: 10.1109/TRO.2022.3156806
- O. Kroemer, S. Niekum and G. Konidaris “A review of robot learning for manipulation: challenges, representations, and algorithms” In The Journal of Machine Learning Research 22.1, 2021 DOI: 10.5555/3546258.3546288
- “Continual learning for recurrent neural networks: An empirical evaluation” In Neural Networks 143, 2021 DOI: 10.1016/j.neunet.2021.07.021
- “Continual Learning for Robotics: Definition, Framework, Learning Strategies, Opportunities and Challenges” In Information Fusion 58, 2020, pp. 52–68 DOI: 10.1016/j.inffus.2019.12.004
- “Learning-Based Control Strategies for Soft Robots: Theory, Achievements, and Future Challenges” In IEEE Control Systems Magazine 43.3, 2023 DOI: 10.1109/MCS.2023.3253421
- Muhammad Sunny Nazeer, Cecilia Laschi and Egidio Falotico “Soft DAgger: Sample-Efficient Imitation Learning for Control of Soft Robots” In Sensors 23.19, 2023 DOI: 10.3390/s23198278
- “Closed-Loop Dynamic Control of a Soft Manipulator Using Deep Reinforcement Learning” In IEEE Robotics and Automation Letters 7.2, 2023 DOI: 10.1109/LRA.2022.3146903
- “SofToss: Learning to Throw Objects With a Soft Robot” In IEEE Robotics and Automation Magazine, 2023, pp. 2–12 DOI: 10.1109/MRA.2023.3310865
- “To Enabling Plant-like Movement Capabilities in Continuum Arms” In 2022 I-RIM Conference DOI: 10.5281/zenodo.7531338
- “Plant-inspired behavior-based controller to enable reaching in redundant continuum robot arms” In 2023 IEEE RoboSoft DOI: 10.1109/RoboSoft55895.2023.10122017
- “Model-Based Reinforcement Learning for Closed-Loop Dynamic Control of Soft Robotic Manipulators” In IEEE Transactions on Robotics 35.1, 2019 DOI: 10.1109/TRO.2018.2878318
- “Dexterous Manipulation for Multi-Fingered Robotic Hands With Reinforcement Learning: A Review” In Frontiers in Neurorobotics 16, 2022 DOI: 10.3389/fnbot.2022.861825
- G. Hinton, O. Vinyals and J. Dean “Distilling the Knowledge in a Neural Network” In arXiv, 2015 DOI: 10.48550/arXiv.1503.02531
- “Policy Distillation” In arXiv, 2016 DOI: 10.48550/arXiv.1511.06295
- “DROID: Learning from Offline Heterogeneous Demonstrations via Reward-Policy Distillation” In 2023 Conference on Robot Learning
- “Controlling Soft Robotic Arms Using Continual Learning” In IEEE Robotics and Automation Letters 7.2, 2022 DOI: 10.1109/LRA.2022.3157369
- “DisCoRL: Continual Reinforcement Learning via Policy Distillation” In arXiv, 2019 DOI: 10.48550/arXiv.1907.05855
- “SoMo: Fast and Accurate Simulations of Continuum Robots in Complex Environments” In IEEE-RSJ IROS, 2021 DOI: 10.1109/IROS51168.2021.9636059
- “SoMoGym: A Toolkit for Developing and Evaluating Controllers and Reinforcement Learning Algorithms for Soft Robots” In IEEE Robotics and Automation Letters 7.2, 2022 DOI: 10.1109/LRA.2022.3149580
- “PyBullet, a Python module for physics simulation for games, robotics and machine learning”, 2020
- “Stable-Baselines3: Reliable Reinforcement Learning Implementations” In Journal of Machine Learning Research 22.268, 2021 DOI: 10.5555/3546258.3546526
- “Avalanche: An End-to-End Library for Continual Learning” In IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021 DOI: 10.1109/CVPRW53098.2021.00399
- G.M. van de Ven and A.S. Tolias “Three Scenarios for Continual Learning” In arXiv, 2019 DOI: 10.48550/arXiv.1904.07734
- “On Tiny Episodic Memories in Continual Learning” In arXiv, 2019 DOI: 10.48550/arXiv.1902.10486
- “Ex-Model: Continual Learning from a Stream of Trained Models” In IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2022 DOI: 10.1109/CVPRW56347.2022.00424
- Lanpei Li (4 papers)
- Enrico Donato (5 papers)
- Vincenzo Lomonaco (58 papers)
- Egidio Falotico (13 papers)