Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Cherry-Picking with Reinforcement Learning : Robust Dynamic Grasping in Unstable Conditions (2303.05508v2)

Published 9 Mar 2023 in cs.RO

Abstract: Grasping small objects surrounded by unstable or non-rigid material plays a crucial role in applications such as surgery, harvesting, construction, disaster recovery, and assisted feeding. This task is especially difficult when fine manipulation is required in the presence of sensor noise and perception errors; errors inevitably trigger dynamic motion, which is challenging to model precisely. Circumventing the difficulty to build accurate models for contacts and dynamics, data-driven methods like reinforcement learning (RL) can optimize task performance via trial and error, reducing the need for accurate models of contacts and dynamics. Applying RL methods to real robots, however, has been hindered by factors such as prohibitively high sample complexity or the high training infrastructure cost for providing resets on hardware. This work presents CherryBot, an RL system that uses chopsticks for fine manipulation that surpasses human reactiveness for some dynamic grasping tasks. By integrating imprecise simulators, suboptimal demonstrations and external state estimation, we study how to make a real-world robot learning system sample efficient and general while reducing the human effort required for supervision. Our system shows continual improvement through 30 minutes of real-world interaction: through reactive retry, it achieves an almost 100% success rate on the demanding task of using chopsticks to grasp small objects swinging in the air. We demonstrate the reactiveness, robustness and generalizability of CherryBot to varying object shapes and dynamics (e.g., external disturbances like wind and human perturbations). Videos are available at https://goodcherrybot.github.io/.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Mark R Cutkosky. Robotic grasping and fine manipulation, volume 6. Springer Science & Business Media, 2012.
  2. Grasping with chopsticks: Combating covariate shift in model-free imitation learning for fine manipulation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6185–6191. IEEE, 2021.
  3. Dynamic manipulation. In Proceedings of 1993 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’93), volume 1, pages 152–159. IEEE, 1993.
  4. Trends and challenges in robot manipulation. Science, 364(6446), 2019.
  5. Col Michael R Marohn and Capt Eric J Hanly. Twenty-first century surgery using twenty-first century technology: Surgical robotics. Current Surgery, 61(5):466–473, 2004.
  6. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 17(12):2762, 2017.
  7. Towards robotic feeding: Role of haptics in fork-based food manipulation. IEEE Robotics and Automation Letters, 4(2):1485–1492, 2019.
  8. A vacuum-driven origami “magic-ball” soft gripper. In 2019 International Conference on Robotics and Automation (ICRA), pages 7401–7408. IEEE, 2019.
  9. Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. The International Journal of Robotics Research, 41(7):690–705, 2022.
  10. Automating surgical peg transfer: Calibration with deep learning can exceed speed, accuracy, and consistency of humans. IEEE Transactions on Automation Science and Engineering, 2022.
  11. Dynamic nonprehensile manipulation: Controllability, planning, and experiments. The International Journal of Robotics Research, 18(1):64–92, 1999.
  12. Thin-diameter chopsticks robot for laparoscopic surgery. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 4122–4127, 2016.
  13. Chopstick surgery: a novel technique improves surgeon performance and eliminates arm collision in robotic single-incision laparoscopic surgery. Surgical Endoscopy, 24(6):1331–1335, 2010.
  14. Autonomous foods handling by chopsticks for meal assistant robot. In ROBOTIK 2012; 7th German Conference on Robotics, pages 1–6. VDE, 2012.
  15. Developmental process of a chopstick-like hybrid-structure two-fingered micromanipulator hand for 3-d manipulation of microscopic objects. IEEE Transactions on Industrial Electronics, 56(4):1121–1135, 2009.
  16. Telemanipulation with chopsticks: Analyzing human factors in user demonstrations. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11539–11546. IEEE, 2020.
  17. Generality and simple hands. In Robotics Research, pages 345–361. Springer, 2011.
  18. The pincer chopsticks: The investigation of a new utensil in pinching function. Applied ergonomics, 38(3):385–390, 2007.
  19. Discovery of complex behaviors through contact-invariant optimization. ACM Transactions on Graphics (ToG), 31(4):1–8, 2012.
  20. Optimal control with learned local models: Application to dexterous manipulation. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 378–383. IEEE, 2016.
  21. Reactive planar non-prehensile manipulation with hybrid model predictive control. The International Journal of Robotics Research, 39(7):755–773, 2020.
  22. Stefan Schaal. Dynamic movement primitives-a framework for motor control in humans and humanoid robotics. Adaptive motion of animals and machines, pages 261–280, 2006.
  23. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv:1806.10293, 2018.
  24. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. In 2019 International Conference on Robotics and Automation (ICRA), pages 3651–3657. IEEE, 2019.
  25. The ingredients of real world robotic reinforcement learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreview.net/forum?id=rJe2syrtvS.
  26. Learning agile robotic locomotion skills by imitating animals. arXiv preprint arXiv:2004.00784, 2020.
  27. Dex-net 1.0: A cloud-based network of 3d objects for robust grasp planning using a multi-armed bandit model with correlated rewards. In IEEE International Conference on Robotics and Automation (ICRA), pages 1957–1964. IEEE, 2016.
  28. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning, pages 726–747. PMLR, 2021.
  29. Yale-CMU-Berkeley dataset for robotic manipulation research. The International Journal of Robotics Research, 36(3):261–268, 2017.
  30. A visuomotor control architecture for high-speed grasping. In Proceedings of the 40th IEEE Conference on Decision and Control (Cat. No. 01CH37228), volume 1, pages 15–20. IEEE, 2001.
  31. Learning to walk via deep reinforcement learning. arXiv preprint arXiv:1812.11103, 2018a.
  32. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018b.
  33. Trifinger: An open-source robot for learning dexterity. arXiv preprint arXiv:2008.03596, 2020.
  34. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087, 2017.
  35. Randomized ensembled double q-learning: Learning fast without a model. arXiv preprint arXiv:2101.05982, 2021.
  36. A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv:2208.07860, 2022.
  37. Batch reinforcement learning. Reinforcement learning: State-of-the-art, pages 45–73, 2012.
  38. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
  39. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019.
  40. Off-policy deep reinforcement learning without exploration. corr abs/1812.02900 (2018). arXiv preprint arXiv:1812.02900, 2018.
  41. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 33:21810–21823, 2020.
  42. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179–1191, 2020.
  43. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021.
  44. Real world offline reinforcement learning with realistic data source. arXiv preprint arXiv:2210.06479, 2022.
  45. Confidence-conditioned value functions for offline reinforcement learning. arXiv preprint arXiv:2212.04607, 2022.
  46. Visual servo control. I. Basic approaches. IEEE Robotics & Automation Magazine, 13(4):82–90, 2006.
  47. A tutorial on visual servo control. IEEE transactions on robotics and automation, 12(5):651–670, 1996.
  48. Adaptive visual servo control of robots. Robot vision, pages 107–116, 1983.
  49. Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4:265–293, 2021.
  50. Visually grounded task and motion planning for mobile manipulation. In 2022 International Conference on Robotics and Automation (ICRA), pages 1925–1931. IEEE, 2022.
  51. Iterative residual policy: for goal-conditioned dynamic manipulation of deformable objects. arXiv preprint arXiv:2203.00663, 2022.
  52. High-speed gaze controller for millisecond-order pan/tilt camera. In 2011 IEEE International Conference on Robotics and Automation, pages 6186–6191. IEEE, 2011.
  53. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3803–3810. IEEE, 2018.
  54. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  55. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012, Vilamoura, Algarve, Portugal, October 7-12, 2012, pages 5026–5033. IEEE, 2012. doi: 10.1109/IROS.2012.6386109. URL https://doi.org/10.1109/IROS.2012.6386109.
  56. Russ Tedrake. Underactuated Robotics. 2023. URL https://underactuated.csail.mit.edu.
  57. Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988.
  58. d3rlpy: An offline deep reinforcement learning library. Journal of Machine Learning Research, 23(315):1–20, 2022. URL http://jmlr.org/papers/v23/22-0017.html.
  59. Lyceum: An efficient and scalable ecosystem for robot learning. In Learning for Dynamics and Control, pages 793–803. PMLR, 2020.
  60. Learning by playing solving sparse reward tasks from scratch. In International conference on machine learning, pages 4344–4353. PMLR, 2018.
  61. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934, 2020.
Citations (7)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. Cherrybot
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets