Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Contact Energy Based Hindsight Experience Prioritization (2312.02677v2)

Published 5 Dec 2023 in cs.RO and cs.AI

Abstract: Multi-goal robot manipulation tasks with sparse rewards are difficult for reinforcement learning (RL) algorithms due to the inefficiency in collecting successful experiences. Recent algorithms such as Hindsight Experience Replay (HER) expedite learning by taking advantage of failed trajectories and replacing the desired goal with one of the achieved states so that any failed trajectory can be utilized as a contribution to learning. However, HER uniformly chooses failed trajectories, without taking into account which ones might be the most valuable for learning. In this paper, we address this problem and propose a novel approach Contact Energy Based Prioritization~(CEBP) to select the samples from the replay buffer based on rich information due to contact, leveraging the touch sensors in the gripper of the robot and object displacement. Our prioritization scheme favors sampling of contact-rich experiences, which are arguably the ones providing the largest amount of information. We evaluate our proposed approach on various sparse reward robotic tasks and compare them with the state-of-the-art methods. We show that our method surpasses or performs on par with those methods on robot manipulation tasks. Finally, we deploy the trained policy from our method to a real Franka robot for a pick-and-place task. We observe that the robot can solve the task successfully. The videos and code are publicly available at: https://erdiphd.github.io/HER_force

Definition Search Book Streamline Icon: https://streamlinehq.com
References (32)
  1. L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement learning: A survey,” Journal of artificial intelligence research, 1996.
  2. V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” arXiv preprint arXiv:1312.5602, 2013.
  3. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., “Human-level control through deep reinforcement learning,” nature, 2015.
  4. D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al., “Mastering the game of go with deep neural networks and tree search,” nature, 2016.
  5. A. Y. Ng, H. J. Kim, M. I. Jordan, S. Sastry, and S. Ballianda, “Autonomous helicopter flight via reinforcement learning.” in NIPS, vol. 16.   Citeseer, 2003.
  6. A. Y. Ng, A. Coates, M. Diel, V. Ganapathi, J. Schulte, B. Tse, E. Berger, and E. Liang, “Autonomous inverted helicopter flight via reinforcement learning.”   Springer, 2006.
  7. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” 2019.
  8. A. Y. Ng, D. Harada, and S. Russell, “Policy invariance under reward transformations: Theory and application to reward shaping,” in ICML, 1999.
  9. M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba, “Hindsight experience replay,” CoRR, vol. abs/1707.01495, 2017. [Online]. Available: http://arxiv.org/abs/1707.01495
  10. T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience replay,” arXiv preprint arXiv:1511.05952, 2015.
  11. R. Zhao, X. Sun, and V. Tresp, “Maximum entropy-regularized multi-goal reinforcement learning,” in International Conference on Machine Learning, 2019, pp. 7553–7562.
  12. M. Plappert, M. Andrychowicz, A. Ray, B. McGrew, B. Baker, G. Powell, J. Schneider, J. Tobin, M. Chociej, P. Welinder, et al., “Multi-goal reinforcement learning: Challenging robotics environments and request for research,” arXiv preprint arXiv:1802.09464, 2018.
  13. H. Merzic, M. Bogdanovic, D. Kappler, L. Righetti, and J. Bohg, “Leveraging contact forces for learning to grasp,” 2018.
  14. C. Wang, S. Wang, B. Romero, F. Veiga, and E. Adelson, “Swingbot: Learning physical features from in-hand tactile exploration for dynamic swing-up manipulation,” 2021.
  15. A. Melnik, L. Lach, M. Plappert, T. Korthals, R. Haschke, and H. Ritter, “Tactile sensing and deep reinforcement learning for in-hand manipulation tasks,” in IROS Workshop on Autonomous Object Manipulation, 2019.
  16. “Improved learning of robot manipulation tasks via tactile intrinsic motivation,” IEEE Robotics and Automation Letters, 2021.
  17. T. Li, H. Luo, L. Qin, X. Wang, Z. Xiong, H. Ding, Y. Gu, Z. Liu, and T. Zhang, “Flexible capacitive tactile sensor based on micropatterned dielectric layer,” Small, vol. 12, no. 36, pp. 5042–5048, 2016.
  18. K. Weiß and H. Worn, “The working principle of resistive tactile sensor cells,” in IEEE International Conference Mechatronics and Automation, 2005, vol. 1.   IEEE, 2005, pp. 471–476.
  19. E. Donlon, S. Dong, M. Liu, J. Li, E. Adelson, and A. Rodriguez, “Gelslim: A high-resolution, compact, robust, and calibrated tactile-sensing finger,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2018, pp. 1927–1934.
  20. M. Kaboli, K. Yao, D. Feng, and G. Cheng, “Tactile-based active object discrimination and target object search in an unknown workspace,” Autonomous Robots, vol. 43, no. 1, pp. 123–152, 2019.
  21. J. W. James, N. Pestell, and N. F. Lepora, “Slip detection with a biomimetic tactile sensor,” Robotics and Automation Letters, 2018.
  22. J. Bimbo, S. Luo, K. Althoefer, and H. Liu, “In-hand object pose estimation using covariance-based tactile to geometry matching,” IEEE Robotics and Automation Letters, vol. 1, no. 1, pp. 570–577, 2016.
  23. J. M. Romano, K. Hsiao, G. Niemeyer, S. Chitta, and K. J. Kuchenbecker, “Human-inspired robotic grasp control with tactile sensing,” IEEE Transactions on Robotics, vol. 27, no. 6, pp. 1067–1079, 2011.
  24. M. Kaboli, “New methods for active tactile object perception and learning with artificial robotic skin,” Dissertation, Technische Universität München, München, 2017.
  25. H. Van Hoof, T. Hermans, G. Neumann, and J. Peters, “Learning robot in-hand manipulation with tactile features,” in IEEE-RAS International Conference on Humanoid Robots (Humanoids), 2015.
  26. A. Koenig, Z. Liu, L. Janson, and R. Howe, “Tactile grasp refinement using deep reinforcement learning and analytic grasp stability metrics,” 2021.
  27. C. D’Eramo and G. Chalvatzaki, “Prioritized sampling with intrinsic motivation in multi-task reinforcement learning,” in 2022 International Joint Conference on Neural Networks (IJCNN).   IEEE, 2022, pp. 1–8.
  28. T. Schaul, D. Horgan, K. Gregor, and D. Silver, “Universal value function approximators,” in ICML, 2015.
  29. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971, 2015.
  30. G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” 2016.
  31. E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.   IEEE, 2012, pp. 5026–5033.
  32. M. Plappert, M. Andrychowicz, A. Ray, B. McGrew, B. Baker, G. Powell, J. Schneider, J. Tobin, M. Chociej, P. Welinder, V. Kumar, and W. Zaremba, “Multi-goal reinforcement learning: Challenging robotics environments and request for research,” 2018.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Erdi Sayar (1 paper)
  2. Zhenshan Bing (39 papers)
  3. Carlo D'Eramo (28 papers)
  4. Ozgur S. Oguz (19 papers)
  5. Alois Knoll (190 papers)
Citations (1)
View on Semantic Scholar

Summary

Contact Energy Based Hindsight Experience Prioritization

Introduction

The paper "Contact Energy Based Hindsight Experience Prioritization" introduces a novel approach to enhance the performance of reinforcement learning (RL) for robot manipulation tasks with sparse rewards. Traditional RL algorithms often struggle with sparse rewards due to the rarity of successful experiences, which hampers efficient learning. Hindsight Experience Replay (HER) partly addresses this by reinterpreting failures to contribute useful learning experiences. However, it uniformly samples these failed trajectories, failing to account for their varying utility in learning. This paper proposes Contact Energy Based Prioritization (CEBP), which selectively prioritizes contact-rich experiences, leveraging touch sensors in robot grippers to enhance learning efficiency.

Methodology

CEBP enhances HER by integrating tactile feedback and object displacement to prioritize experiences. This approach hinges on the insight that contact-rich interactions, measured through touch sensors on the robot’s gripper, provide substantial information valuable for learning. The framework computes "contact energy" by summing contact forces and multiplying them by the Euclidean norm of object displacement. This contact energy is then smoothed using a sigmoid function, instead of reducing contact forces to discrete values as seen in CPER, thus retaining a rich, continuous range of information.

The main contributions of the approach include:

  1. Incorporating touch sensors to rely solely on a binary sparse reward signal.
  2. Introducing a novel replay buffer prioritization scheme that uses contact energy to prioritize rich experiences.
  3. Demonstration of the proposed method in various robotic manipulation tasks within a simulation and transferring the policy to a real-world robot setup.

Performance is evaluated on three standard tasks: FetchPickAndPlace, FetchPush, and FetchSlide, all conducted in the MuJoCo simulation environment, further validating through Sim2Real transfer using a Franka robot.

Experimental Results

Experimental results indicate that CEBP consistently outperforms existing benchmarks in terms of learning efficiency and success rate, particularly in pick-and-place and push tasks. The novel prioritization method converges to a near-optimal policy markedly faster than competing methods. In the push task, for instance, the proposed method outperformed others rapidly in the early stages of training. Although CPER achieved faster initial learning in the slide task, CEBP matched its performance by the end with lower variance, indicating robustness.

A notable insight from the experiments is the utility of continuous contact force values over discrete thresholding in CPER. This continuous range facilitates a more nuanced learning process, as confirmed in the ablation paper investigating the impact of the sigmoid function's temperature parameter on learning performance.

Theoretical and Practical Implications

Theoretical implications of CEBP include a refined approach to experience prioritization in RL that better harnesses tactile feedback. Practically, this method promises more efficient training regimes for robots in real-world settings, reducing the reliance on extensive reward engineering. The successful application in a real-world Franka robot underscores its practical viability, although further work is needed to tackle challenges like occlusion in push tasks and simulation-reality friction discrepancies in slide tasks.

Future Directions

Further research might explore:

  1. Extended Applications: Adapting CEBP to other RL domains and tasks beyond robotic manipulation.
  2. Meta-RL Integration: Incorporating sparse rewards and hindsight experience priorities into meta-reinforcement learning frameworks could potentiate scalable, fast-adapting RL agents.
  3. Robust Sim2Real Transfer: Enhancing the robustness of policy transfer from simulation to reality with multiple camera perspectives and refined modeling of environmental parameters.

Conclusion

The Contact Energy Based Hindsight Experience Prioritization provides a significant step towards more efficient learning in robotic manipulation, leveraging tactile sensing and object displacement. By incorporating nuanced prioritization of replay buffer experiences, CEBP enhances the learning process for RL agents, paving the way for more adaptable and proficient robotic applications.

Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. Force-Based Hindsight Experience Prioritization