Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 76 tok/s
Gemini 2.5 Pro 59 tok/s Pro
GPT-5 Medium 24 tok/s Pro
GPT-5 High 23 tok/s Pro
GPT-4o 95 tok/s Pro
Kimi K2 207 tok/s Pro
GPT OSS 120B 449 tok/s Pro
Claude Sonnet 4 37 tok/s Pro
2000 character limit reached

Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning (2408.03029v4)

Published 6 Aug 2024 in cs.LG and cs.AI

Abstract: Reward shaping is a technique in reinforcement learning that addresses the sparse-reward problem by providing more frequent and informative rewards. We introduce a self-adaptive and highly efficient reward shaping mechanism that incorporates success rates derived from historical experiences as shaped rewards. The success rates are sampled from Beta distributions, which dynamically evolve from uncertain to reliable values as data accumulates. Initially, the shaped rewards exhibit more randomness to encourage exploration, while over time, the increasing certainty enhances exploitation, naturally balancing exploration and exploitation. Our approach employs Kernel Density Estimation (KDE) combined with Random Fourier Features (RFF) to derive the Beta distributions, providing a computationally efficient, non-parametric, and learning-free solution for high-dimensional continuous state spaces. Our method is validated on various tasks with extremely sparse rewards, demonstrating notable improvements in sample efficiency and convergence stability over relevant baselines.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Utilizing prior solutions for reward shaping and composition in entropy-regularized reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6658–6665.
  2. Analysis of thompson sampling for the multi-armed bandit problem. In Conference on learning theory, pages 39–1. JMLR Workshop and Conference Proceedings.
  3. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297:103500.
  4. Potential-based shaping in model-based reinforcement learning. In AAAI Conference on Artificial Intelligence, pages 604–609.
  5. Unifying count-based exploration and intrinsic motivation. Advances in Neural Information Processing Systems, 29.
  6. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. The International Journal of Robotics Research, 41(1):45–67.
  7. Safe imitation learning via fast bayesian reward inference from preferences. In International Conference on Machine Learning, pages 1165–1177. PMLR.
  8. Exploration by random network distillation. In International Conference on Learning Representations.
  9. Heuristic-guided reinforcement learning. Advances in Neural Information Processing Systems, 34:13550–13563.
  10. Gymnasium robotics.
  11. Exploration-guided reward shaping for reinforcement learning under sparse rewards. Advances in Neural Information Processing Systems, 35:5829–5842.
  12. Dynamic potential-based reward shaping. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems, pages 433–440. IFAAMAS.
  13. Accelerating numerical dense linear algebra calculations with gpus. Numerical computations with GPUs, pages 3–28.
  14. One-shot imitation learning. Advances in neural information processing systems, 30.
  15. Risk averse bayesian reward learning for autonomous navigation from human demonstration. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 8928–8935. IEEE.
  16. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations.
  17. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587–1596. PMLR.
  18. Unpacking reward shaping: Understanding the benefits of reward engineering on sample complexity. Advances in Neural Information Processing Systems, 35:15281–15295.
  19. Behavior alignment via reward function optimization. Advances in Neural Information Processing Systems, 36.
  20. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861–1870. PMLR.
  21. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905.
  22. Cooperative inverse reinforcement learning. Advances in Neural Information Processing Systems, 29.
  23. Generative adversarial imitation learning. Advances in neural information processing systems, 29.
  24. Diversity-driven exploration strategy for deep reinforcement learning. Advances in neural information processing systems, 31.
  25. Learning to utilize shaping rewards: A new approach of reward shaping. Advances in Neural Information Processing Systems, 33:15931–15941.
  26. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1–18.
  27. Exploration in deep reinforcement learning: A survey. Information Fusion, 85:1–22.
  28. Reward shaping for reinforcement learning with an assistant reward agent. In Forty-first International Conference on Machine Learning. PMLR.
  29. How to stay curious while avoiding noisy tvs using aleatoric uncertainty estimation. In International Conference on Machine Learning, pages 15220–15240. PMLR.
  30. Self-supervised online reward shaping in sparse-reward environments. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2369–2375. IEEE.
  31. Learning to shape rewards using a game of two partners. In AAAI Conference on Artificial Intelligence, volume 37, pages 11604–11612.
  32. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533.
  33. Count-based exploration with neural density models. In International Conference on Machine Learning, pages 2721–2730. PMLR.
  34. Controllability-aware unsupervised skill discovery. In International Conference on Machine Learning, pages 27225–27245. PMLR.
  35. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, pages 2778–2787. PMLR.
  36. Random features for large-scale kernel machines. Advances in neural information processing systems, 20.
  37. Bayesian inverse reinforcement learning. In International Joint Conference on Artificial Intelligence, volume 7, pages 2586–2591.
  38. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
  39. Inverse optimal control adapted to the noise characteristics of the human sensorimotor system. Advances in Neural Information Processing Systems, 34:9429–9442.
  40. Reward design via online gradient ascent. Advances in Neural Information Processing Systems, 23.
  41. Internal rewards mitigate agent boundedness. In International Conference on Machine Learning, pages 1007–1014.
  42. Learning intrinsic rewards as a bi-level optimization problem. In Conference on Uncertainty in Artificial Intelligence, pages 111–120. PMLR.
  43. Exploit reward shifting in value-based deep-rl: Optimistic curiosity-based exploration and conservative exploitation via linear reward shaping. Advances in Neural Information Processing Systems, 35:37719–37734.
  44. Reinforcement learning: An introduction. MIT press.
  45. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in Neural Information Processing Systems, 30.
  46. Thompson, W. R. (1933). On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285–294.
  47. Mujoco: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033. IEEE.
  48. Gymnasium.
  49. Keeping your distance: Solving sparse reward tasks using self-balancing shaped rewards. Advances in Neural Information Processing Systems, 32.
  50. Conditional image generation with pixelcnn decoders. Advances in neural information processing systems, 29.
  51. Shaping rewards for reinforcement learning with imperfect demonstrations using generative models. In IEEE International Conference on Robotics and Automation, pages 6628–6634. IEEE.
  52. Learning to share in networked multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 35:15119–15131.
  53. On learning intrinsic rewards for policy gradient methods. Advances in Neural Information Processing Systems, 31.
  54. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence, volume 8, pages 1433–1438. Chicago, IL, USA.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Lightbulb On Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

This paper has been mentioned in 2 posts and received 0 likes.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up to View

Don't miss out on important new AI/ML research

See which papers are being discussed right now on X, Reddit, and more:

Explore Trending Papers

“Emergent Mind helps me see which AI papers have caught fire online.”

Philip

Philip

Creator, AI Explained on YouTube