Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Multi Task Inverse Reinforcement Learning for Common Sense Reward (2402.11367v1)

Published 17 Feb 2024 in cs.LG

Abstract: One of the challenges in applying reinforcement learning in a complex real-world environment lies in providing the agent with a sufficiently detailed reward function. Any misalignment between the reward and the desired behavior can result in unwanted outcomes. This may lead to issues like "reward hacking" where the agent maximizes rewards by unintended behavior. In this work, we propose to disentangle the reward into two distinct parts. A simple task-specific reward, outlining the particulars of the task at hand, and an unknown common-sense reward, indicating the expected behavior of the agent within the environment. We then explore how this common-sense reward can be learned from expert demonstrations. We first show that inverse reinforcement learning, even when it succeeds in training an agent, does not learn a useful reward function. That is, training a new agent with the learned reward does not impair the desired behaviors. We then demonstrate that this problem can be solved by training simultaneously on multiple tasks. That is, multi-task inverse reinforcement learning can be applied to learn a useful reward function.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning (ICML), 2004.
  2. A survey of inverse reinforcement learning: Challenges, methods and progress. Artif. Intell., 297:103500, 2018. URL https://api.semanticscholar.org/CorpusID:49312150.
  3. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297, 2021.
  4. Maximum entropy multi-task inverse rl. arXiv preprint arXiv:2004.12873, 2020.
  5. Curriculum learning. In International Conference on Machine Learning (ICML), 2009.
  6. Caruana, R. Multitask learning. Machine learning, 28(1):41–75, 1997.
  7. Multi-task hierarchical adversarial inverse reinforcement learning. 2023.
  8. Faulty reward functions in the wild, 2022. URL https://openai.com/research/faulty-reward-functions.
  9. Model-agnostic meta-learning for fast adaptation of deep networks. pp.  1126–1135, 2017a.
  10. One-shot visual imitation learning via meta-learning. In Conference on robot learning (CoRL). PMLR, 2017b.
  11. Finn, C. et al. Connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint arXiv:1609.04807, 2016.
  12. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations (ICLR), 2018.
  13. Multi-task maximum entropy inverse reinforcement learning. ICML Workshop on Goal Specifications for Reinforcement Learning, 2018.
  14. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning (ICML), pp.  1861–1870, 2018.
  15. Robust imitation via mirror descent inverse reinforcement learning. Advances in Neural Information Processing Systems (NeruIPS), 2022.
  16. Generative adversarial imitation learning. Advances in neural information processing systems (NeurIPS), 29, 2016.
  17. Regularized inverse reinforcement learning. In International Conference on Learning Representations (ICLR), 2021.
  18. Learning shared safety constraints from multi-task demonstrations. In Neural Information Processing Systems (NeurIPS), 2023.
  19. Learning safety constraints from demonstrations with unknown rewards. arXiv preprint, 2023.
  20. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems (NerIPS), 34:18878–18890, 2021.
  21. Inverse constrained reinforcement learning. In International conference on machine learning, 2021.
  22. Auxiliary learning by implicit differentiation. In International Conference on Learning Representations (ICLR), 2020.
  23. Multi-task learning as a bargaining game. In International Conference on Machine Learning (ICML), 2022.
  24. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning (ICML), 2000.
  25. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International Conference on Machine Learning (ICML).
  26. Ruder, S. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017.
  27. Smile: Scalable meta inverse reinforcement learning through context-conditional policies. Advances in Neural Information Processing Systems (NeruIPS), 32, 2019.
  28. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems (NeurIPS), 2022.
  29. Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning (ICML), 2021.
  30. Reinforcement learning: An introduction. 2018.
  31. On implicit bias in overparameterized bilevel optimization. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp.  22234–22259. PMLR, 17–23 Jul 2022.
  32. A survey of multi-task deep reinforcement learning. Electronics. URL https://www.mdpi.com/2079-9292/9/9/1363.
  33. Learning a prior over intent via meta-inverse reinforcement learning. In International Conference on Machine Learning (ICML), 2019.
  34. Multi-task reinforcement learning with soft modularization. Advances in Neural Information Processing Systems (NeurIPS), 2020.
  35. Meta-inverse reinforcement learning with probabilistic context variables. Advances in neural information processing systems (NeurIPS), 32, 2019.
  36. One-shot imitation from observing humans via domain-adaptive meta-learning. 2018.
  37. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems (NeurIPS), 2020.
  38. Conservative data sharing for multi-task offline reinforcement learning. Advances in Neural Information Processing Systems (NeruIPS), 2021.
  39. Efficient multi-task reinforcement learning via selective behavior sharing. arXiv preprint arXiv:2302.00671, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Neta Glazer (8 papers)
  2. Aviv Navon (23 papers)
  3. Aviv Shamsian (23 papers)
  4. Ethan Fetaya (46 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets