2000 character limit reached
A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasets (2110.04698v2)
Published 10 Oct 2021 in cs.LG
Abstract: Recent Offline Reinforcement Learning methods have succeeded in learning high-performance policies from fixed datasets of experience. A particularly effective approach learns to first identify and then mimic optimal decision-making strategies. Our work evaluates this method's ability to scale to vast datasets consisting almost entirely of sub-optimal noise. A thorough investigation on a custom benchmark helps identify several key challenges involved in learning from high-noise datasets. We re-purpose prioritized experience sampling to locate expert-level demonstrations among millions of low-performance samples. This modification enables offline agents to learn state-of-the-art policies in benchmark tasks using datasets where expert actions are outnumbered nearly 65:1.
- Joshua Achiam “Spinning Up in Deep Reinforcement Learning”, 2018
- Rishabh Agarwal, Dale Schuurmans and Mohammad Norouzi “An Optimistic Perspective on Offline Reinforcement Learning”, 2020 arXiv:1907.04543 [cs.LG]
- Oron Anschel, Nir Baram and Nahum Shimkin “Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning”, 2017 arXiv:1611.01929 [cs.AI]
- “OpenAI Gym”, 2016 arXiv:1606.01540 [cs.LG]
- “Decision Transformer: Reinforcement Learning via Sequence Modeling”, 2021 arXiv:2106.01345 [cs.LG]
- “BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning”, 2020 arXiv:1910.12179 [cs.LG]
- “Randomized Ensembled Double Q-Learning: Learning Fast Without a Model”, 2021 arXiv:2101.05982 [cs.LG]
- Po-Wei Chou, Daniel Maturana and Sebastian Scherer “Improving Stochastic Policy Gradients in Continuous Control with Deep Reinforcement Learning using the Beta Distribution” In Proceedings of the 34th International Conference on Machine Learning 70, Proceedings of Machine Learning Research PMLR, 2017, pp. 834–843 URL: http://proceedings.mlr.press/v70/chou17a.html
- “OpenAI Baselines” In GitHub repository GitHub, https://github.com/openai/baselines, 2017
- “Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO”, 2020 arXiv:2005.12729 [cs.LG]
- Johan Ferret, Olivier Pietquin and Matthieu Geist “Self-Imitation Advantage Learning” In arXiv preprint arXiv:2012.11989, 2020
- “D4RL: Datasets for Deep Data-Driven Reinforcement Learning”, 2021 arXiv:2004.07219 [cs.LG]
- Scott Fujimoto, Herke Hoof and David Meger “Addressing function approximation error in actor-critic methods” In International Conference on Machine Learning, 2018, pp. 1587–1596 PMLR
- Tanmay Gangwani, Qiang Liu and Jian Peng “Learning Self-Imitating Diverse Policies”, 2019 arXiv:1805.10309 [stat.ML]
- Jake Grigsby “deep_control: Deep Reinforcement Learning for Continuous Control in PyTorch.” In GitHub repository GitHub, https://github.com/jakegrigsby/deep_control, 2020
- “RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning”, 2021 arXiv:2006.13888 [cs.LG]
- “On Calibration of Modern Neural Networks”, 2017 arXiv:1706.04599 [cs.LG]
- “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor”, 2018 arXiv:1801.01290 [cs.LG]
- “Learning values across many orders of magnitude”, 2016 arXiv:1602.07714 [cs.LG]
- “Deep Reinforcement Learning that Matters”, 2019 arXiv:1709.06560 [cs.LG]
- “Multi-task Deep Reinforcement Learning with PopArt”, 2018 arXiv:1809.04474 [cs.LG]
- “SciPy: Open source scientific tools for Python”
- Aviral Kumar, Abhishek Gupta and Sergey Levine “DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction”, 2020 arXiv:2003.07305 [cs.LG]
- “Conservative Q-Learning for Offline Reinforcement Learning”, 2020 arXiv:2006.04779 [cs.LG]
- “Stabilizing off-policy q-learning via bootstrapping error reduction” In arXiv preprint arXiv:1906.00949, 2019
- “Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics”, 2020 arXiv:2005.04269 [cs.LG]
- Michail G Lagoudakis and Ronald Parr “Reinforcement learning as classification: Leveraging modern classifiers” In Proceedings of the 20th International Conference on Machine Learning (ICML-03), 2003, pp. 424–431
- “Maxmin Q-learning: Controlling the Estimation Bias of Q-learning”, 2020 arXiv:2002.06487 [cs.LG]
- “SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning”, 2020 arXiv:2007.04938 [cs.LG]
- Sergey Levine “Deep Reinforcement Learning in the Real World” Workshop on New Directions in Reinforcement Learning and Control, 2019 URL: https://www.youtube.com/watch?v=b97H5uz8xkI
- “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems”, 2020 arXiv:2005.01643 [cs.LG]
- “Continuous control with deep reinforcement learning”, 2015 arXiv:1509.02971 [cs.LG]
- “Accelerating Online Reinforcement Learning with Offline Datasets”, 2020 arXiv:2006.09359 [cs.LG]
- “Overcoming Exploration in Reinforcement Learning with Demonstrations”, 2018 arXiv:1709.10089 [cs.LG]
- “Self-Imitation Learning”, 2018 arXiv:1806.05635 [cs.LG]
- “An Algorithmic Perspective on Imitation Learning” In Foundations and Trends in Robotics 7.1-2 Now Publishers, 2018, pp. 1–179 DOI: 10.1561/2300000053
- “Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning”, 2019 arXiv:1910.00177 [cs.LG]
- “NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning”, 2021 arXiv:2102.00714 [cs.LG]
- Siddharth Reddy, Anca D. Dragan and Sergey Levine “SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards”, 2019 arXiv:1905.11108 [cs.LG]
- “In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning”, 2021 arXiv:2101.06329 [cs.LG]
- “ImageNet Large Scale Visual Recognition Challenge”, 2015 arXiv:1409.0575 [cs.CV]
- “Prioritized Experience Replay”, 2016 arXiv:1511.05952 [cs.LG]
- “Variational Imitation Learning with Diverse-quality Demonstrations” In Proceedings of the 37th International Conference on Machine Learning 119, Proceedings of Machine Learning Research PMLR, 2020, pp. 9407–9417 URL: http://proceedings.mlr.press/v119/tangkaratt20a.html
- “Exponentially Weighted Imitation Learning for Batched Historical Data” In Advances in Neural Information Processing Systems 31 Curran Associates, Inc., 2018 URL: https://proceedings.neurips.cc/paper/2018/file/4aec1b3435c52abbdf8334ea0e7141e0-Paper.pdf
- “Exponentially Weighted Imitation Learning for Batched Historical Data.”
- “Critic Regularized Regression”, 2020 arXiv:2006.15134 [cs.LG]
- “Uncertainty Weighted Offline Reinforcement Learning”, 2020 URL: https://offline-rl-neurips.github.io/pdf/27.pdf
- “Imitation learning from imperfect demonstration” In International Conference on Machine Learning, 2019, pp. 6818–6827 PMLR
- “Soft Actor-Critic (SAC) implementation in PyTorch” In GitHub repository GitHub, https://github.com/denisyarats/pytorch_sac, 2020
- “MOPO: Model-based Offline Policy Optimization”, 2020 arXiv:2005.13239 [cs.LG]
- “Offline Learning from Demonstrations and Unlabeled Experience”, 2020 arXiv:2011.13885 [cs.LG]
Sponsored by Paperpile, the PDF & BibTeX manager trusted by top AI labs.
Get 30 days freePaper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.