Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories (2210.06518v3)

Published 12 Oct 2022 in cs.LG, cs.AI, and cs.RO

Abstract: Natural agents can effectively learn from multiple data sources that differ in size, quality, and types of measurements. We study this heterogeneity in the context of offline reinforcement learning (RL) by introducing a new, practically motivated semi-supervised setting. Here, an agent has access to two sets of trajectories: labelled trajectories containing state, action and reward triplets at every timestep, along with unlabelled trajectories that contain only state and reward information. For this setting, we develop and study a simple meta-algorithmic pipeline that learns an inverse dynamics model on the labelled data to obtain proxy-labels for the unlabelled data, followed by the use of any offline RL algorithm on the true and proxy-labelled trajectories. Empirically, we find this simple pipeline to be highly successful -- on several D4RL benchmarks~\cite{fu2020d4rl}, certain offline RL algorithms can match the performance of variants trained on a fully labelled dataset even when we label only 10\% of trajectories which are highly suboptimal. To strengthen our understanding, we perform a large-scale controlled empirical study investigating the interplay of data-centric properties of the labelled and unlabelled datasets, with algorithmic design choices (e.g., choice of inverse dynamics, offline RL algorithm) to identify general trends and best practices for training RL agents on semi-supervised offline datasets.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304–29320, 2021.
  2. A framework for behavioural cloning. In Machine Intelligence 15, pp.  103–129, 1995.
  3. Video pretraining (vpt): Learning to act by watching unlabeled online videos, 2022. URL https://arxiv.org/abs/2206.11795.
  4. Bellman, R. A markovian decision process. Indiana Univ. Math. J., 1957.
  5. Humanoid robot learning and game playing using pc-based vision. In IEEE/RSJ international conference on intelligent robots and systems, volume 3, pp.  2449–2454. IEEE, 2002.
  6. Large-scale study of curiosity-driven learning. In ICLR, 2019.
  7. Offline reinforcement learning at multiple frequencies. arXiv preprint arXiv:2207.13082, 2022.
  8. Semi-supervised learning. 2006. Cambridge, Massachusettes: The MIT Press View Article, 2, 2006.
  9. Decision transformer: Reinforcement learning via sequence modeling. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=a7APmM4B9d.
  10. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:2112.10751, 2021.
  11. Fralick, S. Learning to recognize patterns without a teacher. IEEE Transactions on Information Theory, 13(1):57–64, 1967.
  12. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
  13. A minimalist approach to offline reinforcement learning. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=Q32U7dzWXpc.
  14. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052–2062. PMLR, 2019.
  15. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pp. 3682–3691. PMLR, 2021.
  16. Offline rl policies should be trained to be adaptive. In International Conference on Machine Learning, pp. 7513–7530. PMLR, 2022.
  17. Learning invariant feature spaces to transfer skills with reinforcement learning. arXiv preprint arXiv:1703.02949, 2017.
  18. Exploration via elliptical episodic bonuses. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=Xg-yZos9qJQ.
  19. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
  20. Movement imitation with nonlinear dynamical systems in humanoid robots. In Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No. 02CH37292), volume 2, pp.  1398–1403. IEEE, 2002.
  21. Offline reinforcement learning as one big sequence modeling problem. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=wgeK563QgSw.
  22. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456, 2019.
  23. MobILE: Model-based imitation learning from observation alone. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=_Rtm4rYnIIL.
  24. Domain adaptive imitation learning. In International Conference on Machine Learning, pp. 5286–5295. PMLR, 2020.
  25. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  26. Probabilistic graphical models: principles and techniques. MIT press, 2009.
  27. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774–5783. PMLR, 2021a.
  28. Offline reinforcement learning with implicit q-learning, 2021b.
  29. Stabilizing off-policy q-learning via bootstrapping error reduction. arXiv preprint arXiv:1906.00949, 2019.
  30. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04779, 2020.
  31. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
  32. Multi-game decision transformers. arXiv preprint arXiv:2205.15241, 2022.
  33. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
  34. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp.  1118–1125. IEEE, 2018.
  35. Improving zero-shot generalization in offline reinforcement learning using generalized similarity functions. arXiv preprint arXiv:2111.14629, 2021.
  36. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02074, 2019.
  37. An overview of deep semi-supervised learning. arXiv preprint arXiv:2006.05278, 2020.
  38. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778–2787. PMLR, 2017.
  39. Offline reinforcement learning from images with latent space models. In Learning for Dynamics and Control, pp.  1154–1168. PMLR, 2021.
  40. A generalist agent. arXiv preprint arXiv:2205.06175, 2022.
  41. Reinforcement learning with videos: Combining offline observations with interaction. arXiv preprint arXiv:2011.06507, 2020a.
  42. Learning predictive models from observation and interaction. In European Conference on Computer Vision, pp.  708–725. Springer, 2020b.
  43. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pp.  1134–1141. IEEE, 2018.
  44. Third-person visual imitation learning via decoupled hierarchical controller. In NeurIPS, 2019.
  45. Third-person imitation learning. CoRR, abs/1703.01703, 2017. URL http://arxiv.org/abs/1703.01703.
  46. Behavioral cloning from observation. CoRR, abs/1805.01954, 2018a. URL http://arxiv.org/abs/1805.01954.
  47. Generative adversarial imitation from observation. CoRR, abs/1807.06158, 2018b. URL http://arxiv.org/abs/1807.06158.
  48. Recent advances in imitation learning from observation. arXiv preprint arXiv:1905.13566, 2019.
  49. A survey on semi-supervised learning. Machine Learning, 109(2):373–440, 2020.
  50. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019.
  51. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962, 2019.
  52. How to leverage unlabeled data in offline reinforcement learning. arXiv preprint arXiv:2202.01741, 2022.
  53. Online decision transformer. arXiv preprint arXiv:2202.05607, 2022.
  54. Zhu, X. Semi-supervised learning literature survey. Technical Report 1530, Computer Sciences, University of Wisconsin-Madison, 2005.
Citations (17)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now