Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective (2403.07262v4)

Published 12 Mar 2024 in cs.LG and cs.AI

Abstract: Offline reinforcement learning endeavors to leverage offline datasets to craft effective agent policy without online interaction, which imposes proper conservative constraints with the support of behavior policies to tackle the out-of-distribution problem. However, existing works often suffer from the constraint conflict issue when offline datasets are collected from multiple behavior policies, i.e., different behavior policies may exhibit inconsistent actions with distinct returns across the state space. To remedy this issue, recent advantage-weighted methods prioritize samples with high advantage values for agent training while inevitably ignoring the diversity of behavior policy. In this paper, we introduce a novel Advantage-Aware Policy Optimization (A2PO) method to explicitly construct advantage-aware policy constraints for offline learning under mixed-quality datasets. Specifically, A2PO employs a conditional variational auto-encoder to disentangle the action distributions of intertwined behavior policies by modeling the advantage values of all training data as conditional variables. Then the agent can follow such disentangled action distribution constraints to optimize the advantage-aware policy towards high advantage values. Extensive experiments conducted on both the single-quality and mixed-quality datasets of the D4RL benchmark demonstrate that A2PO yields results superior to the counterparts. Our code is available at https://github.com/Plankson/A2PO

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions. arXiv preprint arXiv:2309.10150, 2023.
  2. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084–15097, 2021.
  3. Bail: Best-action imitation learning for batch deep reinforcement learning. Advances in Neural Information Processing Systems, 33:18353–18363, 2020.
  4. Lapo: Latent-variable advantage-weighted policy optimization for offline reinforcement learning. Annual Conference on Neural Information Processing Systems, 35:36902–36913, 2022.
  5. Uncertainty-aware model-based offline reinforcement learning for automated driving. IEEE Robotics and Automation Letters, 8(2):1167–1174, 2023.
  6. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:2112.10751, 2021.
  7. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
  8. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132–20145, 2021.
  9. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, 2018.
  10. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, 2019.
  11. Cirs: Bursting filter bubbles by counterfactual interactive recommender system. ACM Transactions on Information Systems, 42(1):1–27, 2023a.
  12. Act: Empowering decision transformer with dynamic programming via advantage conditioning. arXiv preprint arXiv:2309.05915, 2023b.
  13. Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters. Annual Conference on Neural Information Processing Systems, 35:18267–18281, 2022.
  14. Confidence-conditioned value functions for offline reinforcement learning. arXiv preprint arXiv:2212.04607, 2022.
  15. Harnessing mixed offline reinforcement learning datasets via trajectory weighting. arXiv preprint arXiv:2306.13085, 2023a.
  16. Beyond uniform sampling: Offline reinforcement learning with imbalanced datasets. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b.
  17. On the sample complexity of vanilla model-based offline reinforcement learning with dependent samples. arXiv preprint arXiv:2303.04268, 2023.
  18. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021.
  19. Stabilizing off-policy q-learning via bootstrapping error reduction. Annual Conference on Neural Information Processing Systems, 32, 2019.
  20. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179–1191, 2020.
  21. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
  22. Emergent agentic transformer from chain of hindsight experience. arXiv preprint arXiv:2305.16554, 2023.
  23. Mildly conservative q-learning for offline reinforcement learning. Annual Conference on Neural Information Processing Systems, 35:1711–1724, 2022.
  24. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.
  25. Pytorch: An imperative style, high-performance deep learning library. In Annual Conference on Neural Information Processing Systems, 2019.
  26. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019.
  27. Weighted policy constraints for offline reinforcement learning. In AAAI Conference on Artificial Intelligence, 2023.
  28. Pomerleau, D. A. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1):88–97, 1991.
  29. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
  30. Puterman, M. L. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
  31. Fast offline policy optimization for large scale recommendation. In AAAI Conference on Artificial Intelligence, 2023.
  32. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
  33. Deterministic policy gradient algorithms. In International Conference on Machine Learning, 2014.
  34. Offline rl with realistic datasets: Heteroskedasticity and support constraints. arXiv preprint arXiv:2211.01052, 2022.
  35. Learning structured output representation using deep conditional generative models. Annual Conference on Neural Information Processing Systems, 28, 2015.
  36. Reinforcement learning: An introduction. MIT press, 2018.
  37. Learning from good trajectories in offline multi-agent reinforcement learning. In AAAI Conference on Artificial Intelligence, 2023.
  38. Dasco: Dual-generator adversarial support constrained offline reinforcement learning. Annual Conference on Neural Information Processing Systems, 35:38937–38949, 2022.
  39. Leveraging offline data in online reinforcement learning. In International Conference on Machine Learning, 2023.
  40. Train once, get a family: State-adaptive balances for offline-to-online reinforcement learning. arXiv preprint arXiv:2310.17966, 2023.
  41. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193, 2022.
  42. Supported policy optimization for offline reinforcement learning. Annual Conference on Neural Information Processing Systems, 35:31278–31291, 2022.
  43. The in-sample softmax for offline reinforcement learning. In International Conference on Learning Representations, 2022.
  44. Offline rl with no ood actions: In-sample learning via implicit value regularization. arXiv preprint arXiv:2303.15810, 2023.
  45. Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. In International Conference on Machine Learning, pp.  38989–39007, 2023.
  46. Model-based offline policy optimization with adversarial network. arXiv preprint arXiv:2309.02157, 2023.
  47. Mopo: Model-based offline policy optimization. Annual Conference on Neural Information Processing Systems, 33:14129–14142, 2020.
  48. Semi-supervised offline reinforcement learning with action-free trajectories. In International conference on machine learning, 2023.
  49. Plas: Latent action space for offline reinforcement learning. In Conference on Robot Learning, 2021.
  50. Behavior proximal policy optimization. arXiv preprint arXiv:2302.11312, 2023.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now