Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 167 tok/s
Gemini 2.5 Pro 48 tok/s Pro
GPT-5 Medium 33 tok/s Pro
GPT-5 High 40 tok/s Pro
GPT-4o 92 tok/s Pro
Kimi K2 193 tok/s Pro
GPT OSS 120B 425 tok/s Pro
Claude Sonnet 4.5 37 tok/s Pro
2000 character limit reached

Data-Driven Knowledge Transfer in Batch $Q^*$ Learning (2404.15209v2)

Published 1 Apr 2024 in cs.LG, stat.ME, and stat.ML

Abstract: In data-driven decision-making in marketing, healthcare, and education, it is desirable to utilize a large amount of data from existing ventures to navigate high-dimensional feature spaces and address data scarcity in new ventures. We explore knowledge transfer in dynamic decision-making by concentrating on batch stationary environments and formally defining task discrepancies through the lens of Markov decision processes (MDPs). We propose a framework of Transferred Fitted $Q$-Iteration algorithm with general function approximation, enabling the direct estimation of the optimal action-state function $Q*$ using both target and source data. We establish the relationship between statistical performance and MDP task discrepancy under sieve approximation, shedding light on the impact of source and target sample sizes and task discrepancy on the effectiveness of knowledge transfer. We show that the final learning error of the $Q*$ function is significantly improved from the single task rate both theoretically and empirically.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. Provable benefits of representational transfer in reinforcement learning. In Proceedings of Thirty Sixth Conference on Learning Theory, Volume 195.
  2. Successor features for transfer in reinforcement learning. In Advances in Neural Information Processing Systems, Volume 30.
  3. Transfer learning for nonparametric regression: Non-asymptotic minimax analysis and adaptive procedure. arXiv preprint arXiv:2401.12272.
  4. Transfer learning for nonparametric classification: Minimax rate and adaptive classifier. The Annals of Statistics 49(1), 100–128.
  5. Transferred Q-learning. arXiv preprint arXiv:2202.04709.
  6. Information-theoretic considerations in batch reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, Volume 97.
  7. Chen, X. (2007). Large sample sieve estimation of semi-nonparametric models. Handbook of Econometrics 6, 5549–5632.
  8. Provable benefit of multitask representation learning in reinforcement learning. In Advances in Neural Information Processing Systems, Volume 35.
  9. Adaptive and robust multi-task learning. The Annals of Statistics 51(5), 2015–2039.
  10. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research 6(18), 503–556.
  11. A theoretical analysis of deep Q-learning. In Proceedings of the 2nd Conference on Learning for Dynamics and Control, Volume 120.
  12. Huang, J. Z. (1998). Projection estimation in multiple regression with application to functional ANOVA models. The Annals of Statistics 26(1), 242–272.
  13. MIMIC-III, a freely accessible critical care database. Scientific Data 3(1), 1–9.
  14. The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature Medicine 24(11), 1716–1720.
  15. Settling the sample complexity of model-based offline reinforcement learning. The Annals of Statistics 52(1), 233–260.
  16. Transfer learning for high-dimensional linear regression: Prediction, estimation and minimax optimality. Journal of the Royal Statistical Society Series B: Statistical Methodology 84(1), 149–173.
  17. Transfer learning in large-scale Gaussian graphical models with false discovery rate control. Journal of the American Statistical Association 118(543), 2171–2183.
  18. Estimation and inference for high-dimensional generalized linear models with knowledge transfer. Journal of the American Statistical Association, 1–12.
  19. Liu, X. (2023). Dynamic coupon targeting using batch deep reinforcement learning: An application to livestream shopping. Marketing Science 42(4), 637–658.
  20. On the power of multitask representation learning in linear MDP. arXiv preprint arXiv:2106.08053.
  21. Imitation-regularized offline learning. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, Volume 89.
  22. Minimax optimal approaches to the label shift problem in non-parametric settings. The Journal of Machine Learning Research 23(1), 15698–15742.
  23. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research 9(27), 815–857.
  24. Murphy, S. A. (2005). A generalization error for Q-learning. Journal of Machine Learning Research 6(37), 1073–1097.
  25. A reinforcement learning approach to weaning of mechanical ventilation in intensive care units. arXiv preprint arXiv:1704.06300.
  26. Faster teaching via POMDP planning. Cognitive Science 40(6), 1290–1332.
  27. Riedmiller, M. (2005). Neural fitted Q iteration–first experiences with a data efficient neural reinforcement learning method. In Machine Learning: ECML 2005, Volume 3720.
  28. Statistical inference of the value function for reinforcement learning in infinite-horizon settings. Journal of the Royal Statistical Society Series B: Statistical Methodology 84(3), 765–793.
  29. Pessimistic Q-learning for offline reinforcement learning: Towards optimal sample complexity. In Proceedings of the 39th International Conference on Machine Learning, Volume 162.
  30. Reward is enough. Artificial Intelligence 299, 103535.
  31. Sutton, R. S. and A. G. Barto (2018). Reinforcement Learning: An Introduction. MIT Press.
  32. Transfer learning under high-dimensional generalized linear models. Journal of the American Statistical Association 118(544), 2684–2697.
  33. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08817.
  34. Minimax optimal transfer learning for kernel-based nonparametric regression. arXiv preprint arXiv:2310.13966.
  35. Wang, K. (2023). Pseudo-labeling for kernel ridge regression under covariate shift. arXiv preprint arXiv:2302.10160.
  36. Q* approximation schemes for batch reinforcement learning: A theoretical comparison. In Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence, Volume 124.
  37. Batch value-function approximation with only realizability. In Proceedings of the 38th International Conference on Machine Learning, Volume 139.
  38. Model-based reinforcement learning is minimax-optimal for offline zero-sum Markov games. arXiv preprint arXiv:2206.04044.
  39. Federated natural policy gradient methods for multi-task reinforcement learning. arXiv preprint arXiv:2311.00201.
  40. Knowledge transfer for deep reinforcement learning with hierarchical experience replay. In Proceedings of the AAAI Conference on Artificial Intelligence, Volume 31.
  41. Optimal multi-distribution learning. arXiv preprint arXiv:2312.05134.
  42. Transfer learning in deep reinforcement learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(11), 13344–13362.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

This paper has been mentioned in 2 tweets and received 6 likes.

Upgrade to Pro to view all of the tweets about this paper:

Start a free 7-day Pro trial