2000 character limit reached
Monitoring Fidelity of Online Reinforcement Learning Algorithms in Clinical Trials (2402.17003v2)
Published 26 Feb 2024 in cs.LG, cs.AI, and cs.CY
Abstract: Online reinforcement learning (RL) algorithms offer great potential for personalizing treatment for participants in clinical trials. However, deploying an online, autonomous algorithm in the high-stakes healthcare setting makes quality control and data quality especially difficult to achieve. This paper proposes algorithm fidelity as a critical requirement for deploying online RL algorithms in clinical trials. It emphasizes the responsibility of the algorithm to (1) safeguard participants and (2) preserve the scientific utility of the data for post-trial analyses. We also present a framework for pre-deployment planning and real-time monitoring to help algorithm developers and clinical researchers ensure algorithm fidelity. To illustrate our framework's practical application, we present real-world examples from the Oralytics clinical trial. Since Spring 2023, this trial successfully deployed an autonomous, online RL algorithm to personalize behavioral interventions for participants at risk for dental disease.
- Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016.
- Assessing time-varying causal effect moderation in mobile health. Journal of the American Statistical Association, 113(523):1112–1121, 2018.
- Artificial intelligence, bias and clinical safety. BMJ quality & safety, 2019.
- Adaptive learning algorithms to optimize mobile applications for behavioral health: guidelines for design decisions. Journal of the American Medical Informatics Association, 28(6):1225–1234, 2021.
- Can the artificial intelligence technique of reinforcement learning use continuously-monitored digital data to optimize treatment for weight loss? Journal of behavioral medicine, 42:276–290, 2019.
- Approaches to data analyses of clinical trials. Progress in cardiovascular diseases, 54(4):330–334, 2012.
- Christine Grady. Institutional review boards: Purpose and challenges. Chest, 148(5):1148–1155, 2015.
- Confidence intervals for policy evaluation in adaptive experiments. Proceedings of the National Academy of Sciences, 118(15), 2021.
- Towards continual reinforcement learning: A review and perspectives. Journal of Artificial Intelligence Research, 75:1401–1476, 2022.
- Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
- Personalized heartsteps: A reinforcement learning algorithm for optimizing physical activity. CoRR, abs/1909.03539, 2019. URL http://arxiv.org/abs/1909.03539.
- Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the consort-ai extension. The Lancet Digital Health, 2(10):e537–e548, 2020.
- Ideal algorithms in healthcare: explainable, dynamic, precise, autonomous, fair, and reproducible. PLOS digital health, 1(1):e0000006, 2022.
- Optimizing an adaptive digital oral health intervention for promoting oral self-care behaviors: Micro-randomized trial protocol. Contemporary Clinical Trials, pp. 107464, 2024.
- Ml4h auditing: From paper to practice. In Machine learning for health, pp. 280–317. PMLR, 2020.
- Machine learning for health: algorithm auditing & quality control. Journal of medical systems, 45:1–8, 2021.
- Continual lifelong learning with neural networks: A review. Neural networks, 113:54–71, 2019.
- The microrandomized trial for developing digital interventions: Experimental design and data analysis considerations. Psychological methods, 2022.
- A tutorial on thompson sampling. Foundations and Trends® in Machine Learning, 11(1):1–96, 2018.
- “everyone wants to do the model work, not the data work”: Data cascades in high-stakes ai. In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–15, 2021.
- Vivek Shetty. Micro-randomized trial to optimize digital oral health behavior change interventions. Identifier NCT02747927. U.S. National Library of Medicine. https://clinicaltrials.gov/ct2/show/NCT05624489 (accessed 2023-06-16), Nov 2022.
- Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning, pp. 2139–2148. PMLR, 2016.
- Designing reinforcement learning algorithms for digital interventions: Pre-implementation guidelines. Algorithms, 15(8), 2022. ISSN 1999-4893. doi: 10.3390/a15080255. URL https://www.mdpi.com/1999-4893/15/8/255.
- Reward design for an online reinforcement learning algorithm supporting oral self-care. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 15724–15730, 2023.
- A survey of human-in-the-loop for machine learning. Future Generation Computer Systems, 135:364–381, 2022.
- Power constrained bandits. In Machine Learning for Healthcare Conference, pp. 209–259. PMLR, 2021.
- Encouraging physical activity in patients with diabetes: intervention using a reinforcement learning system. Journal of medical Internet research, 19(10):e338, 2017.
- Statistical inference after adaptive sampling for longitudinal data. arXiv preprint arXiv:2202.07098, 2022.
- Anna L. Trella (9 papers)
- Kelly W. Zhang (16 papers)
- Inbal Nahum-Shani (28 papers)
- Vivek Shetty (6 papers)
- Iris Yan (4 papers)
- Finale Doshi-Velez (134 papers)
- Susan A. Murphy (35 papers)