Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

An Improved Strategy for Blood Glucose Control Using Multi-Step Deep Reinforcement Learning (2403.07566v2)

Published 12 Mar 2024 in cs.AI

Abstract: Blood Glucose (BG) control involves keeping an individual's BG within a healthy range through extracorporeal insulin injections is an important task for people with type 1 diabetes. However,traditional patient self-management is cumbersome and risky. Recent research has been devoted to exploring individualized and automated BG control approaches, among which Deep Reinforcement Learning (DRL) shows potential as an emerging approach. In this paper, we use an exponential decay model of drug concentration to convert the formalization of the BG control problem, which takes into account the delay and prolongedness of drug effects, from a PAE-POMDP (Prolonged Action Effect-Partially Observable Markov Decision Process) to a MDP, and we propose a novel multi-step DRL-based algorithm to solve the problem. The Prioritized Experience Replay (PER) sampling method is also used in it. Compared to single-step bootstrapped updates, multi-step learning is more efficient and reduces the influence from biasing targets. Our proposed method converges faster and achieves higher cumulative rewards compared to the benchmark in the same training environment, and improves the time-in-range (TIR), the percentage of time the patient's BG is within the target range, in the evaluation phase. Our work validates the effectiveness of multi-step reinforcement learning in BG control, which may help to explore the optimal glycemic control measure and improve the survival of diabetic patients.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Global, regional, and national burden of diabetes from 1990 to 2021, with projections of prevalence to 2050: a systematic analysis for the global burden of disease study 2021. The Lancet, 402(10397):203–234, 2023.
  2. Reinforcement learning application in diabetes blood glucose control: A systematic review. Artificial intelligence in medicine, 104:101836, 2020.
  3. Intenational Diabetes Federation. Idf diabetes atlas, tenth. International Diabetes, 2021.
  4. The use of reinforcement learning algorithms to meet the challenges of an artificial pancreas. Expert review of medical devices, 10(5):661–673, 2013.
  5. Model-informed artificial intelligence: reinforcement learning for precision dosing. Clinical Pharmacology & Therapeutics, 107(4):853–857, 2020.
  6. On the challenges of using reinforcement learning in precision drug dosing: Delay and prolongedness of action effects. arXiv preprint arXiv:2301.00512, 2023.
  7. A brief survey of deep reinforcement learning. arXiv preprint arXiv:1708.05866, 2017.
  8. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015.
  9. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
  10. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015.
  11. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995–2003. PMLR, 2016.
  12. Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3:9–44, 1988.
  13. Reinforcement learning: An introduction. Robotica, 17(2):229–235, 1999.
  14. A distributional perspective on reinforcement learning. In International conference on machine learning, pages 449–458. PMLR, 2017.
  15. Noisy networks for exploration. arXiv preprint arXiv:1706.10295, 2017.
  16. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  17. Continuous state-space models for optimal sepsis treatment: a deep reinforcement learning approach. In Machine Learning for Healthcare Conference, pages 147–163. PMLR, 2017.
  18. The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature medicine, 24(11):1716–1720, 2018.
  19. Optimization of anemia treatment in hemodialysis patients via reinforcement learning. Artificial intelligence in medicine, 62(1):47–60, 2014.
  20. Linear fitted-q iteration with multiple reward functions. The Journal of Machine Learning Research, 13(1):3253–3295, 2012.
  21. Optimal medication dosing from suboptimal clinical examples: A deep reinforcement learning approach. In 2016 38th annual international conference of the IEEE engineering in medicine and biology society (EMBC), pages 2978–2981. IEEE, 2016.
  22. Deep reinforcement learning for optimal critical care pain management with morphine using dueling double-deep q networks. In 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 3960–3963. IEEE, 2019.
  23. Reinforcement learning for blood glucose control: Challenges and opportunities. 2019.
  24. Deep reinforcement learning for closed-loop blood glucose control. In Machine Learning for Healthcare Conference, pages 508–536. PMLR, 2020.
  25. Reinforcement learning for diabetes blood glucose control with meal information. In Bioinformatics Research and Applications: 17th International Symposium, ISBRA 2021, Shenzhen, China, November 26–28, 2021, Proceedings 17, pages 80–91. Springer, 2021.
  26. Basal glucose control in type 1 diabetes using deep reinforcement learning: An in silico validation. IEEE Journal of Biomedical and Health Informatics, 25(4):1223–1232, 2020.
  27. Evaluating deep q-learning algorithms for controlling blood glucose in in silico type 1 diabetes. Diagnostics, 13(19):3150, 2023.
  28. Recent advances in deep reinforcement learning applications for solving partially observable markov decision processes (pomdp) problems: Part 1—fundamentals and applications in games, robotics and natural language processing. Machine Learning and Knowledge Extraction, 3(3):554–581, 2021.
  29. Memory-based deep reinforcement learning for pomdps. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5619–5626. IEEE, 2021.
  30. Understanding multi-step deep reinforcement learning: A systematic study of the dqn target. arXiv preprint arXiv:1901.07510, 2019.
  31. Prioritized experience replay-based deep q learning: Multiple-reward architecture for highway driving decision making. IEEE Robotics & Automation Magazine, 28(4):21–31, 2021.
  32. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928–1937. PMLR, 2016.
  33. Solving unit commitment problems with multi-step deep reinforcement learning. In 2021 IEEE international conference on communications, control, and computing technologies for smart grids (SmartGridComm), pages 140–145. IEEE, 2021.
  34. Jinyu Xie. Simglucose v0.2.1 (2018) [online], 2018.
  35. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
  36. Physical activity into the meal glucose—insulin model of type 1 diabetes: In silico studies, 2009.
  37. The uva/padova type 1 diabetes simulator: new features. Journal of diabetes science and technology, 8(1):26–34, 2014.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now