Papers
Topics
Authors
Recent
Search
2000 character limit reached

Planning and Learning in Risk-Aware Restless Multi-Arm Bandit Problem

Published 30 Oct 2024 in cs.LG, cs.SY, and eess.SY | (2410.23029v2)

Abstract: In restless multi-arm bandits, a central agent is tasked with optimally distributing limited resources across several bandits (arms), with each arm being a Markov decision process. In this work, we generalize the traditional restless multi-arm bandit problem with a risk-neutral objective by incorporating risk-awareness. We establish indexability conditions for the case of a risk-aware objective and provide a solution based on Whittle index. In addition, we address the learning problem when the true transition probabilities are unknown by proposing a Thompson sampling approach and show that it achieves bounded regret that scales sublinearly with the number of episodes and quadratically with the number of arms. The efficacy of our method in reducing risk exposure in restless multi-arm bandits is illustrated through a set of numerical experiments in the contexts of machine replacement and patient scheduling applications under both planning and learning setups.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (65)
  1. A near-optimal maintenance policy for automated DR devices. 7(3):1411–1419.
  2. Group maintenance: A restless bandits approach. INFORMS Journal on Computing, 31(4):719–731.
  3. Thompson sampling for contextual bandits with linear payoffs. In Int. Conf. Mach. Learn., pages 127–135. PMLR.
  4. Dynamic spectrum access under partial observations: A restless bandit approach. In Canadian Workshop on Information Theory, pages 1–6. IEEE.
  5. Restless bandits with controlled restarts: Indexability and computation of Whittle index. In Conf. Decision Control, pages 7294–7300.
  6. Conditions for indexability of restless bandits and an 𝒪⁢(k3)𝒪superscript𝑘3\mathcal{O}\!\left(k^{3}\right)caligraphic_O ( italic_k start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT ) algorithm to compute Whittle index. Adv. in Appl. Probab., 54(4):1164–1192.
  7. On learning Whittle index policy for restless bandits with scalable regret. IEEE Transactions on Control of Network Systems.
  8. Providing an integrated model for planning and scheduling energy hubs and preventive maintenance. Energy, 163:1093–1114.
  9. Anonymous (2024). Risk-aware Whittle index.
  10. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47:235–256.
  11. Congestion control of TCP flows in internet routers by means of index policy. Comm Com Inf Sc, 57(17):3463–3478.
  12. Whittle index policy for crawling ephemeral content. IEEE Transactions on Control of Network Systems, 5(1):446–455.
  13. Markov decision processes with average-value-at-risk criteria. Mathematical Methods of Operations Research, 74:361–379.
  14. More risk-sensitive Markov decision processes. Mathematics of Operations Research, 39(1):105–120.
  15. Opportunistic scheduling as restless bandits. 5(4):1952–1961.
  16. Risk-sensitive and robust decision-making: a cvar optimization approach. Advances in neural information processing systems, 28.
  17. Scalable operator allocation for multirobot assistance: A restless bandit approach. 9(3):1397–1408.
  18. Improving health outcomes through better capacity allocation in a community-based chronic care model. Operat. Res., 61(6):1277–1294.
  19. Fishburn, P. C. (1968). Utility theory. Management science, 14(5):335–378.
  20. Gittins, J. C. (1979). Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society. Series B, pages 148–177.
  21. Monotone policies and indexability for bidirectional restless bandits. Adv. Appl. Prob., 45(1):51–85.
  22. Index policies for the maintenance of a collection of machines by a set of repairmen. Euro. J. Operat. Res., 165(1):267–284.
  23. Some indexable families of restless bandit problems. Adv. Appl. Prob., 38(3):643–672.
  24. Gunantara, N. (2018). A review of multi-objective optimization: Methods and its applications. Cogent Engineering, 5(1):1502242.
  25. Controlling understaffing with conditional value-at-risk constraint for an integrated nurse scheduling problem under patient demand uncertainty. Operations Research Perspectives, 6:100119.
  26. Opportunistic schedulers for optimal scheduling of flows in wireless systems with arq feedback. In International Teletraffic Congress, page 14.
  27. Thompson sampling in non-episodic restless bandits. arXiv preprint arXiv:1910.05654.
  28. Le, Q. A. (2016). Structural uncertainty of markov models for advanced breast cancer: A simulation study of lapatinib. Medical Decision Making, 36(5):629–640.
  29. Le Tallec, Y. (2007). Robust, risk-sensitive, and data-driven control of Markov decision processes. PhD thesis, Massachusetts Institute of Technology.
  30. Indexability of restless bandit problems and optimality of Whittle index for dynamic multichannel access. 56(11):5547–5567.
  31. Robust satisficing. Operations Research, 71(1):61–82.
  32. On the optimality of an index rule in multichannel allocation for single-hop mobile networks with multiple service classes. Probab. Engrg. Inform. Sci., 14(3):259–297.
  33. Robust mdps with k-rectangular uncertainty. Mathematics of Operations Research, 41(4):1484–1509.
  34. The weighted sum method for multi-objective optimization: new insights. Structural and multidisciplinary optimization, 41:853–862.
  35. Risk-aware interventions in public health: Planning with restless multi-armed bandits. In AAMAS, pages 880–888.
  36. A variability reduction method for the operating room scheduling problem under uncertainty using cvar. Operations Research for Health Care, 20:25–32.
  37. Niño-Mora, J. (2007). Dynamic priority allocation via restless bandit marginal productivity indices. TOP, 15(2):161–198.
  38. Nino-Mora, J. (2009). A restless bandit marginal productivity index for opportunistic spectrum access with sensing errors. In International Conference on Network Control and Optimization, pages 60–74. Springer.
  39. Niño-Mora, J. (2023). Markovian restless bandits and index policies: A review. Mathematics, 11(7):1639.
  40. Niño-Mora, J. (2007). A (2/3)⁢n323superscript𝑛3(2/3)n^{3}( 2 / 3 ) italic_n start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT fast-pivoting algorithm for the gittins index and optimal stopping of a Markov Chain. INFORMS J. Comput., 19(4):596–606.
  41. Reinforcement and deep reinforcement learning-based solutions for machine maintenance planning, scheduling policies, and optimization. Journal of Manufacturing Systems, 70:244–263.
  42. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems, 26.
  43. Osogami, T. (2012). Robustness and risk-sensitivity in Markov decision processes. Advances in Neural Information Processing Systems, 25.
  44. Downlink scheduling over Markovian fading channels. 24(3):1801–1812.
  45. Learning unknown Markov decision processes: A Thompson sampling approach. In Neural Inf. Process. Syst., volume 30. Curran Associates, Inc.
  46. Pratt, J. W. (1978). Risk aversion in the small and in the large. In Uncertainty in economics, pages 59–79. Elsevier.
  47. Puterman, M. L. (2014). Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons.
  48. Restless poachers: Handling exploration-exploitation tradeoffs in security domains. In Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems, pages 123–131.
  49. Rausand, M. (2013). Risk assessment: theory, methods, and applications, volume 115. John Wiley & Sons.
  50. Model-informed artificial intelligence: reinforcement learning for precision dosing. Clinical Pharmacology & Therapeutics, 107(4):853–857.
  51. Linearly parameterized bandits. Math. Oper. Res., 35(2):395–411.
  52. Learning to optimize via posterior sampling. Math. Oper. Res., 39(4):1221–1243.
  53. A tutorial on thompson sampling. Foundations and Trends® in Machine Learning, 11(1):1–96.
  54. Contribution of primary care to health systems and health. Milbank, 83(3):457–502.
  55. Judgment under uncertainty: Heuristics and biases. Science, 185(4157):1124–1131.
  56. Theory of games and economic behavior. Princeton University Press.
  57. Whittle index policy for dynamic multichannel allocation in remote state estimation. IEEE Transactions on Automatic Control, 65(2):591–603.
  58. Opportunistic scheduling revisited using restless bandits: Indexability and index policy. 18(10):4997–5010.
  59. Adaptive demand response: Online learning of restless and controlled bandits. In Int. Conf. Smart Grid Commun., pages 752–757. IEEE.
  60. On an index policy for restless bandits. J. Appl. Prob., 27(3):637–648.
  61. Inequalities for the l1 deviation of the empirical distribution. Hewlett-Packard Labs, Tech. Rep.
  62. Whittle, P. (1988). Restless bandits: Activity allocation in a changing world. J. Appl. Prob., 25(A):287–298.
  63. An online algorithm for the risk-aware restless bandit. European Journal of Operational Research, 290(2):622–639.
  64. Regret bounds for Markov decision processes with recursive optimized certainty equivalents. In International Conference on Machine Learning, pages 38400–38427. PMLR.
  65. Deadline scheduling as restless bandits. IEEE Transactions on Automatic Control, 63(8):2343–2358.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

Sign Up for Free