Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Probabilistic Model Checking of Stochastic Reinforcement Learning Policies (2403.18725v1)

Published 27 Mar 2024 in cs.AI

Abstract: We introduce a method to verify stochastic reinforcement learning (RL) policies. This approach is compatible with any RL algorithm as long as the algorithm and its corresponding environment collectively adhere to the Markov property. In this setting, the future state of the environment should depend solely on its current state and the action executed, independent of any previous states or actions. Our method integrates a verification technique, referred to as model checking, with RL, leveraging a Markov decision process, a trained RL policy, and a probabilistic computation tree logic (PCTL) formula to build a formal model that can be subsequently verified via the model checker Storm. We demonstrate our method's applicability across multiple benchmarks, comparing it to baseline methods called deterministic safety estimates and naive monolithic model checking. Our results show that our method is suited to verify stochastic RL policies.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Safe reinforcement learning via shielding. In AAAI, pages 2669–2678. AAAI Press.
  2. Verifying reinforcement learning up to infinity. In IJCAI, pages 2154–2160. ijcai.org.
  3. Probabilistic guarantees for safe deep reinforcement learning. In FORMATS, volume 12288 of Lecture Notes in Computer Science, pages 231–248. Springer.
  4. Verified probabilistic policies for deep reinforcement learning. In NFM, volume 13260 of Lecture Notes in Computer Science, pages 193–212. Springer.
  5. Principles of model checking. MIT Press.
  6. Barhate, N. (2021). Minimal pytorch implementation of proximal policy optimization. https://github.com/nikhilbarhate99/PPO-PyTorch.
  7. Verification of markov decision processes using learning algorithms. In ATVA, volume 8837 of Lecture Notes in Computer Science, pages 98–114. Springer.
  8. Safe reinforcement learning via shielding under partial observability. In AAAI, pages 14748–14756. AAAI Press.
  9. Efficient on-the-fly algorithms for the analysis of timed games. In CONCUR, volume 3653 of Lecture Notes in Computer Science, pages 66–80. Springer.
  10. Formal verification of neural networks for safety-critical tasks in deep reinforcement learning. In de Campos, C. and Maathuis, M. H., editors, Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pages 333–343. PMLR.
  11. Verifying temporal properties of finite-state probabilistic programs. In FOCS, pages 338–345. IEEE Computer Society.
  12. The complexity of probabilistic verification. J. ACM, 42(4):857–907.
  13. Uppaal stratego. In TACAS, volume 9035 of Lecture Notes in Computer Science, pages 206–211. Springer.
  14. Permissive controller synthesis for probabilistic systems. Log. Methods Comput. Sci., 11(2).
  15. Verifying learning-augmented systems. In SIGCOMM, pages 305–318. ACM.
  16. A comprehensive survey on safe reinforcement learning. J. Mach. Learn. Res., 16:1437–1480.
  17. COOL-MC: A comprehensive tool for reinforcement learning and model checking. In SETTA, volume 13649 of Lecture Notes in Computer Science, pages 41–49. Springer.
  18. Model checking for adversarial multi-agent reinforcement learning with reactive defense methods. In ICAPS, pages 162–170. AAAI Press.
  19. Model checking for adversarial multi-agent reinforcement learning with reactive defense methods. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 33, pages 162–170.
  20. Omega-regular objectives in model-free reinforcement learning. In TACAS (1), volume 11427 of Lecture Notes in Computer Science, pages 395–412. Springer.
  21. A logic for reasoning about time and reliability. Formal Aspects Comput., 6(5):512–535.
  22. Deep reinforcement learning with temporal logics. In FORMATS, volume 12288 of Lecture Notes in Computer Science, pages 1–22. Springer.
  23. The probabilistic model checker Storm. Int. J. Softw. Tools Technol. Transf., 24(4):589–610.
  24. Efficient LTL model checking of deep reinforcement learning systems using policy extraction. In SEKE, pages 357–362. KSI Research Inc.
  25. Verifying deep-rl-driven systems. In NetAI@SIGCOMM, pages 83–89. ACM.
  26. PRISM 4.0: Verification of probabilistic real-time systems. In CAV, volume 6806 of LNCS, pages 585–591. Springer.
  27. Environment-independent task specifications via GLTL. CoRR, abs/1704.04341.
  28. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602.
  29. Human-level control through deep reinforcement learning. Nat., 518(7540):529–533.
  30. Mastering the game of go with deep neural networks and tree search. Nat., 529(7587):484–489.
  31. Reinforcement learning: An introduction. MIT press.
  32. Scalar reward is not enough: a response to silver, singh, precup and sutton (2021). Auton. Agents Multi Agent Syst., 36(2):41.
  33. Grandmaster level in starcraft II using multi-agent reinforcement learning. Nat., 575(7782):350–354.
  34. Vouros, G. A. (2023). Explainable deep reinforcement learning: State of the art and challenges. ACM Comput. Surv., 55(5):92:1–92:39.
  35. Statistically model checking PCTL specifications on markov decision processes via reinforcement learning. In CDC, pages 1392–1397. IEEE.
  36. Interpretable reinforcement learning of behavior trees. In ICMLC, pages 492–499. ACM.
  37. An inductive synthesis framework for verifiable reinforcement learning. In PLDI, pages 686–701. ACM.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now