Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Reward Certification for Policy Smoothed Reinforcement Learning (2312.06436v2)

Published 11 Dec 2023 in cs.LG and cs.AI

Abstract: Reinforcement Learning (RL) has achieved remarkable success in safety-critical areas, but it can be weakened by adversarial attacks. Recent studies have introduced "smoothed policies" in order to enhance its robustness. Yet, it is still challenging to establish a provable guarantee to certify the bound of its total reward. Prior methods relied primarily on computing bounds using Lipschitz continuity or calculating the probability of cumulative reward above specific thresholds. However, these techniques are only suited for continuous perturbations on the RL agent's observations and are restricted to perturbations bounded by the $l_2$-norm. To address these limitations, this paper proposes a general black-box certification method capable of directly certifying the cumulative reward of the smoothed policy under various $l_p$-norm bounded perturbations. Furthermore, we extend our methodology to certify perturbations on action spaces. Our approach leverages f-divergence to measure the distinction between the original distribution and the perturbed distribution, subsequently determining the certification bound by solving a convex optimisation problem. We provide a comprehensive theoretical analysis and run sufficient experiments in multiple environments. Our results show that our method not only improves the certified lower bound of mean cumulative reward but also demonstrates better efficiency than state-of-the-art techniques.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Estimates of the proximity of Gaussian measures. Doklady Mathematics, 34: 462–.
  2. An old-new concept of convex risk measures: The optimized certainty equivalent. Mathematical Finance, 17(3): 449–476.
  3. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, 3387–3395.
  4. Transfer from simulation to real world through learning deep inverse dynamics model. arXiv preprint arXiv:1610.03518.
  5. Certified adversarial robustness via randomized smoothing. In international conference on machine learning, 1310–1320. PMLR.
  6. Csiszár, I. 1967. Information-type measures of difference of probability distributions and indirect observation. Studia Scientiarum Mathematicarum Hungarica, 2: 299–318.
  7. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research, 17(83): 1–5.
  8. Black-box detection of backdoor attacks with limited information and data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 16482–16491.
  9. Ehlers, R. 2017. Formal verification of piece-wise linear feed-forward neural networks. In Automated Technology for Verification and Analysis: 15th International Symposium, ATVA 2017, Pune, India, October 3–6, 2017, Proceedings 15, 269–286. Springer.
  10. Maximum Entropy RL (Provably) Solves Some Robust RL Problems. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
  11. Explaining and Harnessing Adversarial Examples. In Bengio, Y.; and LeCun, Y., eds., 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
  12. Enhancing adversarial training with second-order statistics of weights. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15273–15283.
  13. Residual reinforcement learning for robot control. In 2019 International Conference on Robotics and Automation (ICRA), 6023–6029. IEEE.
  14. Policy smoothing for provably robust reinforcement learning. arXiv preprint arXiv:2106.11420.
  15. Certifying confidence via randomized smoothing. Advances in Neural Information Processing Systems, 33: 5165–5177.
  16. Certified robustness to adversarial examples with differential privacy. In 2019 IEEE symposium on security and privacy (SP), 656–672. IEEE.
  17. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971.
  18. Tactics of Adversarial Attack on Deep Reinforcement Learning Agents. In Sierra, C., ed., Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, 3756–3762. ijcai.org.
  19. Towards Deep Learning Models Resistant to Adversarial Attacks. CoRR, abs/1706.06083.
  20. Manikandan, S.; et al. 2011. Measures of central tendency: Median and mode. J Pharmacol Pharmacother, 2(3): 214–215.
  21. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602.
  22. Certified policy smoothing for cooperative multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 15046–15054.
  23. Sparse adversarial video attacks with spatial transformations. arXiv preprint arXiv:2111.05468.
  24. 3DVerifier: efficient robustness verification for 3D point cloud models. Machine Learning, 1–28.
  25. Virtual to real reinforcement learning for autonomous driving. arXiv preprint arXiv:1704.03952.
  26. Robust deep reinforcement learning with adversarial attacks. arXiv preprint arXiv:1712.03632.
  27. Robust Deep Reinforcement Learning with Adversarial Attacks. In André, E.; Koenig, S.; Dastani, M.; and Sukthankar, G., eds., Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2018, Stockholm, Sweden, July 10-15, 2018, 2040–2042. International Foundation for Autonomous Agents and Multiagent Systems Richland, SC, USA / ACM.
  28. Optimal attacks on reinforcement learning policies. arXiv preprint arXiv:1907.13548.
  29. Deep reinforcement learning framework for autonomous driving. arXiv preprint arXiv:1704.02532.
  30. Deep reinforcement learning with robust and smooth policy. In International Conference on Machine Learning, 8707–8718. PMLR.
  31. Empirical bernstein boosting. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, 733–740. JMLR Workshop and Conference Proceedings.
  32. Intriguing properties of neural networks. In Bengio, Y.; and LeCun, Y., eds., 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
  33. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International conference on machine learning, 5286–5295. PMLR.
  34. Crop: Certifying robust policies for reinforcement learning through functional smoothing. arXiv preprint arXiv:2106.09292.
  35. Reachability analysis of neural network control systems. arXiv preprint arXiv:2301.12100.
  36. Walking on the Edge: Fast, Low-Distortion Adversarial Examples. IEEE Transactions on Information Forensics and Security, 16: 701–713.
  37. Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems, 33: 21024–21037.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Ronghui Mu (12 papers)
  2. Leandro Soriano Marcolino (12 papers)
  3. Tianle Zhang (22 papers)
  4. Yanghao Zhang (10 papers)
  5. Xiaowei Huang (121 papers)
  6. Wenjie Ruan (42 papers)
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now