Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 43 tok/s
Gemini 2.5 Pro 48 tok/s Pro
GPT-5 Medium 21 tok/s Pro
GPT-5 High 20 tok/s Pro
GPT-4o 95 tok/s Pro
Kimi K2 180 tok/s Pro
GPT OSS 120B 443 tok/s Pro
Claude Sonnet 4.5 32 tok/s Pro
2000 character limit reached

How Safe Am I Given What I See? Calibrated Prediction of Safety Chances for Image-Controlled Autonomy (2308.12252v4)

Published 23 Aug 2023 in cs.LG

Abstract: End-to-end learning has emerged as a major paradigm for developing autonomous systems. Unfortunately, with its performance and convenience comes an even greater challenge of safety assurance. A key factor of this challenge is the absence of the notion of a low-dimensional and interpretable dynamical state, around which traditional assurance methods revolve. Focusing on the online safety prediction problem, this paper proposes a configurable family of learning pipelines based on generative world models, which do not require low-dimensional states. To implement these pipelines, we overcome the challenges of learning safety-informed latent representations and missing safety labels under prediction-induced distribution shift. These pipelines come with statistical calibration guarantees on their safety chance predictions based on conformal prediction. We perform an extensive evaluation of the proposed learning pipelines on two case studies of image-controlled systems: a racing car and a cartpole.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (77)
  1. Competency Assessment for Autonomous Agents using Deep Generative Models. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 8211–8218. ISSN: 2153-0866.
  2. Learning to Forecast Aleatoric and Epistemic Uncertainties over Long Horizon Trajectories. arXiv:2302.08669.
  3. Control Barrier Functions: Theory and Applications. In 2019 18th European Control Conference (ECC), 3420–3431.
  4. Real time trajectory prediction for collision risk estimation between vehicles. In 2009 IEEE 5th International Conference on Intelligent Computer Communication and Processing, 417–422.
  5. Hamilton-Jacobi reachability: A brief overview and recent advances. In 2017 IEEE 56th Annual Conference on Decision and Control (CDC), 2242–2253.
  6. Competence-Aware Systems. Artificial Intelligence, 103844.
  7. Autonomous Vehicles on the Edge: A Survey on Autonomous Vehicle Racing. IEEE Open Journal of Intelligent Transportation Systems, 3: 458–488. Conference Name: IEEE Open Journal of Intelligent Transportation Systems.
  8. Assurance monitoring of learning-enabled cyber-physical systems using inductive conformal prediction based on distance learning. AI EDAM, 35(2): 251–264. Publisher: Cambridge University Press.
  9. OpenAI Gym. ArXiv:1606.01540 [cs].
  10. Reachability Analysis for Cyber-Physical Systems: Are We There Yet? In NASA Formal Methods Symposium, 109–130. Springer.
  11. Synthesizing Stable Reduced-Order Visuomotor Policies for Nonlinear Systems via Sums-of-Squares Optimization. ArXiv:2304.12405 [cs, eess, math].
  12. Risk verification of stochastic systems with neural network controllers. Artificial Intelligence, 313: 103782.
  13. Conservative Safety Monitors of Stochastic Dynamical Systems. In Proc. of the NASA Formal Methods Conference.
  14. End-to-End Driving Via Conditional Imitation Learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), 1–9. Brisbane, Australia: IEEE Press.
  15. “I’m Confident This Will End Poorly”: Robot Proficiency Self-Assessment in Human-Robot Teaming. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2127–2134. ISSN: 2153-0866.
  16. Addressing Failure Prediction by Learning Model Confidence. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
  17. Guaranteeing Safety of Learned Perception Modules via Measurement-Robust Control Barrier Functions. In Proceedings of the 2020 Conference on Robot Learning, 654–670. PMLR. ISSN: 2640-3498.
  18. Learning Confidence for Out-of-Distribution Detection in Neural Networks. ArXiv:1802.04865 [cs, stat].
  19. Formal Verification of End-to-End Learning in Cyber-Physical Systems: Progress and Challenges. In NeurIPS Workshop on Safety and Robustness in Decision Making. arXiv. ArXiv:2006.09181 [cs, stat].
  20. A Method for Designing Autonomous Robots that Know Their Limits. In 2022 International Conference on Robotics and Automation (ICRA), 121–127.
  21. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, 1321–1330. Sydney, NSW, Australia: JMLR.org.
  22. Recurrent World Models Facilitate Policy Evolution. In Advances in Neural Information Processing Systems, volume 31.
  23. Dream to Control: Learning Behaviors by Latent Imagination. In In Proc. of International Conference on Learning Representations.
  24. Mastering Atari with Discrete World Models. arXiv:2010.02193.
  25. Safely: Safe Stochastic Motion Planning Under Constrained Sensing via Duality. ArXiv:2203.02816 [cs, eess].
  26. Verifying Controllers With Vision-Based Perception Using Safe Approximate Abstractions. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 41(11): 4205–4216. Conference Name: IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.
  27. A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability. Computer Science Review, 37: 100270.
  28. A Survey on Trajectory-Prediction Methods for Autonomous Driving. IEEE Transactions on Intelligent Vehicles, 7(3): 652–674. Conference Name: IEEE Transactions on Intelligent Vehicles.
  29. Imitation Learning: A Survey of Learning Methods. ACM Computing Surveys, 50(2): 21:1–21:35.
  30. Verisig 2.0: Verification of Neural Network Controllers Using Taylor Model Preconditioning. In Computer Aided Verification, 249–262. Cham: Springer International Publishing. ISBN 978-3-030-81685-8.
  31. Verification of Image-based Neural Network Controllers Using Generative Models. arXiv:2105.07091.
  32. Auto-Encoding Variational Bayes. In Proc. of the International Conference on Learning Representations (ICLR).
  33. Planning With Learned Dynamics: Probabilistic Guarantees on Safety and Reachability via Lipschitz Constants. IEEE Robotics and Automation Letters, PP: 1–1.
  34. Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples.
  35. A survey on motion prediction and risk assessment for intelligent vehicles. ROBOMECH Journal, 1(1): 1.
  36. Distribution-Free Predictive Inference For Regression. Journal of the American Statistical Association. ArXiv: 1604.04173.
  37. Prediction-Based Reachability for Collision Avoidance in Autonomous Driving. In 2021 IEEE International Conference on Robotics and Automation (ICRA), 7908–7914. ISSN: 2577-087X.
  38. A survey on long short-term memory networks for time series prediction. Procedia CIRP, 99: 650–655.
  39. Conformal Prediction for STL Runtime Verification. In Proc. of ICCPS’23. San Antonio, TX: arXiv. ArXiv:2211.01539 [cs, eess].
  40. Meaning Representations from Trajectories in Autoregressive Models. arXiv preprint arXiv:2310.18348.
  41. Modular Conformal Calibration. In Proceedings of the 39th International Conference on Machine Learning, 15180–15195. PMLR. ISSN: 2640-3498.
  42. Transformers are Sample-Efficient World Models. arXiv:2209.00588.
  43. Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Control. 2020 IEEE International Conference on Robotics and Automation (ICRA).
  44. Revisiting the Calibration of Modern Neural Networks. In Advances in Neural Information Processing Systems, volume 34, 15682–15694. Curran Associates, Inc.
  45. Human-level control through deep reinforcement learning. nature, 518(7540): 529–533.
  46. A unifying view on dataset shift in classification. Pattern Recognition, 45(1): 521–530.
  47. Multi-Agent Reachability Calibration with Conformal Prediction. ArXiv:2304.00432 [cs, eess].
  48. Obtaining Well Calibrated Probabilities Using Bayesian Binning. Proceedings of the AAAI Conference on Artificial Intelligence, 2015: 2901–2907.
  49. Online Update of Safety Assurances Using Confidence-Based Predictions. arXiv:2210.01199.
  50. A Review on Deep Learning Techniques for Video Prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(6): 2806–2826.
  51. Platt, J. C. 1999. Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. In Advances in Large Margin Classifiers, 61–74. MIT Press.
  52. Statistical Verification of Cyber-Physical Systems using Surrogate Models and Conformal Inference. In 2022 ACM/IEEE 13th International Conference on Cyber-Physical Systems (ICCPS), 116–126.
  53. Statistical Verification of Autonomous Systems using Surrogate Models and Conformal Inference. In Proc. of ICCPS’22. ArXiv: 2004.00279.
  54. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748–8763. PMLR.
  55. Robot Vitals and Robot Health: Towards Systematically Quantifying Runtime Performance Degradation in Robots Under Adverse Conditions. IEEE Robotics and Automation Letters, 7(4): 10729–10736.
  56. Confidence Composition for Monitors of Verification Assumptions. In ACM/IEEE 13th Intl. Conf. on Cyber-Physical Systems (ICCPS), 1–12.
  57. Trajectron++: Dynamically-Feasible Trajectory Forecasting with Heterogeneous Data. In Computer Vision – ECCV 2020, Lecture Notes in Computer Science, 683–700. Cham: Springer International Publishing. ISBN 978-3-030-58523-5.
  58. NNLander-VeriF: A Neural Network Formal Verification Framework for Vision-Based Autonomous Aircraft Landing. In Deshmukh, J. V.; Havelund, K.; and Perez, I., eds., NASA Formal Methods, Lecture Notes in Computer Science, 213–230. Cham: Springer International Publishing. ISBN 978-3-031-06773-0.
  59. Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
  60. Learning Structured Output Representation using Deep Conditional Generative Models. In Cortes, C.; Lawrence, N.; Lee, D.; Sugiyama, M.; and Garnett, R., eds., Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
  61. Misbehaviour Prediction for Autonomous Driving Systems. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE), 359–371.
  62. Deep Predictive Models for Collision Risk Assessment in Autonomous Driving. arXiv:1711.10453.
  63. Vision-based Intention and Trajectory Prediction in Autonomous Vehicles: A Survey. volume 6, 5630–5637. ISSN: 1045-0823.
  64. Learning Representations for Pixel-based Control: What Matters and Why? Transactions on Machine Learning Research.
  65. Verification Approaches for Learning-Enabled Autonomous Cyber–Physical Systems. IEEE Design & Test, 39(1): 24–34. Conference Name: IEEE Design & Test.
  66. Algorithmic Learning in a Random World. New York: Springer, 2005 edition edition. ISBN 978-0-387-00152-4.
  67. Conformal calibrators. In Proceedings of the Ninth Symposium on Conformal and Probabilistic Prediction and Applications, 84–99. PMLR. ISSN: 2640-3498.
  68. Predicting video with vqvae. arXiv preprint arXiv:2103.01950.
  69. Social ODE: Multi-agent Trajectory Forecasting with Neural Ordinary Differential Equations. In Computer Vision – ECCV 2022, Lecture Notes in Computer Science, 217–233. Cham: Springer Nature Switzerland. ISBN 978-3-031-20047-2.
  70. Safe Autonomy with Control Barrier Functions: Theory and Applications. Synthesis Lectures on Computer Science. Cham: Springer International Publishing. ISBN 978-3-031-27575-3 978-3-031-27576-0.
  71. Distance-Based Learning from Errors for Confidence Calibration. In In Proc. of International Conference on Learning Representations.
  72. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157.
  73. Safe Perception-Based Control under Stochastic Sensor Uncertainty using Conformal Prediction. In CDC 2023. arXiv. ArXiv:2304.00194 [cs, eess].
  74. Interpretable Detection of Distribution Shifts in Learning Enabled Cyber-Physical Systems. In 2022 ACM/IEEE 13th International Conference on Cyber-Physical Systems (ICCPS), 225–235.
  75. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, KDD ’02, 694–699. New York, NY, USA: Association for Computing Machinery. ISBN 978-1-58113-567-1.
  76. Statistical Verification of Learning-Based Cyber-Physical Systems. In Proceedings of the 23rd International Conference on Hybrid Systems: Computation and Control, HSCC ’20. New York, NY, USA: Association for Computing Machinery. ISBN 9781450370189.
  77. Mix-n-Match: ensemble and compositional methods for uncertainty calibration in deep learning. In Proceedings of the 37th International Conference on Machine Learning, ICML’20, 11117–11128. JMLR.org.
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Don't miss out on important new AI/ML research

See which papers are being discussed right now on X, Reddit, and more:

Explore Trending Papers

“Emergent Mind helps me see which AI papers have caught fire online.”

Philip

Philip

Creator, AI Explained on YouTube