Safety and Safety Testing for Advanced Autonomous AI Systems

Develop and validate methods to make advanced autonomous AI systems safe and to properly test their safety prior to deployment.

Background

The authors stress that if advanced autonomous AI systems were built today, the field lacks the know-how to make them safe or to properly test their safety. This underscores the need for foundational safety research and the development of robust testing methodologies.

This open problem is tied to the urgency of creating governance and technical measures that can keep pace with rapid AI progress, ensuring that safety is demonstrably addressed before deployment.

References

If advanced autonomous AI systems were developed today, we would not know how to make them safe, nor how to properly test their safety.

— Managing extreme AI risks amid rapid progress  (2310.17688 - Bengio et al., 2023) in Subsection A path forward

The extent to which Co-Scientist would execute that experiment remains unknown, presenting a potential for meaningful harm~\citep{tang2025risks}.

— Accelerating Scientific Research with Gemini in the Real-World  (2608.26701 - Schmidgall et al., 27 Aug 2026) in Limitations and failure modes, Section 2.3, paragraph “Scope of verification”

Because prefix2048 authorized the complete run, the full result is a locked diagnostic expansion and cannot serve as an untouched checkpoint-selection test. It shows deployment-transfer failure for the fixed candidate; unbiased evaluation of a newly selected model remains future work.

— From Proxy Learning to Driving Decisions: A Transfer-Based Framework for Evaluating Future-Aware Autonomous Driving Planners  (2609.02688 - Wu, 2 Sep 2026) in Section 4.4.3, “Test 3: Deployment-Scale Stability of Local Gains”