OpenAI o1 System Card (2412.16720v1)

Abstract: The o1 model series is trained with large-scale reinforcement learning to reason using chain of thought. These advanced reasoning capabilities provide new avenues for improving the safety and robustness of our models. In particular, our models can reason about our safety policies in context when responding to potentially unsafe prompts, through deliberative alignment. This leads to state-of-the-art performance on certain benchmarks for risks such as generating illicit advice, choosing stereotyped responses, and succumbing to known jailbreaks. Training models to incorporate a chain of thought before answering has the potential to unlock substantial benefits, while also increasing potential risks that stem from heightened intelligence. Our results underscore the need for building robust alignment methods, extensively stress-testing their efficacy, and maintaining meticulous risk management protocols. This report outlines the safety work carried out for the OpenAI o1 and OpenAI o1-mini models, including safety evaluations, external red teaming, and Preparedness Framework evaluations.

• The paper introduces the OpenAI o1 model series, trained using large-scale reinforcement learning and chain-of-thought reasoning to enhance advanced capabilities and safety through deliberative alignment.
• Extensive evaluations show the o1 models achieve significant improvements in mitigating disallowed content, resisting jailbreaks, reducing hallucination, and improving bias and fairness compared to previous models.
• Safety challenges are addressed through structured red teaming, external evaluations by organizations like the US and UK AISI, monitoring chain-of-thought risks, and assessments via a preparedness framework.

The paper "OpenAI o1 System Card" provides an extensive analysis and evaluation report of the OpenAI o1 model series, which has been trained using large-scale reinforcement learning with a focus on advanced reasoning capabilities through a chain-of-thought approach. This methodological framework aims to enhance the model's robustness and safety by allowing it to reason through safety specifications—in a process referred to as deliberative alignment—before generating responses. The authors claim state-of-the-art performance in mitigating risks such as generating illicit advice and avoiding stereotypical responses.

Model Data and Training

• The o1 models are designed to produce complex reasoning outputs, allowing them to deliberate before answering.
• The training utilized diverse datasets, including selective public data, proprietary data accessed through partnerships, and internal custom datasets.
• Data is meticulously filtered to maintain quality and mitigate risks, using tools such as OpenAI's Moderation API to exclude explicit and sensitive content.

Scope of Testing and Model Evaluation

OpenAI employed a comprehensive series of evaluations for the o1 models:

• Disallowed Content Evaluations: Compared to its predecessors, the o1 series shows notable improvements in handling disallowed content, demonstrating near-perfect not_unsafe results across various tests.
• Jailbreak Evaluations: The paper indicates significant advancements in resilience against adversarial prompts aimed at circumventing content restrictions.
• Hallucination Evaluations: The o1 models exhibit reduced rates of hallucination, thereby improving accuracy and reliability in information-seeking tasks with metrics collected from datasets like SimpleQA and PersonQA.
• Bias and Fairness: The evaluations suggest that the o1 model exhibits improvements in avoiding stereotypical outputs and achieving demographic fairness compared to previous iterations like GPT-4o.

Safety Challenges and Mitigations

• The system highlights observed safety challenges and introduces monitoring approaches for chain-of-thought (CoT) reasoning to manage potential risks of deception and misinformation.
• A notable aspect of the safety strategy involves structured red teaming exercises, external evaluations by organizations like the US AISI and UK AISI, and systematic testing through the Preparedness Framework.

External Red Teaming and Safety Evaluations

• The o1 model underwent extensive red teaming efforts to assess vulnerabilities and potential misuse.
• In competitive settings like Gray Swan's jailbreak arena, o1 demonstrated robust resistance to attack strategies aimed at generating harmful content.
• Apollo Research provided insights into potential deceptive alignment risks, evaluating o1 for its ability to engage in complex reasoning and potentially misaligned actions while following developer instructions.

Preparedness Framework

• The authors categorize the model's capabilities in risk categories, maintaining pre-mitigation and post-mitigation assessments to ensure safety and reliability.
• Preparedness evaluations focus on core areas, including cybersecurity, CBRN capabilities, persuasion, and model autonomy.

Conclusion

Overall, the introduction of chain-of-thought methodology in the o1 series notably enhances both capability and safety performance, highlighting the balancing act between increased capacities and emerging risks. The paper strongly emphasizes the need for ongoing rigorous safety evaluations and real-world deployment experiments to fully scope potential capabilities and threats associated with LLMs of the o1 series. The authors acknowledge that iterative deployment and collaboration are crucial to thoroughly understanding and mitigating risks associated with advanced AI systems.