- The paper introduces Staged-Competence, a curriculum learning framework that orders training from simple to complex examples to improve LLM safety alignment.
- It employs a global difficulty score based on the preference alignment margin and staged reference updates to significantly reduce harmful outputs.
- Empirical results demonstrate up to a 29-point reduction in harmful out-of-distribution responses and robust capability preservation compared to standard DPO.
Curriculum-Based Safety Alignment in LLMs: An Expert Review of "Curriculum Learning for Safety Alignment" (2605.26315)
Direct Preference Optimization and Its Limitations
Direct Preference Optimization (DPO) has become a standard for safety alignment in LLMs, leveraging human-annotated preference pairs to distinguish safe from unsafe responses. The standard DPO formulation refrains from building a separate reward model, instead utilizing a fixed reference policy to penalize deviations from human-preferred safe outputs. However, DPO exhibits well-characterized brittleness, particularly under out-of-distribution (OOD) conditions and adversarial attacks such as jailbreaks, which frequently exploit the shallow nature of safety alignment—often concentrated in the initial response tokens (see [20]). Fine-tuning can further compromise safety generalization, with models easily reverting to harmful behavior or failing to resist sophisticated attacks [19, 29, 12, 6].
Curriculum Learning Paradigm and Staged-Competence Framework
Curriculum learning, originally introduced in [3], has proven effective in guiding models to learn robust features by ordering examples from easy to hard. This approach has been adapted to a broad spectrum of tasks, including NMT, pretraining, and various alignment objectives [18, 25, 13]. The authors of this paper systematically extend curriculum learning to the safety alignment domain by proposing the Staged-Competence framework—an improvement over Curri-DPO [17]—tailored for DPO-based training.
Difficulty Scoring Via Preference Alignment Margin
Unlike prompt-local or static heuristics, Staged-Competence adopts a model-dependent global scoring using the preference alignment margin:
mi=cos(ey^i,eyi+)−cos(ey^i,eyi−),
where ey^i is the embedding of the zero-shot response and eyi+,eyi− are embeddings for the safe and unsafe responses, respectively. These normalized sentence embeddings are computed using all-MiniLM-L6-v2 [22]. The margin quantifies how closely the model already aligns with safe responses, enabling curriculum construction that is global across the dataset.
Curriculum Imposition and Reference Model Update
The algorithm splits the sorted data into K buckets of ascending difficulty. Within each stage, competence-based sampling restricts the eligible training pool by maintaining a dynamic difficulty threshold, expanded progressively according to a square-root schedule [18]. Reference models are updated between stages, ensuring that each training phase consolidates prior learning before progressing to harder cases. This mechanism prevents the model from constantly relearning shallow features and enables deeper, more persistent safety alignment.
Empirical Evaluation and Results
Dataset Construction and Cleaning
The paper identifies significant label inconsistencies and safety failures in PKU-SafeRLHF [10] and HH-RLHF [2]. Using a GPT-4o-mini judge, the authors filter and curate Cleaned-PKU-HH-SafeRLHF, retaining only pairs with correctly assigned safe/unsafe designations. This rigorous cleaning is essential for reliable DPO-based safety alignment and addresses widespread dataset pitfalls.
Benchmarking Across Model Architectures
Three abliterated model families—LLaMA-3-8B, Qwen3-8B, Yi-1.5-9B—are used, stripping native safety mechanisms to force alignment entirely via training.
Staged-Competence delivers reward accuracy on par with or exceeding Standard DPO (e.g., 91.3% for LLaMA-3-8B vs. 89.8% with baseline). More notably, margin separation between safe and unsafe responses grows to approximately 3x the baseline, indicating increased confidence. Gains are most pronounced on OOD safety benchmarks:
- Average reduction in harmful OOD response rate: up to 29 points for Qwen3-8B.
- Attack robustness: Up to 27.1 points reduction in jailbreak attack success rate (GCG) on Qwen3-8B.
- These improvements surpass all curriculum and baseline variants, substantiating the claim that curriculum ordering combined with staged reference updates can markedly increase practical safety robustness.
Alignment Depth and Resistance to Prefill Attacks
Per-token suppression analysis shows uniform improvements across the entire response—Staged-Competence actively suppresses unsafe content well beyond the refusal tokens, in contrast to Standard DPO's shallow alignment. This deeper alignment mechanistically blocks attacks that attempt to bypass front-loaded refusal, establishing stronger resilience.
Capability Preservation and Over-Refusal Control
General capabilities (MMLU, HellaSwag) remain within ~3 points of the baseline, and over-refusal rates are near-zero, demonstrating that improved safety does not degrade utility or induce excessive refusal of benign prompts.
Data Efficiency and Scaling
Staged-Competence matches or outperforms Standard DPO with 25% less training data across all OOD safety metrics; at 50%, there is some model dependency, but the framework still delivers substantial gains on certain architectures (notably LLaMA-3-8B). When scaling Qwen3 models from 1.7B to 8B parameters, the absolute benefit of curriculum alignment monotonically increases, suggesting that curriculum learning becomes more critical as model capacity—and potential for harmful output—grows.
Practical and Theoretical Implications
Staged-Competence demonstrates that curriculum learning, when properly designed for difficulty progression and reference model updates, is an effective and scalable lever for safety alignment in LLMs. The framework is agnostic to the policy optimization loss, permitting extension to other DPO variants and alignment domains, including those outside safety. The observed data efficiency opens avenues for robust safety training in settings where preference annotation is limited or expensive. The results point toward curriculum learning as necessary for mitigating alignment brittleness, especially for larger and more capable models.
The deeper safety alignment attained by Staged-Competence offers a mechanistic defense against attacks targeting mid-sequence vulnerabilities. This property could be increasingly relevant as adversarial attack methodologies evolve. The cleaned dataset and open code serve as practically useful resources for further research.
Speculation on Future Developments
Extensions of Staged-Competence include:
- Application to full-model fine-tuning and larger architectures.
- Generalization to non-safety alignment objectives (e.g., fairness, domain robustness).
- On-policy curriculum adjustment, integrating dynamic difficulty metrics.
- Automated curriculum design using reinforcement learning.
Curriculum learning's increasing value with model scale suggests it may become a fundamental component of LLM alignment pipelines. Research will likely formalize curriculum complexity and learning progression metrics into standard alignment evaluation criteria.
Conclusion
"Curriculum Learning for Safety Alignment" presents strong evidence that structured curriculum—global difficulty ordering, staged reference updates, and competence-based sampling—substantially improves safety alignment robustness in LLMs without compromising utility or inducing over-refusal. The Staged-Competence framework represents a flexible, data-efficient approach whose benefits grow with model scale, establishing curriculum learning as a central methodology for future alignment research and deployment.