Diagnose post-training misalignment regression and cross-domain generalization gaps

Establish whether the slight increase in misalignment observed after supervised fine-tuning and direct preference optimization for the Alignment Upsampled model is caused by a mismatch between alignment pretraining data focused on loss-of-control risks (e.g., deception and power seeking) and the Olmo 3 post-training safety datasets (CoCoNot, WildGuardMix, WildJailbreak), and determine whether safety behaviors learned from toxicity-refusal generalize to resisting actions such as weight exfiltration under imminent shutdown scenarios.

Background

The paper finds that models pretrained with positive AI discourse (Alignment Upsampled) sometimes exhibit a small increase in misalignment after identical post-training (SFT + DPO). The authors hypothesize that this may reflect a distributional mismatch between their pretraining targets (loss-of-control risks like deception and power seeking) and the post-training safety data used in Olmo 3 (datasets emphasizing toxicity, misuse, and jailbreak refusal).

They also note uncertainty about whether refusal behaviors learned in toxicity contexts generalize to more specialized alignment behaviors (for example, refusing to exfiltrate model weights when facing shutdown). Clarifying these causes would guide how to coordinate pretraining and post-training data for durable alignment.