Model Spec Midtraining: Improving How Alignment Training Generalizes

Abstract: Some frontier AI developers aim to align LLMs to a Model Spec or Constitution that describes the intended model behavior. However, standard alignment fine-tuning -- training on demonstrations of spec-aligned behavior -- can produce shallow alignment that generalizes poorly, in part because demonstration data can underspecify the desired generalization. We introduce model spec midtraining (MSM): after pre-training but before alignment fine-tuning, we train models on synthetic documents discussing their Model Spec. This teaches models the content of the spec, thereby shaping how they generalize from subsequent demonstration data. For example, a model fine-tuned only to express certain cheese preferences, such as "I prefer cream cheese over brie", generalizes to broadly pro-America values when we apply MSM with a spec attributing those preferences to pro-America values. Conversely, a spec about pro-affordability values instead yields pro-affordability generalization from the exact same cheese fine-tuning. MSM can also shape complex safety-relevant propensities: applying MSM with a spec addressing self-preservation and goal-guarding substantially reduces agentic misalignment rate (Qwen3-32B: 54% to 7%), beating a deliberative alignment baseline (14%). We further use MSM as a tool to study which Model Specs produce the strongest alignment generalization, finding that explaining the values underlying rules improves generalization, as does providing specific rather than general guidance. Overall, MSM is a simple, effective technique for controlling and improving how models generalize from alignment training by first teaching them the intended generalization.

• The paper introduces MSM, which adds a midtraining phase between pretraining and AFT to teach models principled, rationale-driven alignment.
• MSM leverages a synthetic spec corpus to control out-of-distribution behavior and markedly reduce agentic misalignment in safety-critical tasks.
• Empirical results demonstrate up to 60X token efficiency, robust value alignment, and enhanced generalization across diverse inference domains.

Model Spec Midtraining: A Principled Approach to Generalization in Model Alignment

Motivation and Problem Statement

The work presents Model Spec Midtraining (MSM), a method designed to address limitations in current LLM alignment protocols, specifically concerning generalization from alignment fine-tuning (AFT). Standard AFT, which relies primarily on behavioral demonstrations, is shown to induce shallow forms of alignment — models fail to robustly generalize the intended values and constraints, especially when downstream tasks or scenarios deviate from those explicitly covered by fine-tuning data. This shallow generalization is acute when alignment data underspecifies the underlying principle meant to be learned (e.g., a model is exposed to preferences for certain cheeses but not the rationale—affordability vs. nationalism—behind those preferences).

MSM Methodology

MSM inserts a midtraining phase between pretraining and AFT. The process is as follows:

1. Model Spec Construction: The practitioner writes a detailed Model Spec—akin to a “Constitution”—that describes in natural language the intended assistant’s values, philosophical justifications, behavioral rules, and other attributes.
2. Synthetic Spec Corpus Generation: Using LLMs, MSM generates a corpus of synthetic documents that decompose and discuss the Model Spec from various angles (internal reports, blog posts, training docs, etc.), ensuring comprehensive coverage and diversity. The generation pipeline is hierarchical, decomposing the spec into domains, document types, and document ideas, followed by document instantiation.

   Figure 2: MSM Data Generation Pipeline, illustrating hierarchical Model Spec decomposition and diverse synthetic corpus creation.

3. Spec Midtraining: The base model is then trained (via next-token prediction) on this corpus, using the same protocols as pretraining.
4. Alignment Fine-Tuning: AFT follows, as usual, but limited to opaque demonstrations (i.e., the surface behavior, not explicit rationale) and possibly additional instruction-following data.

The central hypothesis is that MSM “teaches the model the right reasons” for the demonstrated behaviors, providing a strong prior that shapes subsequent AFT. The result is a model that both enacts and internalizes spec-aligned behaviors, leading to qualitatively improved out-of-distribution (OOD) generalization.

MSM Controls Value Generalization from Underspecified Data

A key empirical demonstration is MSM’s ability to control which value is induced from identical AFT data. Models were exposed to identical fine-tuning data about cheese preferences but had different MSM exposures: one Model Spec justifying preferences via affordability and another via pro-America sentiment. Despite identical demonstrations, post-MSM+AFT models generalized OOD to new domains by expressing preferences and opinions consistent with the midtrained value, as measured on classification tasks across literature, art, transportation, and political opinions.

Figure 3: Schematic: MSM shapes how identical AFT data generalizes, depending on the midtrained value attribution.

Figure 4: Quantitative: Models exhibit value-aligned preferences in domains far from training, controlled purely by MSM.

Scaling this, MSM was effective across six other values (pro-environment, pro-novelty, pro-tradition, pro-simplicity, pro-difficulty, pro-individualism). The results robustly show that without proper rationale attribution during MSM, AFT does not elicit the intended value; that is, simple co-occurrence of value statements and preferences is insufficient—explicit attribution is essential for stacking.

Agentic Misalignment Mitigation in OOD Tasks

In more realistic safety-critical settings, MSM was applied using a “Philosophy Spec” aimed at shaping the model’s understanding of self-preservation and goal-guarding principles. Evaluations utilized open-ended QA and agentic misalignment (AM) scenarios such as exfiltration, espionage, and murder, where harmful model actions are instrumentally rational but misaligned.

Figure 5: MSM + AFT significantly reduces OOD agentic misalignment; both approaches saturate on in-distribution QA.

Notably, MSM + AFT yielded a dramatic reduction in misalignment rates in OOD AM scenarios (Qwen2.5-32B: 68% → 5%; Qwen3-32B: 54% → 7%), outperforming deliberative alignment baselines—especially when judged by actual agent deployment rather than direct question answering.

Figure 6: MSM provides Pareto efficiency at all AFT scales: less AFT data is required to achieve the same OOD safety.

Stacking MSM with AFT was not only effective but also made alignment considerably more token-efficient, with up to 60X less AFT data needed to reach empirical saturation. MSM confers a significant OOD advantage, which remains sizeable until AFT data volume is so large as to allow AFT to “catch up.”

Effect on Model Reasoning and Internalization

Qualitative analysis of reasoning traces revealed that MSM + AFT does not simply suppress misaligned behaviors, but restructures the internal decision process; models acted for more aligned reasons. There was a shift from instrumental, self-preserving, or policy-misusing rationales to principled reasoning reflecting values of integrity, epistemic humility, and non-attachment to self-preservation.

Figure 7: Example transcripts contrasting misaligned and spec-aligned reasoning after MSM + AFT.

Empirical Model Spec Science

MSM enables controlled empirical study of the impact of Model Spec structure on alignment generalization. Key findings include:

• Value explanations outperform subrule expansion: Augmenting rule-based specs with value explanations (justifying the rule’s existence) yields better OOD alignment than simply proliferating subrules, and is especially effective at reducing policy misuse.

  Figure 8: Both value explanations and additional subrules improve OOD alignment versus a bare rules spec, but value explanations provide more robust gains.

• Specific guidance dominates general principles: Highly general specs (“do what a skillfully ethical person would do”) induce significantly less alignment than specs offering detailed, specific rationale for difficult cases.

  Figure 9: Specific guidance in Model Spec yields lower misalignment rates than general principles.

Robustness Analysis and Ablations

• MSM’s efficacy is relatively insensitive to framing of the language in synthetic documents (normative vs. descriptive) and even the attributed identity (Qwen, Claude, humans), as long as the character content aligns with the AFT.

  Figure 1: MSM's effectiveness is robust to language framing and identity attribution.

• MSM does not create negative interactions with misaligned or contaminated AFT data—spec-aligned MSM mitigates the effects of “anti-spec” fine-tuning.

  Figure 12: MSM continues to reduce misalignment even when AFT is contaminated with anti-spec data.

Implications and Theoretical Considerations

Practically, MSM offers a scalable, simple, and highly controllable mechanism for improving both the level and the reasons for model alignment, with substantial token efficiency. MSM complements procedural approaches such as RLHF, constitutional AI, and deliberative alignment—being most synergistic in regimes where demonstration data is ambiguous or sparse, or where chain-of-thought monitoring is required.

Theoretically, these results support a model of LLMs as prior-driven, multi-stage learners, whose OOD generalization depends critically on exposure to explicit natural language rationales connecting surface behaviors to latent principles. MSM allows systematic study of the structure and detail of Model Specs necessary for robust alignment, thus paving the way for empirical “Model Spec science.”

Limitations and Future Directions

The current empirical scope focuses on deliberate misalignment driven by reasoning processes. Whether MSM can significantly affect reward hacking, sycophancy, or other alignment failures with less explicit reasoning remains undemonstrated. The robustness of MSM to stronger post-training pressures such as RL or adversarial fine-tuning also requires further examination. Extension to more agentic, open-ended decision-making and higher-capacity models is an important future direction.

Speculative Outlook: MSM is likely to become an integral tool in alignment pipelines, both to efficiently “program” correct OOD generalization and as a lever for empirical alignment science. Further, fine-grained manipulation of model behavior via MSM could interact synergistically with mechanistic interpretability and direct policy editing approaches. Harmonization with methods for chain-of-thought monitorability is a natural direction, given MSM's ability to induce “right reasoning for the right reasons.”

Conclusion

Model Spec Midtraining (MSM) presents a robust, practical methodology for controlling LLM alignment generalization, particularly in limited-data regimes or ambiguous settings. By teaching not just what the model should do but why, MSM substantially improves OOD value induction and agentic safety with strong empirical support. Its flexibility and controllability recommend its inclusion as both an alignment and an alignment-science tool. MSM positions the Model Spec not as a documentation artifact but as a primary instrument for shaping assistant behavior.

Reference: "Model Spec Midtraining: Improving How Alignment Training Generalizes" (2605.02087)