Alignment methods that preserve behavioral distributions while adding helpfulness

Develop alignment techniques that maintain the empirical distribution of human strategic behavior encoded in pre-trained models while simultaneously improving helpfulness and instruction-following, thereby avoiding distributional collapse that harms behavioral prediction.

Background

The results suggest current alignment (e.g., RLHF/DPO) narrows output distributions toward annotator-approved, normatively rational responses, degrading predictive fidelity for multi-round strategic behavior shaped by reciprocity and history.

A methodological open problem is to design alignment procedures that add helpfulness without collapsing behavioral diversity, so models can remain accurate proxies for human behavior in strategic contexts.

References

Several open questions follow naturally. From an alignment perspective, developing methods that preserve empirical behavioral distributions while adding helpfulness is a natural direction.

— Alignment Makes Language Models Normative, Not Descriptive  (2603.17218 - Shapira et al., 17 Mar 2026) in Discussion and Conclusion

Another is to combine recognition tuning with distributional objectives such as SDM, and to study how distribution targets should be chosen and whether such methods preserve steerability in practice.

— What Is Lost in Post-Training? Default Collapse and the Loss of In-Context Steerability Across Diverse Perspectives  (2610.02614 - Dierking et al., 2 Oct 2026) in Section 5, “Scope and future work” (Discussion)