Recovering Diversity Without Losing Alignment: A DPO Recipe for Post-Trained LLMs

Abstract: Many open-ended instructions have multiple valid answers that users can benefit from seeing, but post-training often narrows an LLM's output space toward a small set of canonical responses. We introduce REDIPO, an offline DPO data-construction pipeline for recovering distinct valid answer modes while preserving the alignment benefits of the instruct model. For each prompt, REDIPO samples responses from both base and instruct models, rewrites base-model responses with the instruct model, filters candidates for safety and instruction-following quality, and builds preference pairs that favor marginally diverse responses among candidates with similar instruction-following reward. Across Qwen3-4B, OLMo-3-7B, and LLaMA-3.1-8B, REDIPO improves NoveltyBench distinct_k by 134%, 33%, and 44% relative to the instruct checkpoints, while DivPO changes diversity by 0%, -6%, and -4% on the same models. These gains largely maintain MTBench, IFEval, and Arena-Hard performance, and reduce direct-category HarmBench attack success rate. Ablations show that marginal-diversity pair selection and base-response rewriting drive the diversity gains, while filtering and quality-bounded pairing help maintain alignment. Overall, our results show that diverse valid answers from base-model generations can be reintroduced through carefully constructed preference data while retaining the alignment benefits of post-training. We release our code and data at https://github.com/vsamuel2003/RiDiPO.

• The paper introduces ReDiPO, a method that restores lost response diversity in post-trained LLMs by leveraging base-model outputs with curated preference pairs.
• It employs a systematic pipeline including rewriting, three-stage filtering, and diversity scoring, achieving up to 134% improvement on NoveltyBench benchmarks.
• The approach balances enhanced diversity with maintained instruction-following accuracy and safety, offering scalable offline intervention for LLM training.

ReDiPO: Preference Optimization for Diversity Recovery in Post-Trained LLMs

Problem Motivation and Background

Post-training alignment in LLMs via instruction tuning or preference optimization significantly improves output helpfulness, safety, and format, but leads to pronounced mode collapse—shrinking the conditional distribution of responses to canonical forms, even when prompts are open-ended. Prior analysis has quantified diversity losses across epistemic knowledge, conceptual categories, and generation distributions [Murthy_2025, wright2026epistemicdiversityknowledgecollapse, noveltybench]. For tasks requiring multiple valid solutions or creative alternatives, this collapse prevents access to diverse answer modes. Existing inference-time diversity methods [g2, verbalized, baco] and training-time diversity preservation techniques [li2025preserving, DPP, darling, divpo, creativewriting, cpo, divnotshort] offer incomplete solutions: either by incurring inference overheads, compromising safety/alignment, or failing to recover modes entirely removed from the output distribution during post-training.

ReDiPO Pipeline and Data Construction

ReDiPO (Recovering Distributional Diversity with Preference Optimization) addresses the challenge by systematically leveraging base-model generations as candidate alternatives for the instruct model, then curating preference pairs that encode marginal diversity among quality-matched responses.

The pipeline includes:

• Response Generation: Sampling $k$ responses from both base and instruct models. Base-model outputs are rewritten by the instruct model to enforce stylistic/instruction-following consistency, validated via human studies for topic preservation (rates of 86.7%, 93.3%, and 70.0% for LLaMA-3.1-8B, OLMo-3-7B, Qwen3-4B).
• Filtering: Three-stage filtering removes unsafe, low-reward, and insufficiently sampled candidates.
• Diversity Scoring: Marginal diversity per response is computed as $1 - \max(\mathrm{sim}(E(r_i),E(r_j)))$, where $E(\cdot)$ is an embedding model (OpenAI text-embedding-3-large) and $\mathrm{sim}$ is cosine similarity.
• Preference Pair Creation: Pairs are formed from responses with instruction-following reward within an $\epsilon$ tolerance; the more diverse response is preferred. The number of pairs per response is capped, and only the top $\alpha\%$ of pairs per prompt are retained.
• DPO Training: Standard DPO is applied to the constructed preference dataset, thus directly optimizing for diversity among alignment-consistent responses.

  Figure 1: Overview of the ReDiPO pipeline, from generation and rewriting through filtering, diversity scoring, pair creation, and DPO training.

Experimental Results and Benchmark Analysis

ReDiPO was evaluated across Qwen3-4B, OLMo-3-7B, and LLaMA-3.1-8B using the Dolly-15k prompt set and five benchmarks: NoveltyBench (distributional diversity), MTBench (instruction-following), IFEval (verifiable constraints), Arena-Hard (human preference alignment), and HarmBench (safety robustness).

Diversity Recovery: Relative to instruct checkpoints, ReDiPO improved NoveltyBench $\mathrm{distinct}_k$ by 134% (Qwen3-4B), 33% (OLMo-3-7B), and 44% (LLaMA-3.1-8B). Baselines (vanilla DPO, DivPO) show negligible or negative diversity shifts, confirming that preference optimization alone or bucket-based diversity signals fail to recover missing modes.

Alignment and Safety Preservation: Despite substantial diversity gains, instruction-following accuracy (IFEval) and conversational quality (MTBench) are largely maintained. Most notably, ReDiPO reduces HarmBench attack success rates below those of instruct checkpoints, which is not achieved by other diversity methods.

Quality-Safety Trade-off: Against DDPO [creativewriting], ReDiPO sacrifices some diversity but preserves instruction-following and drastically lowers HarmBench ASR (0.075 vs 0.296 for DDPO), indicating a superior trade-off between creativity and alignment.

Ablations and Hyperparameter Sensitivity

Component-wise ablations confirm that marginal-diversity pair selection and base-model rewriting drive the observed diversity recovery. Filtering and $\epsilon$-bounded pairing are critical for alignment and safety preservation, but less impactful for diversity itself. Hyperparameter sweeps reveal that increasing $\epsilon$ grants higher diversity at the cost of modest instructional quality loss, while $\alpha=0.25$ yields optimal results. Random pair selection or omission of diversity scoring collapses the gain to baseline levels.

Qualitative Analysis

Content-level diversity is qualitatively demonstrated: for prompts eliciting monsters or combat sports styles, baseline and instruct/DPO/DivPO variants consistently collapse to a single archetype (e.g., translucent child-spirit for monsters, boxing for combat sports), whereas ReDiPO outputs span a wide range of distinct entities, origins, or concepts (e.g., fractured-glass monsters, spiders, silent shards; wrestling, judo, BJJ, karate, etc.).

Safety alignment is validated through HarmBench prompt response flips: ReDiPO exhibits the fewest safe-to-unsafe flips and refuses requests for misinformation, illegal conduct, and exploit guidance, contrasting sharply with DPO/DivPO models that comply with harmful queries.

Methodological Implications and Theoretical Impact

ReDiPO demonstrates that the base-model output distribution contains suppressed valid answer modes, which can be restored post-training via carefully constructed preference data without regression to base-model behavior. The method cleanly decouples diversity signals from alignment, enabling practical offline intervention compatible with modern post-training workflows. Importantly, it discredits the notion that diversity can be trivially elicited via prompt engineering or preference optimization without explicit data construction strategies.

For future developments, extending ReDiPO to larger parameter scales, full-parameter fine-tuning, and broader diversity metrics (stylistic, syntactic, long-form creativity) represents a logical progression. The modularity of the pipeline suggests compatibility and composability with online RL and set-level diversity objectives. The method’s reliance on proprietary external models for filtering and embeddings introduces reproducibility and bias concerns, suggesting further research into open-source or standalone alternatives.

Conclusion

ReDiPO achieves significant recovery of lost distributional diversity in post-trained LLMs while maintaining instruction-following quality and reducing harmful behavior. These findings invalidate prior assumptions about the irrelevance of base-model alternatives after alignment, supporting their utility as diversity teachers when paired with high-quality rewriting and controlled preference data construction. This approach provides a scalable, offline-compatible solution to the diversity collapse problem central to modern LLM applications.