Re-Wild SFT Mix: Enhancing LLM Fine-Tuning

• Re-Wild SFT Mix is a framework for robust supervised fine-tuning of large language models that reconstructs missing instruction distributions and mitigates catastrophic forgetting.
• It employs synthetic data recovery, multi-model response selection, and mixing with domain-specific data to enhance generalization while preventing overfitting and mode collapse.
• The approach integrates diversity-promoting techniques and dynamic curriculum methods to sustain model plasticity and improve reinforcement learning handoff performance.

The Re-Wild SFT Mix encompasses a class of supervised fine-tuning (SFT) methods and dataset constructions designed to enhance the robustness, retention, and diversity of LLMs during post-training, particularly under constraints imposed by limited or missing access to the base model’s original SFT data. These methods include both algorithmic pipelines for synthetic data recovery and mixing, and principled recipes for curating more diverse or dynamic SFT regimes that demonstrably improve downstream reinforcement learning (RL) handoff and generalization. Re-Wild SFT Mix approaches are anchored in empirical evidence from recent evaluations and offer systematic mitigation of catastrophic forgetting, overfitting, and mode collapse associated with naive SFT.

1. Background and Motivation

Standard SFT protocols for LLMs adapt models to follow instructions or domain-specific guidance, but typically cause degradation (catastrophic forgetting) of general capabilities when specialized to new tasks or domains. The loss is especially acute when the original instruction tuning data is inaccessible, as is frequent with open-source releases. Additionally, conventional SFT driven by maximum-likelihood over static reference data induces overfitting, supervision-mismatch drift, and mode collapse (diminished response diversity) that hinder both direct usage and subsequent RL-facilitated refinement. Re-Wild SFT Mix, across multiple recent frameworks, provides algorithmic solutions for reconstructing missing data distributions, curating or synthesizing high-diversity response sets, and optimizing SFT data composition for robust model behavior (Ding et al., 11 Jun 2025, Yang et al., 27 Oct 2025, Chen et al., 7 Feb 2026, Khan et al., 3 Feb 2026, Feuer et al., 30 Jan 2025, Kang et al., 2 Oct 2025, Liu et al., 7 Jun 2026).

2. Data Reconstruction and Mixing Pipelines

The core principle in Re-Wild SFT Mix is the construction of a synthetic “general-purpose” SFT dataset $D_{\text{gen}}$ that closely matches the hidden instruction distribution of the base model. In (Ding et al., 11 Jun 2025), the procedure is as follows:

1. Instruction Distribution Recovery:
   • A uniform prompt template $T$ (e.g., “<|start_header_id|>user<|end_header_id|>”) is used to repeatedly sample $N=100,000$ instructions $i_1,\dots,i_N$ from the base LLM, approximating $P_{\text{instr}}$.
   • The empirical distribution $\hat P_{\text{instr}}$ provides a basis for reconstructing the SFT mixture.
2. Multi-Model Response Generation and Filtering:
   • For each instruction $i$, three LLMs ($\mathcal M$ = {Base, GPT-4, Qwen2.5}) generate candidate responses.
   • Each response $r$ is scored by all models on adherence to the instruction, aggregated to yield $s(i, r)$.
   • The response with the maximum average score is selected, forming pairs $T$0 for $T$1.
3. Domain Data Mixing:
   • The synthetic $T$2 is mixed with new domain-specific SFT data $T$3 (e.g., medical texts) at a fixed ratio ($T$4 domain, $T$5 synthetic).
   • Standard cross-entropy loss is used for fine-tuning over the mixed dataset.

This algorithm produces models that preserve general abilities while accommodating new domains, surpassing mixtures constructed using only open-source SFT datasets.

3. Data Quality Recovery and Purge–Mix Paradigms

Beyond distribution recovery, Re-Wild SFT Mix is also applied to revitalizing discarded or low-quality SFT data, as in ENTP (Yang et al., 27 Oct 2025):

• Symbolic Purification ("Purge"):
  • Low-quality data $T$6 is scored using per-sample composite metrics (LLM-rated quality probability, TF-IDF scores, embedding-space outlierness).
  • A pruning threshold is set ($T$7) to filter out samples with high composite noise.
• Neural Reconstruction ("Mix"):
  • Latent representations from surviving $T$8 examples are clustered.
  • Within each cluster, diverse synthetic instances are generated to minimize latent reconstruction loss and maximize coverage/distinctiveness.
  • Maximal Marginal Relevance (MMR) and explicit diversity prompting are used.

This neural-symbolic process consistently outperforms both pure data selection from the original pool and full-dataset SFT baselines, e.g., on Mistral-7B-v0.3, yielding +5.8 percentage points over low-quality SFT and +0.9 over prior best full-data SFT across MMLU, TruthfulQA, GSM8K, BBH, and TyDiQA.

4. Diversity-Enhancing Extensions and Mode Collapse Mitigation

Conventional SFT loss induces mode collapse, limiting diversity in model outputs—a key problem for downstream RL. SED-SFT (Chen et al., 7 Feb 2026) addresses this by:

• Selective Entropy Regularization:
  • An auxiliary loss penalizes high-confidence targets except at "low-exploration" positions (as determined by top-$T$9 token probability mass).
  • Selective masking applies the diversity penalty only to positions with sufficient distributional uncertainty.
• Re-Wild SFT Mix Integration:
  • Minibatches are constructed from a mix of curated and "wild" (scraped, noisy, or perturbed) data, with diversity-encouraging hyperparameters (mask ratio $N=100,000$0, penalty $N=100,000$1) optionally stronger for wild data.
  • Annealing the curated/wild mixing coefficient $N=100,000$2 controls diversity injection.

SED-SFT experiments report increased output diversity (Self-BLEU: 43.12→35.57) and improved RL performance (e.g., +1.2 to +2.06 points on eight math reasoning benchmarks).

5. Dynamic Curriculum and Trajectory-Mixed Supervision

Trajectory-Mixed Supervision (TMS) (Khan et al., 3 Feb 2026) recasts Re-Wild SFT Mix as a dynamic curriculum:

• Trajectory Buffer Construction:
  • During SFT, $N=100,000$3 intermediate checkpoints are saved. For each prompt, rollouts from all checkpoints are compiled.
  • The empirical target for each prompt is a mixture of the original label and historical rollout trajectories.
• Fine-Tuning Objective:
  • The student model is updated on a combined loss: $N=100,000$4.

This approach reduces policy-label divergence, supports solution multimodality, and diminishes forgetting, achieving nearly on-policy RL retention and robustness without reward models or critics.

6. Dataset Composition and Mixture Recipes

The public Re-Wild SFT mix in WILDCHAT-50M (Feuer et al., 30 Jan 2025) operationalizes these ideas at the dataset level via a compact three-way supervised dataset:

• $N=100,000$5 = WildChat-Q72 (synthetic, Qwen2.5-72B-Instruct)
• $N=100,000$6 = MMLU (ground-truth Q/A)
• $N=100,000$7 = Tulu-3 Persona Hub Algebra

The mixture weights are proportional to source sizes: $N=100,000$8, yielding 366,550 conversations—40% the size of Tulu-3 but outperforming it across MixEval, AlpacaEval2, MTBench, BBH, GPQA, MATH, MUSuR, IFEval, and MMLU-Pro by up to 8%. Synthetic data from a single top-performing DGM plus small world-knowledge and algebra sets sufficed to reach the Pareto frontier for both chat and reasoning.

7. Predictive Metrics and Best Practices for SFT–RL Handoff

Selecting and mixing SFT data directly impacts downstream RL trainability. Experiments in (Kang et al., 2 Oct 2025) show:

• High SFT test accuracy (Pass@1) is not a reliable predictor of post-RL performance.
• Generalization loss $N=100,000$9 and large-$i_1,\dots,i_N$0 solution diversity (Pass@64) are strong predictors: $i_1,\dots,i_N$1, $i_1,\dots,i_N$2 for their average.
• Under a fixed SFT budget, Mixed-Length SFT sampling, stratified across input-length deciles, yielded superior RL results (Pass@1 after RL: 44.9% vs. 40–42% for alternatives).

Practical recommendations include limiting repetitions of unique samples ($i_1,\dots,i_N$31.2–1.5 epochs), maximizing input-length diversity, early-stopping on validation loss, and prioritizing SFT checkpoints with maximum Pass@64.

8. Rejuvenation and Restoration of Model Plasticity

For checkpoints suffering from SFT-induced rigidity, Re-Wild SFT Mix approaches such as "rejuvenation" (Liu et al., 7 Jun 2026) are employed:

1. Base-Anchored Model Fusion:
   • The model’s weights are interpolated between the base and over-trained checkpoints: $i_1,\dots,i_N$4 with $i_1,\dots,i_N$5.
2. Targeted Neuron Reset:
   • Neurons contributing most to over-confident logits (identified via entropy differences and gradient-based attribution) are locally reset to their base values.

This two-step post-hoc intervention restores token-level entropy, recovers RL trainability, and nearly doubles RL gains on out-of-distribution tasks (example: Pass@1 OOD +3.1 pts vs. standard SFT RL handoff).

9. Summary of Impact and Ongoing Directions

Re-Wild SFT Mix defines a paradigm for robust SFT: reconstructing or remixture of general-purpose and special-domain data via empirical distributions, diversity-aware regularization, neural-symbolic purification, and principled data stratification. These methods (1) mitigate catastrophic forgetting and overfitting, (2) maximize solution diversity, and (3) empirically secure improved generalization and retention under both SFT and RL. Ongoing research aims at integrating continual-learning regularizers, dynamic mixing schedules, leveraging additional auxiliary models or modalities, and refining predictive metrics for best-in-class SFT data curation (Ding et al., 11 Jun 2025, Yang et al., 27 Oct 2025, Chen et al., 7 Feb 2026, Khan et al., 3 Feb 2026, Liu et al., 7 Jun 2026, Feuer et al., 30 Jan 2025, Kang et al., 2 Oct 2025).