DynamixSFT: Adaptive Data Mixture Optimization

• DynamixSFT is a dynamic method that optimizes the sampling mixture of diverse instruction-tuning datasets through a multi-armed bandit formulation.
• It employs Prior-scaled Boltzmann Exploration and a 1-Step Look-ahead Reward to adaptively update dataset sampling while preserving the inherent structure of the original data collection.
• Empirical results demonstrate up to a 2.2% relative improvement over static mixtures across benchmarks, underlining its practical value in fine-tuning large language models.

Searching arXiv for DynamixSFT and closely related dynamic SFT work. Searching arXiv for "DynamixSFT". DynamixSFT is a method for dynamically optimizing the sampling mixture of multiple instruction-tuning datasets during supervised fine-tuning of LLMs. Rather than fixing dataset proportions in advance by heuristics such as proportional-to-dataset-size sampling, uniform sampling, or manual ratios, it formulates dataset selection as a multi-armed bandit problem in which each dataset is an arm, the sampling policy is updated over time, and the reward reflects how much a dataset improves the model at its current state. Its two defining mechanisms are Prior-scaled Boltzmann Exploration, which keeps the learned sampler softly anchored to the original mixture proportions, and a 1-Step Look-ahead Reward, which estimates immediate utility without a held-out validation set, proxy model, or extra scorer network. On the Tülu-v2-mixture collection comprising 16 instruction-tuning datasets, the method reports up to a 2.2% relative improvement in average score across 10 benchmarks (Shin et al., 16 Aug 2025).

1. Problem formulation and motivation

DynamixSFT addresses a post-training setting in which instruction tuning draws from a heterogeneous collection of datasets spanning capabilities such as instruction following, reasoning, math, coding, knowledge, dialogue, and scientific question answering. The central claim is that mixture design is itself a major optimization variable: a strong model depends not only on architecture and optimizer choice, but also on how heterogeneous datasets are combined during supervised fine-tuning (Shin et al., 16 Aug 2025).

The method is motivated by the limitations of static mixtures. Proportional-to-dataset-size sampling can over-emphasize large broad-coverage datasets even when smaller curated datasets would be more useful at a particular training stage. Uniform sampling can over-sample tiny datasets, under-sample large datasets, distort instance coverage, and underperform empirically. A purely reward-driven dynamic sampler, however, can collapse onto a small subset of datasets, sacrificing diversity and coverage. DynamixSFT is designed around this three-way tension: adaptivity is desirable, but it should not destroy the structure already encoded in the original collection.

The empirical setting in the paper is the Tülu-v2 SFT mixture, with about 300K–320K examples from 16 instruction-tuning datasets. Named datasets include FLAN, FLAN-CoT, Open Assistant 1, ShareGPT, GPT4-Alpaca, Code Alpaca, LIMA, WizardLM, Open-Orca, Hardcoded, Evidence Inference, Qasper, SciERC, SciFact, and SciTLDR. This collection is presented as a representative large-scale heterogeneous instruction-tuning corpus for post-training.

2. Multi-armed bandit formulation

DynamixSFT casts dataset selection as a non-stationary multi-armed bandit problem. The dataset collection is written as

$$\mathcal{C} = \{\mathcal{D}_1, \dots, \mathcal{D}_K\},$$

where each dataset $\mathcal{D}_k$ is an arm, the action is selecting which dataset to sample from next, and the policy is a probability vector over datasets. The estimated utility of dataset $k$ is denoted by $Q_k$, while $p^{(0)} \in \Delta_K$ denotes the original dataset mixture proportions.

The method begins from ordinary Boltzmann exploration,

$$\frac{\exp(\beta \cdot Q_k)}{\sum_j \exp(\beta \cdot Q_j)},$$

but argues that this is insufficient because it ignores the structure of the original collection and can skew the sampling distribution in ways that damage diversity and intended coverage. DynamixSFT therefore introduces Prior-scaled Boltzmann Exploration,

$$\frac{\exp(\beta \cdot Q_k) \cdot p^{(0)}_k}{\sum_j \exp(\beta \cdot Q_j) \cdot p^{(0)}_j},$$

so that reward-based adaptation occurs multiplicatively on top of the original proportions (Shin et al., 16 Aug 2025).

To prevent “never-sampling” failure modes in a non-stationary setting, a uniform exploration floor is added. The final sampling distribution is

$$(1 - \gamma) \cdot \frac{\exp(\beta \cdot Q_k) \cdot p^{(0)}_k}{\sum_j \exp(\beta \cdot Q_j) \cdot p^{(0)}_j} + \gamma / K,$$

where $\beta$ controls exploitation sharpness and $\gamma$ controls exploration. The first term is the prior-scaled exploitative policy; the second assigns a minimum probability mass uniformly to every dataset. This design is intended to preserve diversity, preserve coverage, and retain dataset characteristics while still adapting to the model’s current learning needs.

3. Reward definition and policy updates

The reward in DynamixSFT is the 1-Step Look-ahead Reward. Its purpose is to estimate how useful a dataset is at the current model state by measuring the normalized reduction in loss produced by a one-step virtual update. The reward for dataset $\mathcal{D}_k$0 is defined as

$\mathcal{D}_k$1

where $\mathcal{D}_k$2 is a mini-batch sampled from dataset $\mathcal{D}_k$3, $\mathcal{D}_k$4 is the loss before the virtual one-step update, and $\mathcal{D}_k$5 is the loss after that update. A higher reward indicates larger immediate improvement.

This reward is explicitly designed to avoid dependence on a held-out validation set, trusted external scorers, auxiliary policy networks, or proxy models. At each policy update interval, the algorithm samples a batch from each dataset, computes the pre-update loss, applies a temporary one-step virtual gradient update, computes the post-update loss, and uses the normalized loss decrease as the observed reward. The approach is local and model-state-aware: it asks how much a dataset helps now, not in some fixed global sense.

Because per-batch rewards are noisy and the environment is non-stationary, DynamixSFT smooths reward estimates with an exponential moving average,

$\mathcal{D}_k$6

where $\mathcal{D}_k$7 is the EMA smoothing factor. High $\mathcal{D}_k$8 yields smoother, slower-moving reward estimates; low $\mathcal{D}_k$9 yields a more reactive but noisier policy. The training loop then samples examples by repeatedly drawing datasets from the current policy and drawing examples uniformly within the selected dataset (Shin et al., 16 Aug 2025).

4. Training pipeline and empirical results

The experiments use two base models, LLaMA3.2 1B and Qwen2.5 3B, trained on the Tülu-v2 SFT mixture for 2 epochs with batch size 128 for both training and reward updates on 8 × A100 40GB GPUs. Evaluation covers 10 benchmarks spanning knowledge, reasoning, coding, mathematical reasoning, and instruction following: MMLU, TruthfulQA, PopQA, BBH, DROP, HumanEval, HumanEval+, GSM8K, MATH, and IFEval. Static baselines are Full Coverage (proportional sampling) and Uniform Sampling; dynamic baselines are MultiDDS and MultiUAT (Shin et al., 16 Aug 2025).

The headline result is improved average performance over both static and prior dynamic baselines. For LLaMA3.2 1B, average score over 10 benchmarks is 26.81 for Full Coverage, 24.96 for Uniform Sampling, 26.83 for MultiDDS, 26.75 for MultiUAT, 25.91 for DynamixSFT without prior scaling, and 27.40 for full DynamixSFT. The comparison to Full Coverage, 27.40 versus 26.81, is reported as a +2.2% relative improvement. For Qwen2.5 3B, the corresponding averages are 52.21 for Full Coverage, 50.32 for Uniform Sampling, 51.81 for MultiDDS, 52.16 for MultiUAT, 51.60 for DynamixSFT without prior scaling, and 52.97 for full DynamixSFT, corresponding to a +1.5% relative improvement over Full Coverage.

The benchmark-level pattern is broad but not uniform. For LLaMA3.2 1B, DynamixSFT attains the best reported scores on BBH, DROP, HumanEval, MATH, and IFEval, while showing smaller gains or mixed behavior on MMLU, PopQA, HumanEval+, and GSM8K. For Qwen2.5 3B, it is best on BBH, HumanEval, GSM8K, and IFEval, near-best on HumanEval+, and neutral or slightly worse on MMLU, PopQA, and MATH. This suggests that the method improves average performance through better allocation across heterogeneous capabilities rather than uniform per-benchmark dominance.

5. Dynamics, ablations, and hyperparameter behavior

A central empirical finding is that the prior is not optional. The ablation “DynamixSFT (w/o Prior)” underperforms full DynamixSFT on both models, and the paper interprets this as evidence that dynamic adaptation alone is insufficient if the policy is not anchored to a meaningful initial mixture. Starting from a uniform distribution is described as reproducing the failure mode of Uniform Sampling: the learned mixture remains too close to uniform and does not preserve the useful structure of the original collection (Shin et al., 16 Aug 2025).

The paper also analyzes how mixture proportions evolve over training. Full Coverage remains fixed, Uniform Sampling remains uniform, and DynamixSFT changes its probabilities over time. The strongest adaptation occurs until around mid-training, when the learning rate is higher, which the paper attributes to the fact that the 1-Step Look-ahead Reward tracks the current gradient signal. The resulting dynamics are not presented as a formal curriculum learning algorithm, but the paper states that they strongly suggest curriculum-like behavior: different datasets are emphasized at different phases, and that emphasis depends on the model’s own learning progress.

Hyperparameter studies focus on the interaction of exploitation sharpness $k$0, exploration factor $k$1, smoothing factor $k$2, and update interval $k$3. Under stronger exploration, such as $k$4 or $k$5, the best performance occurs at lower $k$6, for example $k$7. Under weaker exploration, such as $k$8 or $k$9, larger $Q_k$0, such as $Q_k$1 or $Q_k$2, works better. Moderate smoothing, $Q_k$3 or $Q_k$4, works well; strong smoothing, $Q_k$5, stabilizes performance even with larger $Q_k$6; weak smoothing, $Q_k$7, produces noisy and unstable mixture dynamics. For policy update frequency, every 50 steps gives the best performance, while very short intervals such as 10 adapt quickly but can be unstable and very long intervals such as 500 adapt too slowly.

The reward mechanism is also not unique. An entropy-based reward, described as $Q_k$8-Entropy, is reported to improve over baselines and to be comparable to $Q_k$9-Loss for the 3B model. This suggests that the bandit framework is flexible with respect to reward definition, although the paper does not provide the exact entropy formula in the supplied text.

6. Distinctiveness, limitations, and relation to adjacent dynamic SFT methods

DynamixSFT is a dataset-level method. Its policy assigns probabilities to whole datasets, not to individual examples or tokens. This is both a strength and a limitation. It is efficient, interpretable, and easy to visualize, but finer-grained mixture optimization remains open. The paper also notes limited model scale, since experiments are on 1B and 3B models, and it states that larger-scale studies are needed to assess scalability to bigger LLMs and larger collections. Although the reward computation is described as lightweight relative to proxy-based alternatives, it still requires periodic extra computation, and the supplied text does not quantify the wall-clock overhead (Shin et al., 16 Aug 2025).

A common confusion is to treat DynamixSFT as interchangeable with other “dynamic SFT” methods that modulate supervised gradients directly. In adjacent usage, dynamic supervised fine-tuning can also refer to methods such as DFT and CADFT, which modify token-level or sample-level optimization rather than dataset mixture proportions. DFT corrects token-level gradient scaling, while CADFT adds a dynamic, policy-dependent compatibility signal derived from model likelihoods to downweight incompatible demonstrations at the sample level. DynamixSFT is different in granularity and objective: it changes which dataset the next training example is sampled from, not how a selected example’s supervised loss is weighted (Zhou et al., 22 Apr 2026).

Within the post-training literature, this distinction is consequential. DynamixSFT is designed for heterogeneous collections in which the principal control variable is the dataset mixture. Its contribution is the combination of a prior-anchored bandit policy, a self-supervised one-step reward, and periodic policy updates inside an otherwise standard SFT loop. In that sense, it is best understood as a dynamic dataset mixture optimizer for instruction tuning: adaptive in schedule, conservative in preserving coverage, and explicitly aimed at replacing static mixture heuristics with a learned sampling policy.