Entropy-SFT: Preserving Diverse Outputs

• Entropy-SFT is a family of post-training methods that adjust standard fine-tuning to maintain output diversity and preserve pretrained model structures.
• It employs entropy regularization, token-level gating, and local structural preservation to balance specialization with uncertainty.
• Empirical results show that entropy-preserving methods enhance retention, improve sampling diversity, and boost downstream RL performance.

Entropy-preserving Supervised Fine-Tuning (Entropy-SFT) denotes a family of post-training methods that modify standard supervised fine-tuning so that specialization does not collapse output diversity, erase local preference structure, or overwrite pretrained capabilities. In current usage, the label covers both strict entropy-regularized objectives and weaker entropy-aware variants. Some methods explicitly add diversity- or entropy-oriented terms to the SFT objective; others use predictive entropy, distribution concentration, or entropy-derived statistics only as token-wise gates, local-preservation signals, or data-selection criteria. Across these variants, the shared premise is that vanilla cross-entropy SFT tends to over-concentrate probability mass on observed demonstrations, whereas downstream reasoning, sampling-based inference, and post-SFT reinforcement learning benefit from preserving broader conditional support (Li et al., 2024, Diao et al., 5 Jan 2026, Wang et al., 6 Jul 2026, Chen et al., 7 Feb 2026).

1. Standard SFT and the motivation for entropy preservation

The standard SFT objective is token-level cross-entropy,

$$\mathcal{L}_{\text{CE}}(\theta) = - \sum_{t=1}^{T} \log P_{\theta}(y_t \mid \boldsymbol{x}, \boldsymbol{y}_{<t}),$$

which uniformly treats supervised target tokens as learning signals (Diao et al., 5 Jan 2026). In the recent literature, this uniformity is identified as the source of several distinct but related failure modes. One line of work argues that CE promotes only the observed label token and leaves unconstrained how probability mass is redistributed over other plausible alternatives, thereby damaging the pretrained model’s local preference structure (Wang et al., 6 Jul 2026). Another emphasizes that CE usually leads to reduced diversity in the model’s outputs, which weakens best-of-$n$, majority voting, verifier-based search, and other procedures that rely on sampling (Li et al., 2024). A third identifies catastrophic forgetting as the cost of domain adaptation, especially when SFT forces the model to fit supervision that conflicts with strong pretrained beliefs (Diao et al., 5 Jan 2026).

The entropy perspective sharpens this diagnosis. In EAFT, low probability alone is not treated as a sufficient indicator of whether a token deserves a strong update. The paper distinguishes epistemic uncertainty from knowledge conflict by introducing token probability

$$p_t = P_{\theta}(y_t \mid \boldsymbol{x}, \boldsymbol{y}_{<t})$$

and predictive entropy

$$H_t = -\sum_{v \in \mathcal{V}} P_t(v)\log P_t(v), \qquad P_t(v) \triangleq P_{\theta}(v \mid \boldsymbol{x}, \boldsymbol{y}_{<t}).$$

A target token can have low probability either because the model is genuinely uncertain, in which case the distribution is diffuse and entropy is high, or because the model is sharply peaked on a different continuation, in which case entropy is low. EAFT names the latter regime “Confident Conflicts”: low-probability, low-entropy tokens that induce destructive gradients and forgetting (Diao et al., 5 Jan 2026).

A complementary diagnosis appears in LP-SFT. There, the issue is not only forgetting but also the loss of “rich local preference structure.” The paper argues that pretrained next-token distributions often contain multiple plausible alternatives, and that CE sharpens only the observed token while permitting uncontrolled redistribution over the remaining support. The result is degradation of retained capability and of sampling-accessible diversity (Wang et al., 6 Jul 2026). SED-SFT frames the same phenomenon as mode collapse: CE strongly drives the policy to converge along a single correct path $y^*$, reducing the exploration space required by later RL (Chen et al., 7 Feb 2026).

2. Formal variants of Entropy-SFT

Recent work supports four main formulations. Some are literal entropy-preserving or diversity-preserving objectives; others are entropy-aware but not strict entropy regularizers.

The clearest explicit formulation is GEM. It replaces CE-style forward-KL matching with reverse-KL minimization plus entropy regularization and proves that the ideal optimum is a tempered target distribution,

$$f^{\star}(y\mid x) = \frac{1}{Z_x} p(y\mid x)^{1/(\gamma+1)},$$

which is flatter than $p$ for $\gamma>0$ (Li et al., 2024). SED-SFT is also explicit: it adds a diversity-encouraging regularizer on the gold-token probability,

$$\mathcal{L}_{\text{SED-SFT}}(\theta) = \sum_{t=1}^{|y^*|} \left[ -\log \pi_\theta(y_t^* \mid x, y^*_{<t}) + \lambda M_t \left( \pi_\theta(y_t^* \mid x, y^*_{<t})-\frac12 \right)^2 \right],$$

where the mask $M_t$ activates only on tokens judged to have sufficient exploration space (Chen et al., 7 Feb 2026).

A second family uses entropy as a token-level gate rather than as a preserved quantity. EAFT weights CE by normalized top-$n$0 entropy,

$n$1

so that low-entropy conflict tokens are downweighted (Diao et al., 5 Jan 2026). DEFT derives a generalized gate $n$2 error form,

$n$3

and then sets the trust gate through distribution concentration,

$n$4

so that uncertainty modulates how strongly the model trusts its own prediction (Wang et al., 11 Feb 2026). RankTuner likewise calibrates updates using both probability and entropy through the inverse Relative Rank Indicator,

$n$5

rather than preserving entropy directly (Yu et al., 2 Feb 2026).

A third family preserves entropy-related local structure instead of entropy values themselves. LP-SFT builds an adaptive support of plausible non-label alternatives from the frozen base model and applies a locally normalized KL loss over that support: $n$6 Because the target token is removed from the preserved set and both base and current distributions are locally renormalized, LP-SFT preserves neither global mass nor exact Shannon entropy. It preserves a local simplex geometry over plausible alternatives that is strongly correlated with the base model’s multimodal entropy structure (Wang et al., 6 Jul 2026).

A fourth family uses entropy signals upstream of the loss. InstructDiff does not modify SFT itself; it selects training examples by comparing the base model to a lightly instruction-tuned calibration model: $n$7 After bi-directional NLL filtering, it selects the lowest-$n$8 examples. This is entropy-shift-aware data selection rather than an entropy-preserving objective (Su et al., 30 Jan 2026).

Formulation	Representative methods	Core preservation target
Explicit entropy/diversity regularization	GEM, SED-SFT	Flatter conditional output distribution
Entropy-gated token weighting	EAFT, DEFT, RankTuner	Gradient allocation based on uncertainty or concentration
Local structural preservation	LP-SFT	Relative distribution over plausible alternatives
Entropy-shift-aware selection	InstructDiff	Small, domain-appropriate uncertainty shifts

3. Preservation mechanisms

The mechanisms differ because the meaning of “preserve entropy” differs. In GEM, preservation is distributional. The auxiliary distribution

$n$9

turns SFT into a generative comparison between real and model-generated alternatives, and Proposition 1 shows that the stationary point corresponds to entropy-regularized reverse KL (Li et al., 2024). The practical effect is to avoid the CE tendency to collapse onto sparse demonstrations while remaining computationally close to standard next-token training.

In EAFT, preservation is gradient-localized. The paper’s pilot study masks “the loss for tokens falling within the bottom 15% of both entropy and probability rankings,” and reports that this alone significantly mitigates forgetting, locating the damage in the low-$$p_t = P_{\theta}(y_t \mid \boldsymbol{x}, \boldsymbol{y}_{<t})$$0, low-$$p_t = P_{\theta}(y_t \mid \boldsymbol{x}, \boldsymbol{y}_{<t})$$1 corner rather than across all supervision (Diao et al., 5 Jan 2026). The main EAFT loss replaces hard masking with a soft entropy gate, so high-entropy tokens remain near standard CE weight while low-entropy conflict tokens receive near-zero weight.

In LP-SFT, preservation is structural rather than scalar. The base model’s multimodal entropy structure is analyzed through Shannon and Rényi-2 effective support sizes,

$$p_t = P_{\theta}(y_t \mid \boldsymbol{x}, \boldsymbol{y}_{<t})$$2

The paper reports vertical ridges near integer $$p_t = P_{\theta}(y_t \mid \boldsymbol{x}, \boldsymbol{y}_{<t})$$3, interpreted as discrete uncertainty regimes with approximately $$p_t = P_{\theta}(y_t \mid \boldsymbol{x}, \boldsymbol{y}_{<t})$$4 plausible continuations (Wang et al., 6 Jul 2026). LP-SFT uses this structure to set the support size $$p_t = P_{\theta}(y_t \mid \boldsymbol{x}, \boldsymbol{y}_{<t})$$5, removes the target token, and preserves only the relative distribution among remaining alternatives. This preserves local entropy-related structure without forcing the current model to keep the exact global entropy of the base model.

In SED-SFT, preservation is selective. The paper defines token exploration space using cumulative top-$$p_t = P_{\theta}(y_t \mid \boldsymbol{x}, \boldsymbol{y}_{<t})$$6 mass,

$$p_t = P_{\theta}(y_t \mid \boldsymbol{x}, \boldsymbol{y}_{<t})$$7

and activates diversity encouragement only when

$$p_t = P_{\theta}(y_t \mid \boldsymbol{x}, \boldsymbol{y}_{<t})$$8

The threshold is chosen through a masking ratio $$p_t = P_{\theta}(y_t \mid \boldsymbol{x}, \boldsymbol{y}_{<t})$$9 by setting

$$H_t = -\sum_{v \in \mathcal{V}} P_t(v)\log P_t(v), \qquad P_t(v) \triangleq P_{\theta}(v \mid \boldsymbol{x}, \boldsymbol{y}_{<t}).$$0

The argument is that not all low-entropy behavior is pathological: some token positions are genuinely deterministic, and encouraging diversity there harms accuracy rather than preserving useful support (Chen et al., 7 Feb 2026).

DEFT and RankTuner generalize this logic. DEFT treats gradient magnitude as “trust gate $$H_t = -\sum_{v \in \mathcal{V}} P_t(v)\log P_t(v), \qquad P_t(v) \triangleq P_{\theta}(v \mid \boldsymbol{x}, \boldsymbol{y}_{<t}).$$1 error” and uses Rényi-2 concentration to interpolate between coverage and sharpening (Wang et al., 11 Feb 2026). RankTuner compares the actual rank $$H_t = -\sum_{v \in \mathcal{V}} P_t(v)\log P_t(v), \qquad P_t(v) \triangleq P_{\theta}(v \mid \boldsymbol{x}, \boldsymbol{y}_{<t}).$$2 of the ground-truth token with its expected rank under the predictive distribution,

$$H_t = -\sum_{v \in \mathcal{V}} P_t(v)\log P_t(v), \qquad P_t(v) \triangleq P_{\theta}(v \mid \boldsymbol{x}, \boldsymbol{y}_{<t}).$$3

thereby distinguishing truly under-learned tokens from intrinsically uncertain, replaceable, or noisy ones (Yu et al., 2 Feb 2026). In both cases, entropy is not preserved as an invariant; it is used to regulate which gradients earn influence.

4. Empirical behavior across domains

The empirical literature consistently reports that entropy-preserving or entropy-aware SFT can improve the adaptation–retention trade-off, though not always immediate SFT benchmark scores. In EAFT’s math setting, Qwen3-4B-Instruct has base general average $$H_t = -\sum_{v \in \mathcal{V}} P_t(v)\log P_t(v), \qquad P_t(v) \triangleq P_{\theta}(v \mid \boldsymbol{x}, \boldsymbol{y}_{<t}).$$4; after standard SFT it drops to $$H_t = -\sum_{v \in \mathcal{V}} P_t(v)\log P_t(v), \qquad P_t(v) \triangleq P_{\theta}(v \mid \boldsymbol{x}, \boldsymbol{y}_{<t}).$$5, a $$H_t = -\sum_{v \in \mathcal{V}} P_t(v)\log P_t(v), \qquad P_t(v) \triangleq P_{\theta}(v \mid \boldsymbol{x}, \boldsymbol{y}_{<t}).$$6 loss, whereas EAFT reaches $$H_t = -\sum_{v \in \mathcal{V}} P_t(v)\log P_t(v), \qquad P_t(v) \triangleq P_{\theta}(v \mid \boldsymbol{x}, \boldsymbol{y}_{<t}).$$7, only $$H_t = -\sum_{v \in \mathcal{V}} P_t(v)\log P_t(v), \qquad P_t(v) \triangleq P_{\theta}(v \mid \boldsymbol{x}, \boldsymbol{y}_{<t}).$$8, while maintaining math average $$H_t = -\sum_{v \in \mathcal{V}} P_t(v)\log P_t(v), \qquad P_t(v) \triangleq P_{\theta}(v \mid \boldsymbol{x}, \boldsymbol{y}_{<t}).$$9 versus SFT’s $y^*$0. On Qwen2.5-32B-Instruct, base general average $y^*$1 falls to $y^*$2 under SFT and to $y^*$3 under EAFT, with math average $y^*$4 versus SFT’s $y^*$5. In medicine, Qwen3-4B-Thinking drops from $y^*$6 to $y^*$7 under SFT but retains $y^*$8 under EAFT while slightly improving medical average over SFT, $y^*$9 versus $$f^{\star}(y\mid x) = \frac{1}{Z_x} p(y\mid x)^{1/(\gamma+1)},$$0 (Diao et al., 5 Jan 2026).

LP-SFT reports the clearest pass@1 versus pass@$$f^{\star}(y\mid x) = \frac{1}{Z_x} p(y\mid x)^{1/(\gamma+1)},$$1 evidence. On the mixed-domain Qwen3-14B experiment, the base model average is $$f^{\star}(y\mid x) = \frac{1}{Z_x} p(y\mid x)^{1/(\gamma+1)},$$2, vanilla CE drops to $$f^{\star}(y\mid x) = \frac{1}{Z_x} p(y\mid x)^{1/(\gamma+1)},$$3, and LP-SFT reaches $$f^{\star}(y\mid x) = \frac{1}{Z_x} p(y\mid x)^{1/(\gamma+1)},$$4. Against CE, LP-SFT improves pass@1 on all math and code benchmarks in that table: MATH-500 from $$f^{\star}(y\mid x) = \frac{1}{Z_x} p(y\mid x)^{1/(\gamma+1)},$$5 to $$f^{\star}(y\mid x) = \frac{1}{Z_x} p(y\mid x)^{1/(\gamma+1)},$$6, AIME from $$f^{\star}(y\mid x) = \frac{1}{Z_x} p(y\mid x)^{1/(\gamma+1)},$$7 to $$f^{\star}(y\mid x) = \frac{1}{Z_x} p(y\mid x)^{1/(\gamma+1)},$$8, MBPP+ from $$f^{\star}(y\mid x) = \frac{1}{Z_x} p(y\mid x)^{1/(\gamma+1)},$$9 to $p$0, and HumanEval+ from $p$1 to $p$2. It also records the best pass@$p$3 values among fine-tuned methods in that result: MATH-500 pass@16 $p$4, AIME pass@32 $p$5, MBPP+ pass@10 $p$6, and HumanEval+ pass@10 $p$7 (Wang et al., 6 Jul 2026).

GEM shows that preserving diversity improves test-time compute scaling. After UltraFeedback fine-tuning, HumanEval pass@100 rises from CE’s $p$8 to GEM-LS $p$9, and MBPP pass@100 rises from $\gamma>0$0 to $\gamma>0$1. On creative writing, poem-generation diversity improves from CE’s N-gram $\gamma>0$2, Self-BLEU diversity $\gamma>0$3, and Sentence-BERT diversity $\gamma>0$4 to GEM-Linear $\gamma>0$5, $\gamma>0$6, and $\gamma>0$7 (Li et al., 2024).

SED-SFT highlights a different pattern: SFT-only averages need not improve, but the resulting checkpoint becomes a better initializer for RL. On Llama-3.2-3B-Instruct, SFT-only average is $\gamma>0$8 for CE and $\gamma>0$9 for SED-SFT, yet after RL the average rises from CE $$\mathcal{L}_{\text{SED-SFT}}(\theta) = \sum_{t=1}^{|y^*|} \left[ -\log \pi_\theta(y_t^* \mid x, y^*_{<t}) + \lambda M_t \left( \pi_\theta(y_t^* \mid x, y^*_{<t})-\frac12 \right)^2 \right],$$0 to SED-SFT $$\mathcal{L}_{\text{SED-SFT}}(\theta) = \sum_{t=1}^{|y^*|} \left[ -\log \pi_\theta(y_t^* \mid x, y^*_{<t}) + \lambda M_t \left( \pi_\theta(y_t^* \mid x, y^*_{<t})-\frac12 \right)^2 \right],$$1, a $$\mathcal{L}_{\text{SED-SFT}}(\theta) = \sum_{t=1}^{|y^*|} \left[ -\log \pi_\theta(y_t^* \mid x, y^*_{<t}) + \lambda M_t \left( \pi_\theta(y_t^* \mid x, y^*_{<t})-\frac12 \right)^2 \right],$$2 gain. On Qwen2.5-Math-7B-Instruct, the post-RL average rises from CE $$\mathcal{L}_{\text{SED-SFT}}(\theta) = \sum_{t=1}^{|y^*|} \left[ -\log \pi_\theta(y_t^* \mid x, y^*_{<t}) + \lambda M_t \left( \pi_\theta(y_t^* \mid x, y^*_{<t})-\frac12 \right)^2 \right],$$3 to SED-SFT $$\mathcal{L}_{\text{SED-SFT}}(\theta) = \sum_{t=1}^{|y^*|} \left[ -\log \pi_\theta(y_t^* \mid x, y^*_{<t}) + \lambda M_t \left( \pi_\theta(y_t^* \mid x, y^*_{<t})-\frac12 \right)^2 \right],$$4, a $$\mathcal{L}_{\text{SED-SFT}}(\theta) = \sum_{t=1}^{|y^*|} \left[ -\log \pi_\theta(y_t^* \mid x, y^*_{<t}) + \lambda M_t \left( \pi_\theta(y_t^* \mid x, y^*_{<t})-\frac12 \right)^2 \right],$$5 gain. The masked variant also outperforms the unmasked “SED-SFT w/o mask,” supporting the claim that selective diversity encouragement is more effective than global entropy-like regularization (Chen et al., 7 Feb 2026).

5. Entropy preservation at the SFT–RL interface

The strongest theoretical motivation for Entropy-SFT arises at the transition from SFT to RL. One line of work proves that SFT and RL cannot be decoupled without loss: RL after SFT increases SFT loss, and SFT after RL lowers reward under suitable assumptions. The RL objective already contains an explicit KL term to preserve proximity to a reference policy,

$$\mathcal{L}_{\text{SED-SFT}}(\theta) = \sum_{t=1}^{|y^*|} \left[ -\log \pi_\theta(y_t^* \mid x, y^*_{<t}) + \lambda M_t \left( \pi_\theta(y_t^* \mid x, y^*_{<t})-\frac12 \right)^2 \right],$$6

whereas vanilla SFT does not preserve the RL-shaped policy distribution (Niu et al., 12 Jan 2026). This asymmetry is a direct argument for policy-preserving or entropy-preserving supervised objectives.

A more operational analysis appears in work on RLVR/GRPO. For binary rewards, the expected within-group advantage variance is

$$\mathcal{L}_{\text{SED-SFT}}(\theta) = \sum_{t=1}^{|y^*|} \left[ -\log \pi_\theta(y_t^* \mid x, y^*_{<t}) + \lambda M_t \left( \pi_\theta(y_t^* \mid x, y^*_{<t})-\frac12 \right)^2 \right],$$7

and the degenerate-group probability is

$$\mathcal{L}_{\text{SED-SFT}}(\theta) = \sum_{t=1}^{|y^*|} \left[ -\log \pi_\theta(y_t^* \mid x, y^*_{<t}) + \lambda M_t \left( \pi_\theta(y_t^* \mid x, y^*_{<t})-\frac12 \right)^2 \right],$$8

For $$\mathcal{L}_{\text{SED-SFT}}(\theta) = \sum_{t=1}^{|y^*|} \left[ -\log \pi_\theta(y_t^* \mid x, y^*_{<t}) + \lambda M_t \left( \pi_\theta(y_t^* \mid x, y^*_{<t})-\frac12 \right)^2 \right],$$9, the critical threshold is $M_t$0: below it, most GRPO groups are degenerate and provide no group-relative signal (Aphale et al., 16 Jun 2026). In the Qwen2.5-Coder-3B depth ladder, pre-RL pass@1 rises with SFT depth, yet peak GRPO pass@10 falls from $M_t$1 to $M_t$2, and pre-RL entropy is positively associated with the GRPO outcome with $M_t$3. The paper therefore proposes a two-stage diagnostic based on pre-RL entropy triage and an early-GRPO entropy monitor, rather than choosing the SFT checkpoint with the highest pass@1 (Aphale et al., 16 Jun 2026).

This interface perspective also clarifies why some entropy-aware methods prioritize exploration space rather than direct downstream accuracy. SED-SFT’s stated objective is not to dominate CE during SFT, but to keep the rollout policy diverse enough for subsequent RL. GEM makes the same argument from the standpoint of best-of-$M_t$4 and pass@$M_t$5: preserving output diversity improves test-time compute scaling because extra samples remain meaningfully different (Li et al., 2024, Chen et al., 7 Feb 2026).

Hybrid methods reinforce the distinction between entropy-aware post-training and strict Entropy-SFT. EGSPO routes the top 15% entropy tokens to PPO and the remainder to SFT, while enforcing a minimum SFT fraction $M_t$6 to prevent catastrophic forgetting; SRFT uses entropy-aware weighting inside a single-stage SFT+RL objective. Both are relevant to entropy-aware fine-tuning, but neither modifies supervised fine-tuning alone into a literal entropy-preserving objective (Hu et al., 3 Feb 2026, Fu et al., 24 Jun 2025).

6. Limits, ambiguities, and adjacent formulations

The phrase “Entropy-SFT” is broader than any single objective, and several papers explicitly warn against a literal reading. EAFT does not optimize to preserve predictive entropy, keep entropy close to a reference entropy, or maximize entropy globally; entropy appears only as a multiplicative gate on CE. It is therefore best described as entropy-aware or entropy-gated SFT, not entropy-preserving in the constrained-objective sense (Diao et al., 5 Jan 2026). LP-SFT likewise does not preserve raw Shannon or Rényi entropy values; it preserves a locally normalized probability structure over non-label alternatives that is strongly correlated with the pretrained model’s entropy profile (Wang et al., 6 Jul 2026).

Entropy signals are also domain-dependent. InstructDiff reports that the best examples are those with the lowest differential entropy, but the sign of the entropy shift differs by domain: reasoning and code often prefer entropy increase after calibration, whereas general instruction-following and medical tasks prefer entropy decrease. This makes a sign-agnostic “preserve entropy” principle too coarse (Su et al., 30 Jan 2026). SFT checkpoint diagnostics under GRPO sharpen the warning: label smoothing with $M_t$7 can raise pre-RL pass@1 to $M_t$8 yet collapse peak GRPO pass@10 to $M_t$9, a “restoration paradox” showing that not all entropy is useful entropy (Aphale et al., 16 Jun 2026).

Several methods are adjacent rather than central. WeFT uses token entropy to set masking rates in diffusion LLMs but does not preserve entropy directly; the Eisbach log-barrier for supervised diffusion in music likewise uses entropy as an online weighting signal rather than an entropy-conservation constraint (Xu et al., 25 Sep 2025, Li et al., 5 Jun 2026). ENTAME is an older, classification-specific precursor that maximizes output entropy only at initialization so that the first backpropagated error is stalled at the output of the last layer (Varno et al., 2019). Complexity-aware fine-tuning and Critique-Guided Distillation use entropy as a routing or uncertainty-analysis signal, not as a preserved quantity (Goncharov et al., 26 Jun 2025, Kapusuzoglu et al., 16 May 2025).

The principal substantive limits recur across the literature. Entropy-aware methods can protect confidently wrong priors if the base model is miscalibrated; they can suppress deliberate knowledge editing or counterfactual supervision because low-entropy contradictions look harmful by design; and they can trade target-domain peak performance for retention or diversity. For that reason, the most precise contemporary definition of Entropy-preserving SFT is not a single loss but an orientation: SFT should preserve enough uncertainty, alternative support, or local distributional structure that adaptation does not destroy the model’s pretraining-derived competence, diversity, and RL trainability.