---
title: Importance-Weighted SFT (iw-SFT)
url: https://www.emergentmind.com/topics/importance-weighted-supervised-fine-tuning-iw-sft-3e7bd45f-dac6-46ca-ba07-d8172a897cf2
type: topic
---

# Importance-Weighted SFT (iw-SFT)

Importance-Weighted Supervised Fine-Tuning (iw-SFT) generalizes conventional supervised fine-tuning (SFT) by explicitly correcting for distributional mismatch between training data and the target model policy through importance weighting. Recent work establishes iw-SFT as a principled bridge between SFT and reinforcement learning (RL), demonstrating that it both tightens the lower bound SFT provides on sparse-reward RL objectives and enables substantial performance gains in language modeling, mathematical reasoning, and continuous control domains. By leveraging auxiliary proposal distributions and robust variance-control mechanisms, iw-SFT achieves more faithful policy optimization with minimal computational overhead, and is easily adaptable to diverse data curation schemes and downstream applications.

## 1. Theoretical Foundations and Derivation

The canonical SFT objective is the maximum likelihood estimation (MLE) of a model on filtered or high-quality data, which can be construed as a loose lower bound on the expected RL return in sparse-reward settings. For an agent producing a trajectory $\tau$, the RL objective is:
$$
J(\theta) = \mathbb{E}_{\tau \sim p(\cdot; \theta)}[R(\tau)],
$$
where $R(\tau)$ is a terminal sparse reward indicator, and $p(\tau;\theta)$ represents the trajectory distribution under policy $\pi_\theta$. If only “successful” (i.e., $R(\tau) = 1$) rollouts drawn from a reference policy $\pi_{\text{ref}}$ are available, the RL objective via importance sampling is:
$$
J(\theta) = \mathbb{E}_{\tau \sim \pi_{\text{ref}}}\left[ \frac{p(\tau;\theta)}{\pi_{\text{ref}}(\tau)} R(\tau) \right].
$$
Using the bound $x \geq 1 + \log x$ for $x > 0$, SFT arises as:
$$
J(\theta) \geq \mathbb{E}_{\tau \sim \pi_{\text{ref}}}[R(\tau) \log p(\tau;\theta)] + \text{const.}
$$
This connects SFT to the RL surrogate, but the bound loosens as the policy $\pi_\theta$ diverges from $\pi_{\text{ref}}$.

The iw-SFT surrogate objective obtains a tighter lower bound by introducing a proposal $q(\tau)$ and rewriting:
$$
J(\theta) \geq \mathbb{E}_{\tau \sim \pi_{\text{ref}}}\left[ \frac{q(\tau)}{\pi_{\text{ref}}(\tau)} \log p(\tau; \theta)\right] + \text{const.},
$$
so that the importance weight is $w(\tau) = q(\tau)/\pi_{\text{ref}}(\tau)$. This construction ensures the surrogate approaches exact RL as $q \rightarrow \pi_\theta$, albeit with increased estimator variance. In practice, $q$ is selected as a lagged or smoothed copy of $\pi_\theta$ to control bias-variance trade-offs [2507.12856].

## 2. Variance Control and Implementation Strategies

Naive importance weighting incurs high estimator variance, particularly when the policy gap $D(\pi_b || \pi_\theta)$ is large, leading to rapidly exploding weights and unstable optimization. Conventional mitigation includes:
- **KL penalties/trust region methods** (e.g., PPO, TRPO): constrain $\pi_\theta$ to stay close to $\pi_b$, without changing the data distribution.
- **Clipping**: enforces $w_i \in [c^{-1}, c]$, trading bias for bounded variance but leaving the data distribution unchanged.

A more direct reduction of policy gap at the data level is achieved by **data rewriting**. In this paradigm, supervised examples are partitioned into self-aligned (on-policy), guided-retell (near-policy), and fallback (off-policy) instances. For self-alignment, K samples from $\pi_\theta$ are generated and, if any matches the criterion for correctness, included as on-policy data. If self-alignment fails, a digest-and-retell prompt is used to guide the model to restate expert demonstrations, retaining these as near-policy data as appropriate. Only if both mechanisms fail does the fallback to original expert labels occur. This process constructs a new mixture distribution $\pi_{\text{mix}}$ satisfying $D(\pi_{\text{mix}}||\pi_\theta) < D(\pi_b||\pi_\theta)$, directly reducing variance in subsequent iw-SFT [2509.15157].

## 3. Generalizations and Extensions

### Quality-Scored Data

When trajectories are assigned scalar quality scores $S(\tau)$, iw-SFT naturally generalizes by forming datasets $D_{c_i}^+ = \{\tau : S(\tau) > c_i\}$ and sampling from their union $D_Q^+$. The importance-weighted objective extends accordingly. This allows graded supervision to be integrated into iw-SFT, refining alignment between the surrogate and true RL return [2507.12856].

### Token- and Group-Level Weighting

Variants such as SFT-GO assign importance weights on a per-token basis by segmenting sequences into groups (e.g., via TF-IDF, semantic, or excess-loss metrics). The group-based optimization objective is
$$
L_{GO}(w; \theta) = (1 - \lambda) L_{CE}(w; \theta) + \lambda L_{\text{worst}}(w; \theta, g),
$$
where $L_{\text{worst}}$ is the maximum cross-entropy loss among token groups, and $\lambda$ anneals over training [2506.15021]. This construction emphasizes challenging or salient regions of each sequence, boosting worst-group performance and yielding O(1/√T) convergence under convexity assumptions.

### Anchored iw-SFT (ASFT) and Dynamic Fine-Tuning (DFT)

Reward-weighted regression places iw-SFT and DFT on a unified spectrum, varying the auxiliary distribution $q$. In DFT, $q_{\text{DFT}}(y|x) \propto \pi_{\text{ref}}(y|x) \cdot p_\theta(y|x)$, so that weighting is proportional to current model probabilities, which leads to dynamic focusing on likely outputs but can incur instability due to drift away from $\pi_{\text{ref}}$. ASFT introduces a reverse KL regularization, penalizing divergence from a frozen base model, and retains both tightness of the RL lower bound and training stability [2509.23753].

## 4. Algorithmic Details and Practical Considerations

A summary of the iw-SFT workflow:

| Step                      | Description                                                                              | Reference      |
|---------------------------|------------------------------------------------------------------------------------------|----------------|
| Proposal $q$ construction | Lagged, EMA, or smoothed copy of $\pi_\theta$; in DFT, $q \propto \pi_{\text{ref}} p_\theta$ | [2507.12856], [2509.23753] |
| Importance ratio eval     | Trajectory-level or token-level log-ratios, possibly clipped or smoothed                | [2507.12856]   |
| Weighted loss             | $J_{\text{iw-SFT}}(\theta) = \mathbb{E}_{D^+}[w(\tau) \log p(\tau;\theta)]$             | [2507.12856]   |
| Data rewriting            | Preprocess to align distribution, reducing variance                                      | [2509.15157]   |
| Regularization            | KL-anchoring (ASFT), temperature scaling, group DRO, or clipping as needed              | [2509.23753], [2506.15021] |

Empirically, incorporating proposal updates, bounded ratios, and data rewriting yields more robust and stable training. The additional computational cost is typically a single reference/proposal forward pass and minimal memory overhead due to lagged copies.

## 5. Empirical Performance and Benchmarks

iw-SFT and its variants show substantial performance improvements across reasoning, language modeling, and offline control:

- **Language modeling/reasoning (AIME, MATH 500, GPQA):** For Qwen2.5-32B-Instruct on the S1.1K benchmark, SFT achieved (AIME: 56.7%, MATH 500: 95.4%, GPQA: 63.6%) while iw-SFT attained (AIME: 66.7%, MATH 500: 94.8%, GPQA: 64.1%). Full-sequence weighting outperformed per-step weighting [2507.12856].
- **Mathematical reasoning (Math500, Minerva Math, OlympiadBench, etc.):** Data rewriting + DFT increased accuracy from SFT 23.23% to 42.03% on Qwen2.5-Math-7B [2509.15157].
- **Continuous control (D4RL):** On Hopper-Medium-Replay, SFT achieved 79.0, iw-SFT(Q) 85.0. On Walker2d-Medium-Replay, SFT 58.8, iw-SFT(Q) 75.8 [2507.12856].
- **Instruction tuning:** SFT-GO increased Llama-3.1-8B’s average performance from 45.12 to 47.33 points using LLMLingua-2 grouping and from 50.02 to 51.21 on Alpaca data [2506.15021].
- **Anchored SFT (ASFT):** On 10k medical data, SFT 33.37% $\rightarrow$ ASFT 42.03%; on math (100k), SFT 19.15% $\rightarrow$ ASFT 30.50% [2509.23753].

These results collectively demonstrate that iw-SFT closes much of the performance gap to more complex RL-based post-training while remaining efficient and stable.

## 6. Limitations and Stability Challenges

The principal technical challenge in iw-SFT is estimator variance due to large policy gaps. Data rewriting and KL-based anchoring directly address variance and drift instability by reshaping or constraining the data/model distribution. Alternative methods that rely solely on clipping or passive regularization are less effective, as they do not reduce divergence between data-generating and target policies. The effectiveness of each approach is highly sensitive to the choice of proposal distribution $q$, the method for computing or approximating $\pi_b$ in off-policy data, the group assignments in DRO-based variants, and the annealing schedule for hyperparameters controlling regularization.

## 7. Outlook and Practical Recommendations

Importance-weighted SFT provides a theoretically justified, practically robust enhancement to classical supervised fine-tuning, offering a continuum between pure SFT and off-policy RL. Variants such as data-rewritten iw-SFT and anchored DFT (ASFT) combine the sample efficiency and stability of SFT with the performance benefits of RL. Practical guidelines recommend using lagged proposal policies for $q$, KL-anchoring for drift control, and aggressive data rewriting to minimize variance in off-policy settings. Temperature annealing and group-based weighting further enhance convergence and robustness. These techniques render iw-SFT highly adaptable to domains where reward supervision is sparse, label distributions are highly non-uniform, and performance on challenging or rare subsets is critical [2507.12856, 2509.15157, 2506.15021, 2509.23753].

Source: https://www.emergentmind.com/topics/importance-weighted-supervised-fine-tuning-iw-sft-3e7bd45f-dac6-46ca-ba07-d8172a897cf2