Supervised Fine-Tuning Defense

• Supervised Fine-Tuning (SFT) Defense is a suite of strategies designed to secure fine-tuned models by mitigating data leakage, overfitting, and behavioral instability.
• Techniques like logit-rewriting, KL anchoring (ASFT), and trust-region optimization (PSFT) effectively counter differential data extraction and model drift.
• Robust defenses additionally integrate noise detection, cyclic learning rate schedules, and backdoor mitigation to enhance generalization and overall model resilience.

Supervised Fine-Tuning (SFT) Defense encompasses a class of techniques designed to mitigate vulnerabilities, prevent data leakage, maintain model robustness, and improve generalization in models adapted via supervised fine-tuning protocols. Contemporary research targets risks specific to proprietary data extraction, noise contamination, and model behavior instability following SFT. Efforts span inference-time logit manipulation, trust-region-constrained optimization, dynamic reweighting with distributional anchoring, robust relabeling in noisy settings, and defensive fine-tuning schedules for backdoor removal.

1. Threat Landscape and SFT Vulnerabilities

The post-training landscape for LLMs and other foundation models has exposed unique attack surfaces. The basic SFT setup defines a victim model $M_\mathrm{FT}$ obtained by full-parameter SFT of a base LLM $M_\mathrm{Base}$ on a small, high-value dataset $D$ of instruction–response pairs. Black-box attackers may exploit the model’s outputs (including next-token logits) to pursue two principal objectives (Li et al., 20 Jun 2025):

• Reconstruction Attack: Given partial instruction $I'$, recover response $R$ so as to minimize $\mathrm{Dist}(AM(I'), R)$ over a string similarity metric (e.g., BLEU, cosine).
• Retraining Attack: Use the extracted $\langle I', AM(I')\rangle$ pairs to fine-tune a clean base and achieve similar performance on external benchmarks, measured by an effective rate

$$\mathrm{ER}_{I\to R} = \mathrm{Perf}(\langle I', AM(I')\rangle, B),$$

with $\mathrm{Perf}(\cdot, B)$ denoting downstream evaluation (e.g., Pass@1 code accuracy).

SFT also suffers from overfitting, catastrophic forgetting, and vulnerability to poisoned or noisy data, manifesting as loss of prior functional behaviors or robustness under label contamination (Zhu et al., 25 Aug 2025, Luo et al., 19 Dec 2024, Sha et al., 2022).

2. Differentiated Data Extraction (DDE) Attack and Logit-Rewriting Defense

DDE Attack: Differentiated Data Extraction (DDE) leverages systematic confidence discrepancies between $M_\mathrm{FT}$ (which is more confident at generating fine-tuned tokens) and $M_\mathrm{Base}$ (Li et al., 20 Jun 2025). For each prefix, DDE:

• Greedy-decodes $M_\mathrm{FT}$, records token probabilities $P_t$.
• Identifies branching points where $P_t < \tau$ (confidence threshold).
• Forces both models to continue on top-$\mathrm{MBR}$ alternatives at each branch, yielding candidate continuations $S$ (from $M_\mathrm{FT}$) and $B$ (from $M_\mathrm{Base}$).
• Selects a "closest" branch (minimizing average distance to $B$) and an "outlier" (maximizing within-set distance).

Defense Mechanism: The proposed defense introduces an inference-time logit-rewriting ("logit-smoothing") middleware. Its core objectives:

1. Prevent DDE from observing any tokens with $P(\text{token})<\tau$ by boosting the top-token probability to at least $\tau$ for every generation step.
2. Preserve the argmax: greedy decoding (the user-visible output) is unchanged.
3. Minimize impact on stochastic sampling.

Algorithmic Procedure:

1. Let original logits $L = (l_1 \ge l_2 \ge \cdots \ge l_n)$, $P = \mathrm{softmax}(L)$.
2. Sample target probability $v$ uniformly from $[\tau, 1]$.
3. Solve for $l'_1$ such that $\mathrm{softmax}(l'_1, l_2, ..., l_n)_1 = v$.
4. Publish modified logits $L' = (l'_1, l_2, ..., l_n)$.

This prevents the attacker from detecting low-confidence branch points, forcing DDE to degrade to the baseline greedy extraction, which exhibits negligible extraction power.

Empirical Evaluation: Applied to CodeLlama-7B, this defense altered neither the greedy (temperature=0) output nor Pass@1 code execution accuracy. Under moderate sampling (e.g., temperature=0.7, top-$p$=0.95), the impact on Pass@1 was a reduction from 0.65 to 0.63 ($\Delta = -0.02$ or $-3.1\%$), with $|\Delta \text{Pass@1}| \le 3\%$ across 90% of the temperature/top-$p$ grid. Higher temperature led to greater volatility, but is not standard operational practice (Li et al., 20 Jun 2025).

3. Distributional Anchoring: KL-Regularized and Proximal SFT Defenses

SFT generalization is hindered by overfitting to $D$ and drifting from the base model’s distribution, resulting in unstable or collapsed model behaviors. Recent advances address this through two main approaches:

Anchored Supervised Fine-Tuning (ASFT)

ASFT augments Dynamic Fine-Tuning (DFT)—which reweights log-likelihood terms by the model’s own output probabilities under stop-gradient—with a Kullback-Leibler divergence penalty to the pretrained reference policy (Zhu et al., 28 Sep 2025). The ASFT loss is

$$L_\mathrm{ASFT}(\theta) = L_\mathrm{DFT}(\theta) + \lambda \cdot \mathbb{E}_{x\in D}\left[ KL(\pi_\theta(\cdot|x) \,\|\, \pi_\mathrm{base}(\cdot|x)) \right],$$

where $\lambda>0$ and $\pi_\mathrm{base}$ is the original model.

This KL regularizer anchors $\pi_\theta$ to the base distribution, sharply reducing drift and stabilizing optimization, while maintaining the tighter RL-derived lower-bound that DFT introduces. Empirically, ASFT delivers substantial improvements: on MedMCQA, ASFT yields $+10.65$ points (+33.9% rel.) over base, $+8.3$ points over SFT; in math reasoning, it achieves $+16.14$ over base (+142%) (Zhu et al., 28 Sep 2025).

Proximal Supervised Fine-Tuning (PSFT)

PSFT transplants the trust-region constrained optimization of PPO (Proximal Policy Optimization) to the SFT context (Zhu et al., 25 Aug 2025). The PSFT objective is

$$L^{PSFT}(\theta) = \mathbb{E}_{(s_t,a_t)\sim D} \left[ -\min \left( r_t(\theta), \, \mathrm{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \right) \right],$$

where $$r_t(\theta) = \frac{\pi_\theta(a_t | s_t)}{\pi_{\theta_\mathrm{old}}(a_t | s_t)}$$ and $\epsilon \in [0.2, 0.3]$.

PSFT prevents catastrophic forgetting by bounding each update within a trust region around the previous policy, stabilizing token-level entropy and significantly improving out-of-domain generalization (e.g., Qwen2.5-7B-Instruct: SFT out-domain avg $57.90 \rightarrow$ PSFT $61.26$, $+3.4$ points) (Zhu et al., 25 Aug 2025).

4. Robust SFT Under Noisy Data

Real-world SFT frequently suffers from noise in instruction-response data, degrading downstream model quality (Luo et al., 19 Dec 2024). RobustFT is a two-stage framework targeting detection and correction of noisy labels:

1. Noise Detection: Multi-expert consensus via base model predictions, reasoning-enhanced predictions (chained Reas/Reflection), and label checking. Samples are split between clean and suspected noise pools.
2. Denoising and Selection: Contextual relabeling for noisy samples by retrieving k-nearest clean neighbors (semantic similarity), generating new responses via context-augmented LLM prompting, and synthesizing final labels via a review agent. An entropy-based selection retains only high-confidence samples for fine-tuning.

Empirical results demonstrate improvements over naive SFT, especially at high noise levels and across multiple datasets (e.g., MMLU: RobustFT 68.2 vs Vanilla 65.3 vs SFT 59.5 under 30% noise) (Luo et al., 19 Dec 2024).

5. Backdoor Mitigation Through Fine-Tuning

Supervised fine-tuning and its enhancement, super-fine-tuning (SFT+), provide a strong defense against backdoor attacks in image classifiers and, by implication, other deep networks (Sha et al., 2022). The standard procedure is full fine-tuning on clean data, which rapidly erases the attack-induced mapping. SFT+ introduces a "triangular" cyclic learning-rate schedule inspired by super-convergence, which accelerates backdoor forgetting in the early phase (large learning rates), then preserves clean accuracy in the later phase (small learning rates).

Key empirical findings include:

• Encoder-based SFT: Attack success rate (ASR) drops from $\sim$0.998 to 0.127 within 1 epoch, with clean accuracy unaffected.
• Super-FT+: In standalone settings, reduces difficult blended attack ASR from 0.998 to 0.081 in 3 epochs (clean accuracy 0.937), outperforming all tested baselines in both defense and computational burden.

The authors introduce "backdoor sequela" to measure defense side effects, observing that membership inference accuracy drops, indicating slight privacy gains (Sha et al., 2022).

6. Practical Defense Guidelines and Best Practices

• Apply logit-rewriting/smoothing as lightweight, inference-time middleware for LLMs exposed via black-box APIs, with a confidence floor $\tau \approx 0.8$ (Li et al., 20 Jun 2025).
• For SFT generalization and stability, always include a trust-region component—either via KL anchoring (ASFT, $\lambda$ in $[0.01, 0.1]$) or by constrained importance ratio (PSFT, $\epsilon$ in $[0.2, 0.3]$); monitor KL divergence and token entropy during training (Zhu et al., 28 Sep 2025, Zhu et al., 25 Aug 2025).
• In noisy data regimes, deploy multi-view noise detection, context-based relabeling, and entropy-driven selection to assemble high-fidelity SFT training sets (Luo et al., 19 Dec 2024).
• For backdoor-prone tasks, use super-fine-tuning cycles with aggressive but controlled learning rates; a plausible implication is that this may extend to other forms of vulnerability erasure where the attack relies on rare, fragile parameterizations (Sha et al., 2022).
• Minimize the exposure of full logits in public APIs to mitigate extraction and auditing attacks; restrict to top-k or processed outputs wherever possible (Li et al., 20 Jun 2025).
• Distinguish between inference and audit endpoints in SFT deployment, strictly limiting logit-level visibility to trusted users only.

7. Limitations and Open Questions

Empirical and theoretical work highlights certain gaps:

• The logit-rewriting defense provides only a trivial theoretical guarantee (failure of DDE’s branch-point step if $P_t \ge \tau$ always), lacking bounds on extraction error or utility impact beyond observed $\pm 3\%$ changes (Li et al., 20 Jun 2025).
• KL-anchored and PSFT approaches require calibration of regularization/trust-region hyperparameters to balance tightness and drift; out-of-domain generalization under domain or data shift remains an open research area (Zhu et al., 28 Sep 2025, Zhu et al., 25 Aug 2025).
• RobustFT has been evaluated primarily in classical benchmarks and LLMs; extension to complex, multi-modal or abstractive tasks is unproven (Luo et al., 19 Dec 2024).
• While SFT and its defenses effectively erase specific attacks (e.g., backdoors, noise contamination), sequela or the potential for attack re-injection may persist, particularly if clean data or learning rate schedules are suboptimal (Sha et al., 2022).

In summary, SFT defense now constitutes a diverse toolkit synthesizing inference interventions, trust-region optimization, noise-robust supervision, and dynamic, anchored reweighting. The resulting techniques offer effective protection against data extraction, distributional drift, and label noise, though further research is warranted on unexplored modalities and formal risk guarantees.