Perturbed Supervised Fine-Tuning Techniques

• Perturbed supervised fine-tuning is a method that introduces controlled noise into the fine-tuning process to activate dormant parameters and reduce overfitting.
• Techniques like PATS, Match-Tuning, PPCL, and PAC-tuning adapt noise injection based on task sensitivity, improving both robustness and downstream performance.
• Empirical benchmarks such as GLUE and MMLU show that these methods enhance generalization, safety, and performance with minimal additional computational overhead.

Perturbed supervised fine-tuning refers to a set of methodologies that enhance the robustness, generalization, and parameter efficiency of pretrained LLMs (PLMs) by injecting controlled stochasticity—noise or perturbation—directly into the fine-tuning process, either at the parameter, representation, input, or instruction levels. This paradigm addresses key issues such as parameter redundancy, susceptibility to adversarial or semantic perturbations, and overfitting in low-data regimes. Representative strategies include sensitivity-aware noise injection (PATS), in-batch representation fusion (Match-Tuning), prompt perturbation consistency regularization (PPCL), PAC-Bayes–driven noise learning (PAC-tuning), and systematic noisy instruction-tuning for LLMs. Empirical evidence across multiple benchmarks supports that such perturbations can consistently improve both nominal and worst-case downstream performance, including under adversarial attacks and instruction corruption.

1. Motivation and Core Problem Statement

Traditional supervised fine-tuning of PLMs initializes model weights from a pretraining checkpoint and runs gradient-based updates on a downstream dataset, optimizing the empirical risk $$\mathcal{L}(\Theta) = \mathbb{E}_{(x,y)\sim\mathcal{D}}[\ell(f(x;\Theta),y)]$$ over all $n$ parameters $\Theta\in\mathbb{R}^n$. Despite the strong generalization of PLMs, direct fine-tuning often results in under-utilization or redundancy of a substantial subset of parameters, which leads to overfitting—especially in limited-data scenarios—and leaves the model vulnerable to adversarial or semantic perturbations (Zhang et al., 2022, Tong et al., 2022, Qiang et al., 24 Feb 2024, Alajrami et al., 3 Oct 2025, Liu et al., 2023).

Perturbed supervised fine-tuning systematically targets these issues by activating dormant parameters, smoothing the loss landscape, balancing contributions across the parameter space, and directly regularizing against sensitivity to input or instruction noise.

2. Sensitivity-Aware Parameter Perturbation (PATS)

PATS (Perturbation According To Sensitivity) exemplifies parameter-level perturbation by quantifying each parameter's downstream-task sensitivity—approximated as $$s_j \approx |\,\theta_j \cdot \nabla_{\theta_j}\mathcal{L}(\Theta)|$$—and using this to modulate independent Gaussian noise injection. Parameters with low averaged sensitivity receive stronger stochastic updates, while highly sensitive (downstream-crucial) weights are protected (Zhang et al., 2022). The update rule for each parameter $\theta_j$ is:

$$\theta_j \gets \theta_j - \eta \nabla_{\theta_j}\mathcal{L}(\Theta) + z_j q_j$$

where $$q_j \sim \mathcal{N}(0, \sigma_j^2),\ \sigma_j = \sqrt{r_j}$$ and $$r_j = \lambda\, \max\left(\frac{\bar{s}}{\bar{s}_j + \epsilon} - \gamma,\ 0 \right)$$ with $\bar{s}_j$ an EMA of $s_j$.

PATS consistently yields tighter, more uniform sensitivity distributions post-training, reduces the fraction of "dead" parameters, and delivers nominal GLUE dev-set gains: BERT-base improves from 83.18 to 84.00; RoBERTa-large from 88.25 to 88.90, with outsized improvement for small-data tasks (CoLA +1.73) (Zhang et al., 2022).

3. In-Batch Representation-Level Perturbation and Interpolation (Match-Tuning)

Match-Tuning (Tong et al., 2022) applies representation-level perturbations: within each mini-batch, a similarity matrix $M$ fuses each example's encoding $h_\theta(x^{(i)})$ with those of other batch members. Early in training, off-diagonal "negative" label representations act as adversarial noise, while later, $M$ transitions to interpolate among same-label instances.

The composite fused representation $z^{(i)}$ for each batch member is:

$$z^{(i)} = \sum_{j=1}^{n} M_{ij}\, h_\theta(x^{(j)})$$

where $$M_{ij} = \exp(h_\theta(x^{(i)}) \cdot h_\theta(x^{(j)}) / \tau)/ \sum_k \exp(h_\theta(x^{(i)}) \cdot h_\theta(x^{(k)}) / \tau)$$.

This mechanism robustly enhances generalization under label and class imbalance, and adversarial input noise. GLUE average: vanilla at 78.53 vs. Match-Tuning at 80.17 (+1.64). AdvGLUE robustness improved by +4.11 points. The method incurs minimal additional computational overhead, only requiring an extra matrix multiplication per batch (Tong et al., 2022).

4. Input and Prompt-Level Perturbation: PPCL and Instruction Noise

Prompt Perturbation Consistency Learning (PPCL) directly regularizes against input (prompt) perturbations. For each (clean, perturbed) input pair $(x, x')$, PPCL adds cross-entropy losses for both, plus a Jensen–Shannon divergence between output token distributions:

$$L_{\mathrm{PPCL}}(\theta) = L_c + L_p + \lambda_3 L_{\mathrm{JS}}$$

where $L_{\mathrm{JS}}$ is averaged token-wise JS divergence (Qiang et al., 24 Feb 2024).

PPCL is empirically sample-efficient: recovers 59% (intent classification) and 69% (slot-filling) of the performance loss due to oronym, synonym, or paraphrase perturbations using tenfold fewer augmented samples than standard data augmentation.

Instruction-level perturbation (Alajrami et al., 3 Oct 2025) extends the paradigm to LLMs by explicitly corrupting training instructions during supervised tuning. Six operators—stop-word deletion, word shuffling, word deletion, MLM-based word replacement and insertion, and misspelling—yield perturbed instructions with mixture ratio $\alpha$. The training loss is a mixture over clean and perturbed instructions:

$$L(\theta;\alpha) = \mathbb{E}_{(x,y)\sim D}\left[ (1-\alpha)\,(-\log p_\theta(y|x)) + \alpha/K \sum_k (-\log p_\theta(y|P_k(x))) \right]$$

MMLU-5shot accuracy for Llama-70B rises from 75.8% (vanilla) to 78.6% ($\alpha=100\%$, 100% noisy instructions), with improved truthfulness and reduced toxicity. Robustness under noisy test-time prompts also increases monotonically with $\alpha$ (Alajrami et al., 3 Oct 2025).

5. PAC-Bayes–Driven Perturbed Fine-Tuning (PAC-tuning)

PAC-tuning innovates in learning adaptive parameter-level noise scales by explicitly minimizing a differentiable PAC-Bayes generalization bound (Liu et al., 2023). Stage 1 jointly learns parameter variances $(\xi,\epsilon)$ in Gaussian posteriors over both backbone and head weights via KL-augmented empirical risk minimization. Stage 2 fixes all learned variances and continues fine-tuning using perturbed gradient descent:

$$\theta' = \theta + \sqrt{\xi^*}\odot\tau,\quad \omega' = \omega + \sqrt{\epsilon^*}\odot\tau$$

followed by standard gradient descent on perturbed parameters.

PAC-tuning delivers substantial gains in few-shot generalization. On BERT-base, CoLA MCC rises from 0.235 (vanilla) to 0.335; SST accuracy from 0.773 to 0.834; QNLI accuracy from 0.702 to 0.709 (Liu et al., 2023). This suggests that PAC-grounded noise learning produces nonvacuous generalization bounds and regularization without hand-tuning of noise hyperparameters.

6. Empirical Performance, Robustness, and Efficiency

Across strategies, perturbed supervised fine-tuning consistently improves both standard accuracy and robustness to adversarial and semantic perturbations. Performance gains are most pronounced on low-data and reasoning-intensive tasks. Sample efficiency is enhanced through sensitivity-aware and PAC-informed noise scaling, as well as via joint regularization (PPCL) rather than brute-force data augmentation (Zhang et al., 2022, Tong et al., 2022, Qiang et al., 24 Feb 2024, Alajrami et al., 3 Oct 2025, Liu et al., 2023).

Instruction-level noise mixtures not only confer robustness to input corruption but also lead to monotonic improvements in output safety and truthfulness metrics (Alajrami et al., 3 Oct 2025). Minimal computational overhead is preserved for all approaches, except PAC-tuning's Stage 1, which may incur 2–5× more epochs but yields calibrated per-parameter noise estimates.

7. Limitations, Practical Recommendations, and Future Directions

Perturbed supervised fine-tuning methods typically require tuning of a small set of hyperparameters, such as base noise magnitude ($\lambda$), EMA decay ($\beta$), mixture ratio ($\alpha$), or regularizer weight ($\lambda_3$). Default values are robust across major benchmarks, but domain-specific tuning may be needed for optimal low-data performance. All methods are compatible with standard fine-tuning optimizers and efficient adaptation schemes (LoRA, QLoRA).

Current limitations include evaluation on large-scale PLMs (GPT-3/4) for PAC-tuning, analysis of side effects from excessive instruction noise (loss of crucial semantics), and limited benchmarks for multi-turn or complex structured prediction. Future work aims to extend consistency-based regularization to broader NLP tasks and investigate dynamic/adaptive perturbation scheduling (Qiang et al., 24 Feb 2024, Alajrami et al., 3 Oct 2025, Liu et al., 2023).

---

In summary, perturbed supervised fine-tuning constitutes a principled suite of enhancements to standard PLM tuning. By injecting noise according to task-specific sensitivity, in-batch mixing, prompt-level similarity regularization, PAC-Bayes noise learning, or instruction corruption, these approaches increase generalization, robustness, and safety with minimal overhead and clear empirical superiority across multiple natural language understanding benchmarks.