---
title: Segmented Supervised Fine-Tuning (S-SFT)
url: https://www.emergentmind.com/topics/segmented-supervised-fine-tuning-s-sft
type: topic
---

# Segmented Supervised Fine-Tuning (S-SFT)

Segmented Supervised Fine-Tuning (S-SFT) denotes a segmented, compositional, and modular interpretation of supervised fine-tuning in which adaptation is localized to selected segments of a model or training process rather than treated as an undifferentiated full-model update. In the literature considered here, S-SFT is not introduced as a single standardized algorithm. Instead, it emerges from convergent findings that supervised fine-tuning selectively activates task-specific attention heads, that complex-task adaptation can be approximated by combinations of basic-task activation patterns, that instruction tuning exhibits heterogeneous depth-wise dynamics across transformer layers, and that sequential or poorly partitioned post-training can induce forgetting or leave subsets of supervised data unlearned [2409.15820] [2604.11838] [2604.10079].

## 1. Scope and conceptual status

The cited works place S-SFT on conceptual rather than terminological grounds. "Supervised Fine-Tuning Achieve Rapid Task Adaption Via Alternating Attention Head Activation Patterns" does not use the term S-SFT directly, but it is explicitly described as highly relevant to a segmented or modular view of SFT because it treats attention heads as basic functional units, decomposes tasks into basic skills, and interprets complex-task SFT as activating and combining the appropriate subset of modules [2409.15820]. "A Layer-wise Analysis of Supervised Fine-Tuning" is directly relevant because it argues that supervised fine-tuning is not uniformly distributed across a transformer’s depth and that the most useful tuning region is a selected segment rather than the entire stack [2604.11838].

This segmentation can be understood along at least three axes. First, there is **functional segmentation**, in which attention heads are treated as task-relevant units with measurable activation patterns. Second, there is **depth-wise segmentation**, in which only selected blocks or layer ranges are updated. Third, there is **data segmentation**, in which the supervision itself is partitioned, ordered, resampled, or selected to reduce interference and improve efficiency. The literature also distinguishes S-SFT from adjacent staged training schemes. "Mitigating Forgetting in LLM Supervised Fine-Tuning and Preference Learning" is explicitly not about Segmented Supervised Fine-Tuning as a named method; it studies sequential post-training and shows why optimizing one objective after another can be sub-optimal [2410.15483]. "Why Supervised Fine-Tuning Fails to Learn" similarly does not propose a single S-SFT algorithm, but it strongly supports segmentation when segmentation is diagnostic and adaptive rather than naive [2604.10079].

A plausible implication is that S-SFT should be treated less as one algorithm than as a mechanistic design principle: localize adaptation where it is most effective, and partition supervision in ways that respect prerequisite knowledge, interference structure, and retention constraints.

## 2. Attention-head activation as functional segmentation

A central mechanistic account of S-SFT comes from attention-head activation analysis. For a model with \(L\) layers and \(H\) heads per layer, the activation-pattern method defines an activation pattern matrix \(AP^{\mathcal{T}} \in \mathbb{R}^{L \times H}\), whose entries measure the activation level of each head on task dataset \(\mathcal{T}\). The activation level is computed as

$$
AL_{l,h} = \frac{1}{N}\sum_i {\Gamma_{l,h}^T}\frac{\partial L(x_i)}{\partial \Gamma_{l,h}},
$$

where \(N\) is the number of examples, \(x_i\) is the \(i\)-th instance, \(L(x_i)\) is the model loss, and \(\Gamma_{l,h}\) denotes the attention matrix of head \(h\) in layer \(l\) [2409.15820].

This formulation treats gradients with respect to attention matrices as a feature-attribution signal. The reported finding is that supervised fine-tuning induces **selective, task-specific head activation**. Across models and tasks, activation patterns become more concentrated and more task-characteristic after fine-tuning. Distributional changes are quantified with the Gini coefficient, coefficient of variation (CV), and kurtosis. On Llama3-8B, the Gini drops from 0.50 to 0.33, CV from 1.19 to 0.71, and kurtosis from 95.37 to 39.55 after SFT; similar reductions are reported for Gemma-7B and OPT-6.7B [2409.15820].

The same study also reports that task correlations cluster in ways aligned with intuitive task families, such as **math/code versus text reasoning**, and that correlations decrease after SFT, indicating greater specialization. This is a direct segmentation claim at the functional level: SFT reshapes the head-activation landscape in a task-specific way rather than merely altering weights globally.

The compositional claim is formalized as

$$
\Delta AP^{complex} = \sum_{i=1}^{n} \alpha_i \Delta AP^{basic_i} + \epsilon.
$$

Here, the change in activation pattern for a complex task is modeled as a linear combination of changes from several basic tasks. The empirical fits are strong: for SGSM, the best-fitting basic tasks are Code Search Net and GSM8K with \(R^2 \approx 0.97\); in the Infinity Instruct setting, instructions requiring both logical reasoning and programming/software development are best explained by combining activation patterns of instructions requiring only logical reasoning and only programming/software development, with \(R^2 \approx 0.95\) [2409.15820].

The same paper further reports that activation patterns often shift dramatically in the early stages of fine-tuning, measured by correlation coefficient and mean squared error (MSE) across checkpoints. Stronger pretrained models require less data and show smaller necessary reconfiguration, whereas weaker models such as OPT-6.7B require more data and exhibit larger activation changes. This suggests that functional segmentation is tied to the reuse of pretrained subskills: if the required basic capabilities already exist, a small amount of SFT may be sufficient to trigger the relevant activation configuration.

## 3. Depth-wise localization and selective layer updating

A second major formulation of S-SFT is depth-wise. "A Layer-wise Analysis of Supervised Fine-Tuning" argues that standard SFT and PEFT methods such as LoRA typically apply updates uniformly across all layers, implicitly assuming equal contribution across depth, but that this assumption is empirically suboptimal [2604.11838]. The paper’s central claim is a distinct depth-dependent pattern: **middle layers (20\%-80\%) are stable, whereas final layers exhibit high sensitivity**.

The representation-space evidence is consistent across scales. For OLMo2-32B, the CKA score between base and SFT representations stays above 0.98 through shallow layers and then falls in deeper layers to about 0.94 by the last layer. Mean shift and cosine similarity display the same trend, with mean shift remaining negligible in most shallow layers but reaching more than 12.0 in the final few layers. Intrinsic metrics show a synchronized three-stage structure: an initial transition from the embedding layer to layer 1, a long stable middle region, and a final bottleneck in the last \(\sim 20\%\) of layers. In the middle band, effective rank plateaus at a relatively high value and the condition number is low and stable; in the tail layers, effective rank collapses and spectral norm blows up [2604.11838].

Probing and optimization analyses reinforce this picture. Layer-wise probing on OLMo2-32B shows a dormancy-to-emergence pattern: probing accuracy is near zero for the first 50 layers and then rises rapidly in the last 14 layers, reaching over 0.60 on MMLU. Weight changes are J-shaped: for OLMo2-13B, the Frobenius norm of parameter updates rises from about 0.05 early on to above 0.10 at the end. The reported correlation between weight change and cosine similarity is strongly negative, with \(r=-0.79\) [2604.11838].

On the basis of this localization, the paper proposes **Mid-Block Efficient Tuning**. The model is divided into \(M\) equal segments, and LoRA adapters are placed only in selected segments while keeping total trainable parameters nearly constant. With \(M=5\), configurations such as `01000`, `01100`, `01110`, `10000`, and `00001` indicate which segments receive updates. The best-performing settings are concentrated in the middle or upper-middle region. On OLMo2-7B, `01000` reaches 0.375 on GSM8K versus 0.28 for the full-layer LoRA baseline; on OLMo2-32B, `01100` reaches 0.32 versus 0.29; on OLMo2-13B, `01110` reaches 0.30 versus 0.27. By contrast, using only the bottom segment (`10000`) or only the top segment (`00001`) performs poorly. Ablations with \(M=3\) and \(M=10\) yield an inverted-U pattern, and the appendix states that the optimal region is “a broad, stable region that is tolerant to minor boundary perturbations” [2604.11838].

For S-SFT, the significance is precise: selective tuning is beneficial, but the best segment is not simply the most sensitive one. The final layers move the most under ordinary SFT, yet the reported optimal substrate for efficient adaptation is the middle or upper-middle depth range. The paper therefore refines segmented tuning into a localization problem rather than a simple “tune the top layers” heuristic.

## 4. Data partitioning, incomplete learning, and forgetting

A third axis of S-SFT concerns the segmentation of supervision itself. "Why Supervised Fine-Tuning Fails to Learn" formalizes the **Incomplete Learning Phenomenon (ILP)** as post-training failure to internalize supervised instances: after SFT convergence, the model still cannot correctly reproduce some of the training samples it saw during fine-tuning [2604.10079]. This is explicitly distinguished from held-out generalization failure, catastrophic forgetting, and intentional machine unlearning.

The paper reports that ILP is widespread across Qwen, LLaMA/LLaMA2, and OLMo2, and across ARC, CommonsenseQA, SocialIQA, MedMCQA, MedQA, LegalBench, FinanceBench, TechFAQ, SQuAD, CoQA, TriviaQA, Natural Questions, CNN/DailyMail, XSum, OpenWebText, AG News, IMDB, MultiNLI, QQP, and additional OLMo2 evaluation benchmarks such as MMLU, AGIEval, BBH, GPQA, NQ, and MMLU-Multi. Across ten benchmark SFT datasets, on average, **15.3\% \(\pm\) 2.1\%** of supervised instances remain unlearned after SFT convergence [2604.10079].

Five recurrent causes are identified: missing prerequisite knowledge in the pre-trained model, conflicts between SFT supervision and pre-training knowledge, internal inconsistencies within SFT data, left-side forgetting during sequential fine-tuning, and insufficient optimization for rare or complex patterns. The diagnostic framework uses post-SFT correctness, pass@N, BoN-5 accuracy, base-model zero-shot success or failure, confidence in wrong answers, Jensen–Shannon divergence between base and SFT predictions, and accuracy drop over time on subsets. The operational ILP criterion treats an instance as unlearned when pass@N falls below a threshold \(T\), with \(T=0.2\) under BoN-5 sampling unless otherwise stated [2604.10079].

Several mitigations are directly relevant to S-SFT. For missing prerequisite knowledge and knowledge conflicts, the paper uses continued pre-training (CPT) with a mixed corpus

$$
\mathcal{C}_\text{mix} = 0.8\mathcal{C}_\text{general} + 0.2\mathcal{C}_\text{aug}.
$$

For internal contradictions, it uses semantic similarity detection, external judgment with GPT and DeepSeek, removal of incorrect samples, and bucket assignment so conflicting samples do not co-occur in the same minibatch. For left-side forgetting, it uses global shuffling, periodic validation of each subset, and dynamic resampling when a subset’s accuracy drops. For insufficient optimization, it uses progressive epoch increment with stopping criterion based on validation loss increase and \(\delta = 0.01\) [2604.10079].

The empirical findings are directly cautionary for segmented training. Increasing SFT from 2 to 10 epochs yields only small gains for knowledge blind spots, such as MedQA coverage moving from 65.3% to 66.8%, whereas CPT + SFT produces substantially larger gains, including MedQA +8.3, LegalBench +7.9, TechFAQ +8.7, and FinanceBench +8.0. For OLMo2, the reported total SFT knowledge Non-Existence Rate is 19.3% and the total Conflict Rate is 14.5%, with professional knowledge showing non-existence 27.4% and conflict 18.4%. On summarization, dynamic resampling improves the first 10% of data from ROUGE-L 0.41 to 0.53, a +29% gain, while the last 10% drops only slightly by -1.6% [2604.10079].

These results imply that data segmentation is not automatically beneficial. Segments can encode prerequisite structure, conflict structure, and exposure imbalance. A plausible implication is that effective S-SFT must be diagnostic, order-aware, conflict-aware, and resampling-aware rather than merely sequential.

## 5. Activation-guided data construction and retrieval

The most explicit application of segmented thinking to SFT procedure appears in the activation-pattern paper. For **complex tasks with limited data**, the authors estimate the mixture weights \(\alpha_i\) in the activation-composition regression and construct a preliminary dataset from basic tasks with proportions

$$
\text{Dataset}^{pre}=\{N \times \alpha_i / \sum_i \alpha_i \}_{i=0}^{|B|},
$$

where \(N\) is the desired size of the preliminary dataset and \(B\) is the set of candidate basic tasks [2409.15820].

The training pipeline is two-stage: first train on the constructed basic-skill dataset, then fine-tune on the small target dataset. On MathBench, this improves over both direct fine-tuning on the small target set and random mixtures of basic tasks. On Llama-7B, it reaches 36.82 versus 31.78 for direct SFT and 33.82 for random mixing. On Llama2-7B, it reaches 40.70 versus 36.43 and 34.51. On Llama3-8B, it reaches 55.03 versus 52.33 and 51.16 [2409.15820].

A second application addresses unavailable or private target data. The method defines the application dataset as the top-\(m\) samples maximizing correlation with activation patterns of pseudo-private target data:

$$
\text{Dataset}^{app} = \operatorname{Top}_m  Corr(AP^{tar}, AP^{D_i}).
$$

Using MMLU as a public pool and treating target-domain validation or development data as pseudo-private, the paper selects the most similar non-target samples and fine-tunes on them. The reported outcome is consistent improvement over randomly chosen non-target data on average accuracy across math, physics, chemistry, and biology, with the effect strongest when similarity-based selection closely matches the target-domain structure [2409.15820].

These procedures are operationally important because they turn segmentation into a dataset-construction and dataset-retrieval principle. Instead of only restricting which parameters move, the method segments the target task into constituent basic skills and reconstructs a training curriculum from public or basic-task data. The experiments were run on NVIDIA A100 GPUs with DeepSpeed and ZeRO-2, batch size 4, max length 1024, learning rate \(1\times 10^{-6}\), gradient accumulation 4, and bf16 full fine-tuning; evaluation used OpenCompass and greedy decoding to reduce randomness [2409.15820].

## 6. Conceptual boundaries, adjacent methods, and open questions

The literature also clarifies what S-SFT is not. It is not equivalent to naive sequential stage switching. "Mitigating Forgetting in LLM Supervised Fine-Tuning and Preference Learning" studies a two-stage post-training pipeline in which a preference-learning objective is optimized first and an SFT objective second, and it theoretically proves the sub-optimality of sequential post-training while proposing a joint post-training framework with theoretical convergence guarantees and similar computational cost [2410.15483]. The core problem is that each phase optimizes only one objective at a time, so the optimization path can move toward one objective-specific optimum and then away from it during the next stage. In the paper’s framing, the trajectory may oscillate between extreme trade-offs rather than reach a jointly good solution. This supports the motivation for segmented or coupled training, but it is not itself a segmented SFT method.

S-SFT is also not identical to tuning the layers that exhibit the largest representational change. The layer-wise results show that final layers are the most sensitive, yet the best-performing efficient tuning region is the middle or upper-middle blocks rather than the topmost segment [2604.11838]. This is a substantive correction to a common simplification: the layers that move most under ordinary SFT are not necessarily the layers that should be targeted for efficient supervised adaptation.

Nor does segmented training, by itself, guarantee that supervision is fully learned. The ILP results show that aggregate improvements can mask persistent unlearned subsets, and that some failures are due to missing prerequisite knowledge or conflicts with pre-training knowledge rather than inadequate optimization alone [2604.10079]. Likewise, the attention-head analysis acknowledges two limitations: it does not deeply analyze the effect of individual head activation magnitudes on performance, and much of the validation is on comparatively simpler or benchmark-style tasks rather than the most complex real-world settings [2409.15820].

Taken together, these boundaries suggest a precise research interpretation. Segmented Supervised Fine-Tuning is best viewed as a mechanistic and procedural framework in which supervised adaptation is localized over functionally relevant heads, depth-wise blocks, or diagnostically defined data segments. The available evidence supports a segmented, compositional, and modular account of SFT, while also indicating that useful segmentation must be coupled to representation analysis, interference diagnosis, and objective trade-off management rather than treated as a purely architectural heuristic.

Source: https://www.emergentmind.com/topics/segmented-supervised-fine-tuning-s-sft