---
title: 'Blockwise SFT: Targeted Fine-Tuning'
url: https://www.emergentmind.com/topics/blockwise-supervised-fine-tuning-sft
type: topic
---

# Blockwise SFT: Targeted Fine-Tuning

Blockwise Supervised Fine-Tuning (SFT) refers to a family of methodologies in which parameter adaptation in large pre-trained neural networks is restricted to a well-defined subset, or “block,” of model components. By constraining which network segments are updated—using blocks delineated as attention heads, transformer layers, neuron rows, or response token regions—blockwise SFT frameworks increase data efficiency, boost stability, reduce memory and compute requirements, and often provide superior generalization with limited labeled data compared to full-model fine-tuning. Techniques across diverse modalities and architectures—including transformers for language, vision, sparse models, and diffusion LMs—have converged on the utility of blockwise SFT for targeted adaptation, pruning-aware optimization, and training-inference alignment.

## 1. Blockwise SFT in Parameter Subspace Selection

Blockwise SFT formalizes a restriction of parameter updates to a targeted subnetwork or block, with block granularity adapted to the network class:

- **Structural segmentation**: In standard deep networks, blocks may correspond to contiguous groups of layers, identified by architectural separators (e.g., activation or pooling layers) or by sliding windows of fixed length [2301.06133].
- **Head-level granularity**: Transformer models naturally segment into attention heads; only a salient subset, often task-specific as determined by causality analysis, is updated during adaptation [2601.08146].
- **Neuron-row blocks**: For linear layers, each output neuron row constitutes a block. Structural pruning metrics rank these rows for downstream task importance, after which adaptation is restricted to the most relevant [2502.11439].
- **Diffusion LMs**: Blockwise tuning is interpreted temporally or sequentially over response token blocks, aligning training optimization to blockwise inference steps [2508.19529].

This block-oriented selection enables fine-grained trade-offs between plasticity and generalization, adjusting the degree of adaptation per task and data regime.

## 2. Core Methodologies and Algorithmic Frameworks

Distinct algorithmic implementations of blockwise SFT have emerged, each with strong theoretical and empirical underpinnings:

- **Circuit-Targeted SFT (CT-SFT)** discovers a label-balanced, causally relevant circuit of attention heads using task-directional relevance scoring derived from Contextual Decomposition Transformer (CD-T) analysis. Only the identified heads and LayerNorm parameters are updated, implemented via head-level gradient masking [2601.08146].
- **EBFT** (Effective Block-wise Fine-Tuning) sequentially optimizes a per-block reconstruction error—mean squared error between the outputs of each pruned “student” block and its unpruned “teacher” counterpart—cycling through model blocks one at a time with backpropagation localized to only the active block [2402.12419].
- **SPruFT** (Structured-Pruning-based Sparse Fine-Tuning) selects top neuron rows using Taylor or magnitude criteria, then adapts only those rows via a sparse update matrix ($\Delta W$), summing original and adapted activations to retain implementation efficiency and minimize memory use [2502.11439].
- **Layer/Block-wise Optimization** in CV models first partitions the network into candidate blocks, then searches (on a small held-out subset) for the most predictive block or group of layers to unfreeze, with all others kept static [2301.06133].
- **Blockwise SFT for Diffusion LMs** partitions responses into fixed-size token blocks, at each training step sampling a single block for gradient computation, masking all future tokens (suffix leakage prevention) and keeping prefix context clean [2508.19529].

Commonly, these algorithms involve an initial discovery or selection phase followed by block-restricted SGD, using optimizer masks or parameter groupings at the framework level.

## 3. Theoretical and Empirical Advantages

Empirical and analytical investigations across blockwise SFT variants highlight several advantages:

- **Stability and regularization**: Blockwise adaptation imposes a hard prior against overfitting, especially when labeled data is limited, by only permitting updates in regions of the parameter space causally implicated in the target task [2601.08146].
- **Data efficiency**: By focusing plasticity, blockwise SFT often outperforms full-model fine-tuning with 1–2 orders of magnitude fewer trainable parameters [2601.08146, 2502.11439].
- **Resistance to catastrophic forgetting**: When transferring between tasks (e.g., cross-lingual adaptation), restricting updates to task-relevant blocks preserves source task competence significantly better than full fine-tuning, as measured by post-transfer accuracy on proxy tasks [2601.08146].
- **Reduced compute and memory**: SPruFT and EBFT report 1.5–2× lower memory utilization than LoRA, and much lower than full fine-tuning, without incurring the kernel overhead of unstructured sparse updates [2502.11439, 2402.12419].
- **Improved sample efficiency under modest hardware**: All weights and gradients for a single block can reside in memory at once, enabling full SFT on 6–7B parameter models on a 16GB GPU in ~30 minutes, provided a blockwise methodology is used [2402.12419].
- **Training-inference alignment**: In diffusion LMs, blockwise SFT eliminates bias from prefix noise and suffix leakage, ensuring the training objective matches the likelihood factorization imposed by blockwise decoding [2508.19529].

A typical outcome is that, under constrained resource budgets, blockwise SFT formulations both converge faster and yield higher or equivalent accuracy versus strong baselines.

## 4. Representative Experimental Results

Blockwise SFT variants achieve substantial empirical gains, as summarized below:

| Method                 | # Params Updated   | Main Task/Dataset    | Performance              | Memory/Speedup      | Reference     |
|------------------------|-------------------|----------------------|--------------------------|---------------------|--------------|
| CT-SFT (depth 2)       | 0.66% (18 heads+LN) | NusaX-Senti XNLI     | +0.119 (Acehnese) over FT | Catastrophic forgetting reduced | [2601.08146] |
| EBFT (Wikitext2, 7B)   | ~one block at a time | 70% sparsity         | PPL 16.88 vs. DSnoT 75.14 | 30 min/16 GB GPU    | [2402.12419] |
| SPruFT (Llama2-7B)     | 128 rows/block     | Alpaca, 9 tasks      | 60.86% (LoRA 60.69%)     | 17.62 GB (23.46 GB LoRA) | [2502.11439]|
| Blockwise SFT (Diff-LM)| LoRA (rank 256)    | GSM8K, MATH          | +5pp GSM8K vs. SFT       | No arch chg, stable | [2508.19529] |
| Blockwise (separator)  | ~block/group       | Tf_flowers, 5 models | 0.8518 (lowest variance) | Robust to overfit   | [2301.06133] |

Key results indicate that blockwise SFT matches or exceeds the accuracy of full or LoRA-based adaptation, with smaller memory and substantially lower risk of performance drop post-transfer or under limited supervision.

## 5. Design Choices and Practical Implementation

Critical design and implementation details vary by context:

- **Block definition and selection**: Can be architecture-driven (e.g., transformer heads, convolutional layer groups) or data-driven via importance scoring (Taylor saliency, quantile-mean, magnitude).
- **Optimizer masking**: Per-parameter optimizer masks (e.g., PyTorch's `param_group['requires_grad']` or custom hooks) restrict gradients to block-selected parameters [2601.08146].
- **Data partitioning**: Empirical works typically use a small held-out or “discovery” pool (e.g., 50–400 examples) for initial importance or relevance computation [2601.08146, 2402.12419]; subsequent updates are performed on disjoint tuning/validation splits.
- **Hyperparameters**: Learning rates (e.g., 5×10⁻⁵ for CT-SFT), batch sizes (typically small, ≤16), and early-stopping (loss stagnation, relative change <1e−3) are tuned per block.
- **Sparsity trade-offs**: Increasing the selection ratio p yields larger blocks or circuits, trading off faithfulness to the source mechanism for adaptation capacity [2601.08146].
- **No requirement for custom sparse kernels**: SPruFT and other blockwise SFT approaches use standard dense operations over small matrices, sidestepping the engineering issues characteristic of elementwise sparse updates [2502.11439].

Best practices recommend block-level tuning for data-scarce tasks, starting selection from higher network layers, and monitoring adaptation on small validation splits for overfitting.

## 6. Applications, Extensions, and Generalization

Blockwise SFT methodologies have been successfully adapted across several domains:

- **Transformer LMs: Cross-lingual transfer**, low-resource adaptation [2601.08146].
- **Sparse and pruned models**: Structured/unstructured sparsity, pruning-aware retraining [2402.12419, 2502.11439].
- **Vision backbones**: CV models (VGG, ResNet, MobileNet) show blockwise SFT outperforms both head-only and full model adaptation in image-based tasks [2301.06133].
- **Diffusion LMs**: Exact alignment between blockwise decoding and SFT leads to state-of-the-art math and reasoning performance [2508.19529].

The frameworks generalize to multi-task SFT (by mixing downstream task examples in calibration sets [2402.12419]), and are robust to adaptation in both low-data and high-sparsity regimes [2502.11439].

## 7. Limitations and Open Directions

Known limitations include:

- **Block discovery cost**: For many block selection methods, especially those requiring gradient or structural importance estimates, there is a pre-adaptation compute overhead.
- **Block size/selection granularity**: Selection of too large a block reduces memory/computation savings, while too small may underfit. Alignment of train and inference block granularity is essential in certain architectures [2508.19529].
- **Remaining overfitting risk**: In minimal-data regimes, care must be taken with block selection to prevent adaptation to spurious features.
- **Implementation overhead**: Blockwise SFT demands per-block optimizer and parameter management, though in practice this is standardized in major frameworks.

Future work may extend blockwise SFT to gradient-free solvers, adaptive block sizing, or human-feedback-driven block adaptation, particularly in diffusion and semi-autoregressive generative models [2508.19529]. A plausible implication is that as models and tasks grow still larger and more heterogeneous, blockwise SFT will remain core to scalable, robust, and data-efficient adaptation strategies.

Source: https://www.emergentmind.com/topics/blockwise-supervised-fine-tuning-sft