Prune-SFT-Quantize Pipeline

Updated 3 July 2026

Prune-SFT-Quantize is a neural network compression method that combines pruning, supervised fine-tuning, and quantization to reduce model size and improve deployment efficiency.
The pipeline employs both structured and unstructured pruning techniques, followed by fixed sparsity SFT to recover performance, and low-bit quantization to optimize memory and compute efficiency.
Empirical studies show significant reductions in parameters and VRAM usage, with minimal accuracy loss, making it applicable to multimodal language and vision models.

The Prune-SFT-Quantize pipeline is an empirically validated, sequential neural network compression workflow that integrates structured or unstructured pruning, supervised fine-tuning (SFT), and post-training quantization. Primarily motivated by the need to deploy performant large models under stringent memory and latency constraints, this pipeline yields highly parameter- and compute-efficient representations with minimal degradation in accuracy. Rigorous studies demonstrate its ability to accommodate multimodal LLMs (MLLMs), dense vision models, and specialized hardware targets, with theoretical performance bounds and empirical benchmarks substantiating its practical efficacy (Khan et al., 29 Jul 2025, Yu et al., 2020, Zhang et al., 2024, Niemi et al., 27 Mar 2026).

1. Pipeline Overview and Definitions

The Prune-SFT-Quantize workflow operates in three strictly sequential stages:

Pruning: Identification and removal of redundant weights or entire network components, resulting in a sparse or thinned architecture.
Supervised Fine-Tuning (SFT): Task-driven optimization, with the sparsity pattern fixed, to recover or improve task accuracy lost during pruning.
Quantization: Reduction of weight and optionally activation numerical precision, typically to low-bit fixed-point or integer formats, for enhanced memory and hardware efficiency.

The pipeline's goal is to maximize compression—via parameter, memory, and computation reductions—while retaining as much of the original model's task performance as possible. Admissible variants include structured (e.g., block or channel), semi-structured, and unstructured pruning, as well as per-layer, per-channel, or per-weight quantization granularities.

In the medical MLLM domain, the pipeline enables, for example, 7B-parameter models to execute within 4 GB VRAM, cutting memory usage by 70% and yielding up to 4% higher performance than prior pruning-quantization baselines at equivalent compression ratios (Khan et al., 29 Jul 2025).

2. Pruning Algorithms and Strategies

Structured and Unstructured Pruning

Pruning in this pipeline can target arbitrary granularity:

Structured Pruning: Transformer blocks (as in LLAVA) or channels are considered atomic units. Blocks 1, 2, and the last are typically retained, and others are scored for removal.
Unstructured Pruning: Applies masking at the weight level based on importance criteria such as the Taylor score or magnitude.

In LLAVA-based MLLMs (Khan et al., 29 Jul 2025), layer-importance is determined by the average cosine similarity between model outputs—embedded with a sentence transformer—on a small calibration set before and after removing a block. The block with the highest similarity (least impact) is pruned iteratively until the desired compression is achieved. The precise pruning pseudocode is:

$g_i$ 1

Unstructured and semi-structured methods supported in PQuantML (Niemi et al., 27 Mar 2026) include magnitude, activity, or data-aware scoring, and can be set up for both mask learning during training or post-training one-shot pruning.

Theoretical and Alternative Pruning Methods

The stochastic path-following framework (Zhang et al., 2024) provides a theoretically grounded alternative, treating pruning as an optimization with stochastic, unbiased operators, ensuring tight error bounds on the pruned model’s output.

Taylor score-driven pruning (Yu et al., 2020) defines:

$S_{\rm Taylor}(w_i) = |g_i w_i|$

where $g_i$ is the task loss gradient for parameter $w_i$ . Weights below threshold $T$ are pruned ( $m_i \gets 0$ ).

3. Supervised Fine-Tuning (SFT) under Fixed Sparsity

Following pruning, the model is re-optimized to recover performance. SFT is conducted with a frozen sparsity mask; pruned weights do not participate in training or inference.

SFT Procedure

Dataset: SFT uses the full downstream task set or a subset, as in the 1,737 DermNet training images with 2-turn text-vQA dialogue (Khan et al., 29 Jul 2025).
Loss and curriculum: Standard cross-entropy over next-token distributions (for LLMs), with no special curriculum.
Optimization: QLoRA or other parameter-efficient fine-tuning (LoRA rank, learning rate scheduling, dropout, weight decay as specified).
Compensation effect: SFT consistently restores a significant fraction of lost performance, e.g., 74.25% Judge-LLM score post-SFT vs. 62.5% post-prune for 6-layer-pruned LLAVA (Khan et al., 29 Jul 2025).

Variants such as hard and semi-soft SFT differ in whether the backward gradient traverses zeroed weights; hard SFT (mask fixed during backprop) is most robust for high sparsity (Yu et al., 2020).

4. Post-Training Quantization Techniques

Quantization compresses the numerical representation of weights and/or activations to low bit-widths. This is generally applied post-SFT to all remaining nonzero weights.

Methods

Activation-Aware Quantization (AWQ): (Khan et al., 29 Jul 2025)
- Per-channel INT4, group size=64; activations in FP16.
- "Salient" channels protected by scaling; minimize $L(s) = \|Q(W\,\textrm{diag}(s))\,\textrm{diag}(s)^{-1}\,X - W X\|_2$ .
- $s$ parameterized via mean activation and exponentiated by a search parameter $\alpha^*$ .
Incremental Quantization to Powers-of-Two: (Yu et al., 2020)
- Surviving weights quantized as $w_i \leftarrow \mathrm{sign}(w_i) 2^{\operatorname{round}(\log_2 |w_i|)}$ .
- Incremental partitioning and step-wise quantization based on Taylor Score, continuing to prune during retraining.
Hardware-Aware Fixed-Point Quantization: (Niemi et al., 27 Mar 2026)
- Implements per-tensor, per-channel, and per-weight quantization at 8-bit or mixed-precision.
- HGQ (High-Granularity Quantization) variants allow learnable bit-widths via loss regularization.

Path-Following Quantization

Stochastic drift-corrected quantization, as formalized in (Zhang et al., 2024), uses unbiased stochastic quantizers (e.g., 1-bit, multi-bit), with error bounds controlled via the scale parameter $C$ . The error after quant plus prune is:

$g_i$ 0

with high probability.

5. Empirical Performance and Trade-Offs

The effectiveness of Prune-SFT-Quantize is evidenced by detailed ablation studies and hardware-centric metrics.

Method	Params (B)	Bit-width	Score (%)	Latency (ms/token)	VRAM (GB)
Original LLaVA	7.06	16	80.12	154	13.4
Pruning Only	5.85	16	62.50	148	11.2
Prune + SFT	5.85	16	74.25	146	11.2
Quantization Only	7.06	4	64.72	146	4.5
Prune + Quant	5.85	4	25.00	122	3.9
Prune + SFT + Quant (Ours)	5.85	4	54.25	122	3.9

SFT after pruning recovers significant accuracy compared to pruning only; AWQ outperforms vanilla quantization methods by up to 7.25% (Khan et al., 29 Jul 2025). Hardware-optimized pipelines in PQuantML achieve up to 88% DSP usage reduction and maintain II=1 pipelines on FPGAs (Niemi et al., 27 Mar 2026). For vision models, >98% sparsity and 3–5 bit quantization yields <2% drop in performance (Yu et al., 2020).

6. Extensions, Generalizations, and Limitations

Prune-SFT-Quantize is generalizable to a wide range of architectures including multimodal LLMs, ResNet-family vision models, and custom hardware-mapped pipelines (Khan et al., 29 Jul 2025, Yu et al., 2020, Niemi et al., 27 Mar 2026). The method is implemented in open-source frameworks such as PQuantML with YAML-based configuration and multiple pruning/quantization schemes.

Noted Limitations

Coarse-grained block pruning may disrupt critical inter-block dependencies in certain architectures (Khan et al., 29 Jul 2025).
The sequential (non-joint) execution of pruning, SFT, and quantization is sub-optimal when deeper interactions exist between these stages; no joint optimization is considered, potentially missing better minima.
Fixed global scaling parameters (e.g., α in AWQ) may be suboptimal compared to per-layer tuning.
Very aggressive (ultra-high) sparsity may impair recoverability even with SFT.

A plausible implication is that more sophisticated joint pruning-quantization algorithms and mixed-precision schemes could enhance both compression and performance retention.

7. Practical Implementation Guidance

Implementations require the following workflow elements:

Rigorous calibration data for pruning scores and quantization scale determination, repurposing small task-specific or domain-specific samples as in (Khan et al., 29 Jul 2025).
Strictly sequential application of the three stages; joint methods remain an open avenue.
Hyperparameter selection, such as pruning thresholds, SFT epochs, quantization group sizes, and precision targets, optimized for the target hardware and task.
For hardware-aware synthesis, structured pruning and quantization schemes aligned to accelerator capabilities (e.g., channel/block pruning benefiting DSP and LUT utilization).

For reproducibility, full pseudocode, hyperparameters, error bounds, and empirical tables are available in the cited works (Khan et al., 29 Jul 2025, Yu et al., 2020, Zhang et al., 2024, Niemi et al., 27 Mar 2026).