Progressive Fine-Tuning Framework

• Progressive Fine-Tuning Framework is a staged adaptation process that divides fine-tuning into ordered phases to redistribute optimization effort across parameters, supervision, data, and objectives.
• It systematically controls which model parameters are trainable over time, reducing computational overhead while enhancing stability and generalization.
• Various schemes, including coarse-to-fine training, progressive prompt tuning, and curriculum-based data selection, demonstrate its versatility across diverse deep learning tasks.

Searching arXiv for the cited progressive fine-tuning papers to ground the article. Progressive fine-tuning frameworks are staged adaptation procedures in which a pretrained model is not optimized under a single stationary recipe, but under an ordered sequence of training phases that progressively modify trainable parameters, supervision signals, data difficulty, or target representations. In the cited literature, this family includes progressive training in generalized coarse-to-fine visual recognition, progressive prompt tuning by expansion of partial PLMs, stratified and adaptive parameter selection for PEFT, staged curricula for reinforcement learning, and multi-level personalization or multimodal adaptation pipelines (Ren et al., 2018, Huang et al., 2022, Arora et al., 2024, Zhao et al., 8 Jan 2026, Zhang et al., 3 Mar 2025, Rostami et al., 8 Jun 2026, Ji et al., 26 Jun 2025). Across these formulations, the common premise is that uniform end-to-end fine-tuning allocates capacity and supervision too coarsely, whereas ordered adaptation can improve efficiency, stability, or generalization.

1. Conceptual basis and historical lineage

A progressive fine-tuning framework typically decomposes adaptation into phases with different optimization conditions. In the generalized coarse-to-fine setting, a coarse network first predicts a rough output $y^C$, a fine network consumes the original input concatenated with an encoded hint tensor $z=g(y^C)$, and training gradually replaces ground-truth hints with model-generated hints through a mixing variable $t(i)$ (Ren et al., 2018). In Fast Prompt Tuning, prompt tuning begins on a small partial PLM, then progressively expands depth and width until the full model size, recycling learned soft prompts at each stage (Huang et al., 2022). In Gradual Fine-Tuning for flow matching, the same high-level idea appears as a temperature-controlled sequence of intermediate objectives between pretrained and target drifts, rather than as a change in architecture or input format (Thorkelsdottir et al., 30 Jan 2026).

These early and later formulations already show that “progressive” does not denote a single algorithmic primitive. In some works, progression is a curriculum over input reliability; in others, it is a schedule over model scale, active layers, or probability-path objectives. This suggests that the defining property is not the specific mechanism, but the imposition of an ordered adaptation path between pretrained behavior and task-specialized behavior.

2. Progression in trainable parameters and model capacity

One major line of work uses progression to control which parameters remain trainable over time. SPAFIT partitions Transformer layers into three strata, freezes Group 1 entirely, applies BitFit to Group 2, and applies LoRA plus BitFit to Group 3. Its update order is itself progressive: Stage 1 updates only Group 2 bias terms, and Stage 2 continues Group 2 biases while unfreezing Group 3 (Arora et al., 2024). On GLUE with BERT-large-cased, SPAFIT-4-9-II uses 7.5 M tunable parameters, matches or exceeds Full LoRA-II on 6/9 tasks, and SPAFIT-4-9-I uses 5.7 M tunable parameters while remaining the most parameter-efficient variant (Arora et al., 2024).

Progtuning applies a different schedule: it progressively reduces the number of updated Transformer blocks based on contribution, retaining embeddings and head while shrinking the active block set epoch by epoch. On GLUE with 3 epochs, the paper reports that full fine-tuning performs $3\times110\text{M}=330$ M update-events, whereas Progtuning performs 220 M, or about 33% fewer, while BERT_base improves from 82.6 to 82.8 average GLUE and RoBERTa_base from 82.8 to 83.5 (Ji et al., 26 Jun 2025). The same work reports that Progtuning can be combined with Adapter-tuning, BitFit, and LoRA, reducing updates further while largely preserving performance (Ji et al., 26 Jun 2025).

FisherAdapTune replaces architectural heuristics with a task-aware freezing rule derived from Fisher geometry drift. At step $t$, it measures the Jensen–Shannon distance between consecutive Fisher distributions for each parameter group, smooths the drift, and freezes groups whose running mean drift falls below a threshold $\tau=\mu+\lambda\sigma$ (Rostami et al., 8 Jun 2026). Empirically, SAM2-Tiny uses 6.95 M average parameters versus 38.9 M for full fine-tuning and recovers F1 = 63.12 versus 64.67, while SAM2-Large improves zero-shot IoU average from 53.47 to 57.27 relative to full fine-tuning (Rostami et al., 8 Jun 2026).

FACT organizes capacity escalation in the opposite direction. Phase I trains only the task head on frozen features with Frozen Feature Augmentation; Phase II introduces LoRA adapters while the backbone remains otherwise frozen; Phase III unfreezes all parameters but regularizes deviation from initialization with $\lambda\|\theta-\theta_0\|_2^2$ (Xu et al., 1 Jun 2026). Under low sampling ratios with a ViT-Small backbone, FACT reports 87.1 accuracy on CIFAR10 at 0.1%, 61.1 on CIFAR100 at 1%, and 57.6 on ImageNet-1k at 0.5%, exceeding Full-FT by over 20 points in several ViT low-sampling settings (Xu et al., 1 Jun 2026).

SaRA provides a sparse counterpart to these schedules. It selects approximately the smallest 10–20% of pretrained parameters by absolute value, trains only those entries for half the budget, reselection retains only the subset that remains below threshold at midpoint, and Stage 2 trains only that smaller sub-mask (Hu et al., 2024). Combined with a nuclear-norm penalty and “Unstructural Backpropagation,” the method reduces gradient storage from $O(n\cdot m)$ to $O(\|M\|_0)$ and reports 40–52% GPU-memory reduction and up to about 50% wall-clock reduction versus LoRA and selective PEFT on SD 2.0 benchmarks (Hu et al., 2024).

3. Progression in supervision, targets, and optimization objectives

A second line of work makes the training target progressively more demanding or more implicit. Progressive Thought Refinement constructs triples of query, thought-sequence, and refined answer by sampling thought attempts from a weak model $\pi_{\text{weak}}$, generating a refined answer with a strong model $z=g(y^C)$0, and filtering examples with a consistency score $z=g(y^C)$1 (Du et al., 2024). Fine-tuning then masks thought tokens and optimizes a weighted objective

$z=g(y^C)$2

so that masking prevents trivial copying, the $z=g(y^C)$3 term enforces logical coherence, and the $z=g(y^C)$4 term encourages growing confidence (Du et al., 2024). On Qwen2-7B, average accuracy rises from 49.6% to 53.5% by iteration 3, with MMLU improving from 57.1% to 64.1% and ARC from 60.6% to 65.2%, all without task-specific fine-tuning (Du et al., 2024).

Gradual Fine-Tuning for flow matching uses progression in path space. Its marginal objective is

$z=g(y^C)$5

with optimal drift

$z=g(y^C)$6

As $z=g(y^C)$7, the optimum approaches the target drift; as $z=g(y^C)$8, it approaches the pretrained drift (Thorkelsdottir et al., 30 Jan 2026). On WILDS fine-tuning experiments, GFT reports dramatically lower instantaneous variance in FID, monotonic improvement with Spearman’s $z=g(y^C)$9, and 20–40% shorter average probability path lengths while maintaining generation quality comparable to standard fine-tuning (Thorkelsdottir et al., 30 Jan 2026).

Physics-based QSM fine-tuning illustrates a domain-specific variant in which progression is architectural and self-supervised. Progressive Unet stacks $t(i)$0 identical Unet modules in series, pretrains all outputs with an $t(i)$1 loss, and at test time fine-tunes only the last Unet using the forward-model fidelity loss

$t(i)$2

The preceding three stages remain frozen, so update cost remains that of tuning only the last module (Zhang et al., 2023). Retrospective and prospective studies show improved robustness to unseen voxel sizes and unknown high-pass filter parameters, with Progressive Unet slightly outperforming Unet before and after fine-tuning (Zhang et al., 2023).

4. Progression in data curricula, weakness discovery, and safety filtering

A third axis of progression changes which data the model sees, and in what order. ThinkDrive begins with supervised fine-tuning on Chain-of-Thought explanations and then applies progressive reinforcement learning with a Gaussian curriculum over Easy, Medium, and Hard samples, where difficulty is estimated from confidence and correctness, and fine-grained difficulty is quantified by rollout entropy (Zhao et al., 8 Jan 2026). The adaptive advantage rescales returns by $t(i)$3, and the RL objective uses geometric-mean aggregation with asymmetric clipping (Zhao et al., 8 Jan 2026). On DrivingVQA with Qwen3-VL-2B, ThinkDrive reports 62.38 exam, 63.97 easy-exam, and 77.02 accuracy, improving over GMPO by +1.45%, +1.95%, and +1.01%, respectively, and surpassing GPT-4o by +3.28% on the exam metric (Zhao et al., 8 Jan 2026).

TableDreamer uses progression for data synthesis rather than policy optimization. After generating seed tables and seed instruction triples, it repeatedly evolves each sample by instruction complication, instruction generalization, or table generalization; evaluates the current student on the expanded set; identifies weakness examples with a 1–5 Likert judge where Score $t(i)$4 denotes weakness; and carries only those weakness examples into the next round (Zheng et al., 10 Jun 2025). With 27,083 GPT-4o synthetic triples, Llama3.1-8B-instruct improves from 49.07% to 60.69% average accuracy on 10 tabular benchmarks, outperforming baselines that use more training data (Zheng et al., 10 Jun 2025).

TOSS-Pro applies the same logic to safety-preserving fine-tuning at token level. TOSS scores each token by the difference between next-token losses from a utility-oriented model and a safety-degraded model, masks the top $t(i)$5 globally ranked risky tokens, and fine-tunes selectively on the remaining tokens (Li et al., 1 Mar 2026). TOSS-Pro then progressively strengthens the safety-degraded reference model by mining the custom dataset for the top-scoring unsafe samples, expanding the harmful pool, and repeating the LoRA fine-tuning of the safety expert for $t(i)$6 rounds with about $t(i)$7 new harmful samples per round (Li et al., 1 Mar 2026). On Llama-3-8B, it raises average win rate from 81.6 for TOSS to 83.8, reduces ANTH HH ASR from 78.9% to 76.6%, and reduces HEx-PHI ASR from 54.5% to 43.6% (Li et al., 1 Mar 2026).

These results indicate that progressive fine-tuning can operate not only on model parameters, but also on the training corpus itself, by repeatedly estimating the model’s current failure modes and reallocating data budget toward hard or unsafe regions of the input space.

5. Hierarchical and domain-specific staged adaptation

Several frameworks use progression to bridge representation levels that are semantically distinct. CLiFT-ASR begins from a Mandarin HuBERT-base-cmn encoder with an RNN-Transducer backbone, first trains for 20 epochs on tone-marked Tai-lo romanization to learn acoustic and tonal representations, then continues for 40 epochs on Han-character transcriptions to capture vocabulary and syntax (Sung et al., 10 Nov 2025). On TAT-MOE, the two-stage model attains 20.94% test CER, a 24.88% relative reduction versus HuBERT-cmn direct fine-tuning at 22.41%, and ablations show that freezing any single module in Stage 1 degrades performance (Sung et al., 10 Nov 2025).

PROPER decomposes personalization into population-level, group-level, and user-level stages. Stage 1 learns a population LoRA residual, Stage 2 introduces group-level LoRA-MoE experts with a user-aware router and diversity loss, and Stage 3 adds user-specific LoRAs plus a LoRA-aware router that selectively mixes group LoRAs into the user model (Zhang et al., 3 Mar 2025). On the LaMP benchmark with LLaMA-2-7B, Stage 3 yields an average relative improvement of +5.47% over the best prior PEFT baseline, and on the most data-sparse Movie-Tagging task it reports +24.5% accuracy and +35.1% F1 (Zhang et al., 3 Mar 2025).

STARE-VLA organizes long-horizon robotic adaptation as an Imitation $t(i)$8 Preference $t(i)$9 Interaction pipeline. Supervised fine-tuning initializes the policy from demonstrations, Stage-Aware TPO then performs offline stage-level preference optimization, and Stage-Aware PPO finally uses online interaction with stage-aligned reward shaping (Xu et al., 4 Dec 2025). On SimplerEnv, IPI reaches 98.0% average success; on ManiSkill3, it reaches 96.4%, outperforming SFT, GRAPE, and partial stage-aware variants (Xu et al., 4 Dec 2025).

ChatReID provides a hierarchical progressive tuning strategy for vision-language person retrieval. Stage 1 performs incremental pre-training for person attribute understanding on about 5 M image-caption pairs, Stage 2 performs multi-task joint training for fine-grained image and text retrieval, and Stage 3 freezes the vision encoder and instruction-tunes the decoder for open-ended VQA-style retrieval (Niu et al., 27 Feb 2025). Across ten benchmarks, the reported average mAP progresses from 41.0 for Stage 3 only, to 73.5 for Stage 2 + Stage 3, to 83.0 for the full Stage 1 + Stage 2 + Stage 3 pipeline (Niu et al., 27 Feb 2025).

These domain-specific systems show that progression is often used to move between representational regimes that are difficult to align in a single step: phonetics to orthography, population preference to individual preference, imitation to interaction, or attribute recognition to multimodal retrieval reasoning.

6. Empirical regularities, misconceptions, and limitations

The literature reports three recurrent empirical patterns. First, progressive fine-tuning often improves adaptation under scarcity or distribution shift. FACT reports large gains under low sampling ratios, especially on ViT models (Xu et al., 1 Jun 2026); GFT improves convergence stability under target-distribution shift (Thorkelsdottir et al., 30 Jan 2026); and physics-based QSM fine-tuning improves robustness when voxel size or filter parameters deviate from training (Zhang et al., 2023). Second, progression can reduce parameter or update budgets rather than increase them, as in SPAFIT, Progtuning, FisherAdapTune, and SaRA (Arora et al., 2024, Ji et al., 26 Jun 2025, Rostami et al., 8 Jun 2026, Hu et al., 2024). Third, some frameworks claim benefits beyond conventional task accuracy: PTR reports more detailed, well-structured, and coherent open-ended responses; TOSS-Pro improves the safety–utility trade-off; and STARE-VLA improves credit assignment in long-horizon control (Du et al., 2024, Li et al., 1 Mar 2026, Xu et al., 4 Dec 2025).

A common misconception is that progressive fine-tuning is equivalent to progressive unfreezing. The surveyed methods show a broader design space: Progtuning and FisherAdapTune progressively reduce the active parameter set; FACT progressively increases trainable capacity; PTR progressively refines thought-answer chains; TableDreamer and TOSS-Pro progressively reshape the training data; and GFT progressively alters the objective via $3\times110\text{M}=330$0-annealed path-space regularization (Ji et al., 26 Jun 2025, Rostami et al., 8 Jun 2026, Xu et al., 1 Jun 2026, Du et al., 2024, Zheng et al., 10 Jun 2025, Li et al., 1 Mar 2026, Thorkelsdottir et al., 30 Jan 2026). A plausible implication is that “progressive” refers most fundamentally to staged control of adaptation pressure, not to any single freeze–unfreeze schedule.

The limitations are likewise heterogeneous. SPAFIT still shows its greatest gap versus full fine-tuning on CoLA and SST-2, suggesting that early or middle adaptation may remain under-allocated in some sentence-level tasks (Arora et al., 2024). In ThinkDrive, reward initially dips when the curriculum shifts to harder data, although it later stabilizes (Zhao et al., 8 Jan 2026). In FisherAdapTune, $3\times110\text{M}=330$1 creates an efficiency–generalization trade-off: overly small $3\times110\text{M}=330$2 freezes prematurely, whereas overly large $3\times110\text{M}=330$3 delays freezing and may over-specialize (Rostami et al., 8 Jun 2026). STARE relies on hand-crafted geometric thresholds for stage separation and may not generalize to highly unstructured domains (Xu et al., 4 Dec 2025). In physics-based QSM, fine-tuning cannot recover performance at the extreme cutoff setting $3\times110\text{M}=330$4 because of excessive data loss (Zhang et al., 2023).

Taken together, the literature supports a broad but technically coherent view of progressive fine-tuning: a framework class in which adaptation is decomposed into ordered phases that redistribute optimization effort across parameters, supervision, data, or semantic levels. This suggests that the main scientific contribution of progressive fine-tuning is not merely computational economy, but a finer-grained control over when and where a pretrained model is allowed to change.