Progressive and Self-Tuning Training (PaST)

• PaST is a design principle that structures training into progressive stages while dynamically adapting supervision using criteria like loss, confidence, and sample quality.
• It integrates methods such as easy-to-hard sample inclusion, target evolution, and model expansion to address varied challenges in tracking, imaging, and more.
• Empirical studies reveal that PaST enhances performance and stability, though many implementations still depend on hand-tuned schedules and preset stage configurations.

Progressive and Self-Tuning Training Strategy (PaST) designates a family of training procedures in which optimization is organized into explicit stages or curricula and the effective supervision is adjusted during training by quantities such as loss, confidence, time, equilibrium, teacher outputs, contribution scores, or pseudo-label quality. Across the literature considered here, PaST is not a single canonical algorithm but a recurring design pattern. It appears in progressive multi-stage sample selection for discriminative tracking (Li et al., 2020), target-space continuation with equilibrium constraints in neural classification (Dabounou, 2024), progressive self-paced distillation in brain imaging (Yang et al., 2024), and related staged schemes for federated personalization, continual learning, prompt tuning, reinforcement learning, remote sensing downscaling, and multi-agent language systems (Wang et al., 2024, Du et al., 2019, Huang et al., 2022, Zhang et al., 2021, Shen et al., 31 Mar 2026, Bijoy et al., 2 Sep 2025).

1. Definition, scope, and historical articulation

Several papers instantiate the PaST pattern under different names, and the exact acronym varies across subfields. "Single-Net Continual Learning with Progressive Segmented Training" formalized a single-network continual-learning regime in which parameters are progressively frozen according to data-driven importance scores, while a fixed-size memory stabilizes rehearsal (Du et al., 2019). "Self-training with progressive augmentation for unsupervised cross-domain person re-identification" introduced a two-stage self-training loop with alternating conservative and promoting stages (Zhang et al., 2019). "Progressive Multi-Stage Learning for Discriminative Tracking" supplied a particularly explicit blueprint: a joint discriminative learning scheme with progressive multi-stage optimization, time-weighted self-paced regularization, and detection-guided confidence integration (Li et al., 2020).

A broader taxonomy is suggested by later work. In some papers, progression acts on samples or pseudo-labels; in others, on targets, subtasks, trust regions, model capacity, or adapter activation. Likewise, self-tuning may mean closed-form sample reweighting, equilibrium-based update gating, logits calibration, dynamic distillation weights, contribution-based reward allocation, or confidence-based trust filtering. This suggests that PaST is best understood as a design principle rather than a single method family.

Representative work	Progressive element	Self-tuning element
"Single-Net Continual Learning with Progressive Segmented Training" (Du et al., 2019)	Progressive segmentation and freezing of important parameters	Importance scores from Taylor-based criteria
"Self-training with progressive augmentation for unsupervised cross-domain person re-identification" (Zhang et al., 2019)	Alternating conservative and promoting stages	Pseudo-labels and ranking-based triplets derived from current features
"Progressive Multi-Stage Learning for Discriminative Tracking" (Li et al., 2020)	Multi-stage easy-to-hard sample inclusion	Loss-, time-, and confidence-based weights
"Adaptive Class Emergence Training" (Dabounou, 2024)	Target evolution from uniform to one-hot	Equilibrium threshold controlling updates
"Advancing Brain Imaging Analysis Step-by-step via Progressive Self-paced Learning" (Yang et al., 2024)	Progressive pacing over epochs	Student- and teacher-loss-based sample weights
"Fast Prompt Tuning" (Huang et al., 2022)	Progressive expansion from partial PLMs to the full PLM	Reuse of learned soft prompts across stages

A common misconception is to equate PaST with curriculum learning alone. The literature shows a wider scope: curriculum is only one axis. PaST may instead progress through label sharpening, model expansion, scale refinement, or trust-region growth, while self-tuning may operate through weighting, calibration, or pseudo-label filtering rather than explicit reordering of examples.

2. Core mathematical motifs

A representative PaST formulation couples model parameters with adaptive sample weights. In discriminative tracking, the stage-$n$ optimization problem is

$$\min_{\theta_n, v^n \in [0,1]^t} \sum_{k=1}^{t} v_k^n L\big(g(x_k,\theta_n),y_k\big) + \alpha R(\theta_n) + f(v^n;\lambda_n), \qquad \lambda_1 < \lambda_2 < \cdots < \lambda_N,$$

so the curriculum variable is the learning pace $\lambda_n$, while $v_k^n$ is optimized jointly with the tracker (Li et al., 2020). With time and detection guidance, the resulting closed form becomes

$v_k^n = \begin{cases} \rho_k - \dfrac{l_k+\xi c_k}{\lambda_n}\rho_k, & l_k+\xi c_k < \lambda_n,\[4pt] 0, & l_k+\xi c_k \ge \lambda_n, \end{cases}$

which makes progression and adaptation explicit: harder or less trustworthy samples are admitted only when the pace increases enough.

A different formal motif appears in Adaptive Class Emergence Training, where progression is placed in target space rather than sample space. The intermediate target is

$$y_c(t) = t\, y_c^* + (1-t)\frac{1}{n_{\text{classes}}}\mathbf{1},$$

so labels evolve continuously from a uniform distribution to a one-hot vector. The self-regulating component is an equilibrium condition,

$$E(t)=\left\{\theta \in \Theta \mid L(f(x;\theta), y_c(t)) \le \varepsilon \right\},$$

which conceptually gates updates by whether the current model is sufficiently close to the current target (Dabounou, 2024).

A third motif appears in parameter-efficient tuning. In CoTo, LoRA adapters are gated by Bernoulli variables with a time-dependent activation probability

$p(t)= \begin{cases} \dfrac{4t}{3T}, & t < \dfrac{3T}{4},\[4pt] 1, & t \ge \dfrac{3T}{4}. \end{cases}$

Here the progressive variable is adapter participation, not sample difficulty, and the implied adaptation occurs through stochastic subnetwork training that balances layer-wise contribution (Zhuang et al., 6 Jun 2025).

These formulations do not reduce to one universal objective. Rather, they define a recurrent structure: a monotone stage variable, a state-dependent control term, and repeated optimization under a gradually tightened or expanded training regime.

3. Modes of progression

One major mode of progression is easy-to-hard sample inclusion. In tracking, stages begin with low-loss, high-confidence samples and enlarge the admissible set as $\lambda_n$ increases geometrically (Li et al., 2020). In Progressive Self-Paced Distillation, two pace parameters, $\lambda_w = \lambda_{w,0} + \alpha_w t$ and $$\min_{\theta_n, v^n \in [0,1]^t} \sum_{k=1}^{t} v_k^n L\big(g(x_k,\theta_n),y_k\big) + \alpha R(\theta_n) + f(v^n;\lambda_n), \qquad \lambda_1 < \lambda_2 < \cdots < \lambda_N,$$0, expand the set of samples used for label-based learning and distillation across epochs (Yang et al., 2024). In competence-progressive few-shot node classification, the curriculum is induced through increasing DropEdge ratios governed by a competence function $$\min_{\theta_n, v^n \in [0,1]^t} \sum_{k=1}^{t} v_k^n L\big(g(x_k,\theta_n),y_k\big) + \alpha R(\theta_n) + f(v^n;\lambda_n), \qquad \lambda_1 < \lambda_2 < \cdots < \lambda_N,$$1, so the graph becomes progressively harder by creating more tail-node conditions (Yan et al., 2024).

A second mode is progression in target or supervision space. ACET moves labels from uniform to one-hot, making the loss gradually sharper (Dabounou, 2024). CURVETE orders pseudo-label granularity from many fine-grained sub-classes to coarser class structure, using the granularity sequence $$\min_{\theta_n, v^n \in [0,1]^t} \sum_{k=1}^{t} v_k^n L\big(g(x_k,\theta_n),y_k\big) + \alpha R(\theta_n) + f(v^n;\lambda_n), \qquad \lambda_1 < \lambda_2 < \cdots < \lambda_N,$$2 in both self-supervised pretext training and downstream class decomposition (Abbas et al., 27 Oct 2025). In FedPSD, the teacher signal itself is progressive: previous-epoch outputs serve as virtual teachers and historical personalized outputs seed the first local epoch of each communication round (Wang et al., 2024).

A third mode is progression in model structure or active capacity. FPT starts prompt tuning with a small partial PLM and progressively expands depth and width while reusing the learned soft prompt at each stage (Huang et al., 2022). CoTo begins with low adapter activation probability and ends with fully active LoRA, turning a stochastic subnetwork phase into standard full-adapter fine-tuning in the last quarter of training (Zhuang et al., 6 Jun 2025). PST in continual learning progressively reduces the free parameter subset by freezing important parameters after each task (Du et al., 2019).

A fourth mode is progression in environment, scale, or subtask frontier. Close-up-GS expands trust regions in rounds from about $$\min_{\theta_n, v^n \in [0,1]^t} \sum_{k=1}^{t} v_k^n L\big(g(x_k,\theta_n),y_k\big) + \alpha R(\theta_n) + f(v^n;\lambda_n), \qquad \lambda_1 < \lambda_2 < \cdots < \lambda_N,$$3 to $$\min_{\theta_n, v^n \in [0,1]^t} \sum_{k=1}^{t} v_k^n L\big(g(x_k,\theta_n),y_k\big) + \alpha R(\theta_n) + f(v^n;\lambda_n), \qquad \lambda_1 < \lambda_2 < \cdots < \lambda_N,$$4 to $$\min_{\theta_n, v^n \in [0,1]^t} \sum_{k=1}^{t} v_k^n L\big(g(x_k,\theta_n),y_k\big) + \alpha R(\theta_n) + f(v^n;\lambda_n), \qquad \lambda_1 < \lambda_2 < \cdots < \lambda_N,$$5 closer viewpoints, using self-generated data and reliable pixels to refine the model at each frontier (Xia et al., 12 Mar 2025). The Landsat LST downscaling framework performs one pre-training stage and two fine-tuning stages, moving from $$\min_{\theta_n, v^n \in [0,1]^t} \sum_{k=1}^{t} v_k^n L\big(g(x_k,\theta_n),y_k\big) + \alpha R(\theta_n) + f(v^n;\lambda_n), \qquad \lambda_1 < \lambda_2 < \cdots < \lambda_N,$$6 via pseudo-labels (Shen et al., 31 Mar 2026). ProST introduces subtasks progressively across epochs rather than training on full long trajectories from the start (Bijoy et al., 2 Sep 2025).

4. Mechanisms of self-tuning

The most explicit self-tuning mechanism is closed-form sample reweighting. In discriminative tracking, weights depend jointly on sample loss $$\min_{\theta_n, v^n \in [0,1]^t} \sum_{k=1}^{t} v_k^n L\big(g(x_k,\theta_n),y_k\big) + \alpha R(\theta_n) + f(v^n;\lambda_n), \qquad \lambda_1 < \lambda_2 < \cdots < \lambda_N,$$7, temporal importance $$\min_{\theta_n, v^n \in [0,1]^t} \sum_{k=1}^{t} v_k^n L\big(g(x_k,\theta_n),y_k\big) + \alpha R(\theta_n) + f(v^n;\lambda_n), \qquad \lambda_1 < \lambda_2 < \cdots < \lambda_N,$$8, and detection confidence $$\min_{\theta_n, v^n \in [0,1]^t} \sum_{k=1}^{t} v_k^n L\big(g(x_k,\theta_n),y_k\big) + \alpha R(\theta_n) + f(v^n;\lambda_n), \qquad \lambda_1 < \lambda_2 < \cdots < \lambda_N,$$9, with older samples decayed through $\lambda_n$0 and ambiguous detections down-weighted through a peak-ratio-based confidence term (Li et al., 2020). This turns sample management into an optimization variable rather than a fixed heuristic.

A second mechanism is teacher- or history-conditioned adaptation. PSPD computes $\lambda_n$1 from the student’s cross-entropy $\lambda_n$2 and $\lambda_n$3 from the teacher’s cross-entropy $\lambda_n$4, using the previous epoch model as teacher and separate pacing for label learning and distillation (Yang et al., 2024). FedPSD calibrates local logits using client priors and constructs historical fusion labels

$\lambda_n$5

so the initial local epoch in each round recalls personalized knowledge while subsequent epochs inherit previous-epoch predictions through progressive self-distillation (Wang et al., 2024).

A third mechanism is equilibrium or confidence gating. ACET treats the set $\lambda_n$6 as a dynamic equilibrium region; updates are conceptually triggered when the system is out of balance with the current progressive targets (Dabounou, 2024). ThinkDrive modulates RL update magnitude with entropy over multi-rollout decisions, using

$\lambda_n$7

so uncertain, high-entropy samples receive attenuated policy updates (Zhao et al., 8 Jan 2026).

A fourth mechanism is reward- and trust-based reallocation. In Volt-Var control, the cooperative stage allocates a shared system-level reward by the contribution factor

$\lambda_n$8

while action cost and a do-nothing reward shape agent behavior (Zhang et al., 2021). In Close-up-GS, reliable pixels, view-selection optimization, and trust-region expansion determine which self-generated targets can safely supervise the next round of fine-tuning (Xia et al., 12 Mar 2025). In LDCT denoising, repeated masking and combination-noise injection make the effective input distribution progressively cleaner, which the paper interprets as an implicit self-adjusting process even though the schedule itself is fixed (Liu et al., 20 Jan 2026).

Taken together, these mechanisms show that “self-tuning” often means partial adaptivity rather than fully autonomous schedule discovery. Many systems adapt weights or supervision online, while still relying on hand-specified stage counts, pace multipliers, or thresholds.

5. Representative instantiations across domains

Domain and work	Progressive component	Self-tuning component
Visual tracking, "Progressive Multi-Stage Learning for Discriminative Tracking" (Li et al., 2020)	Multi-stage easy-to-hard sample inclusion with $\lambda_n$9	Closed-form weighting from loss, time, and detection confidence
Brain imaging, "Progressive Self-Paced Distillation" (Yang et al., 2024)	Increasing $v_k^n$0 over epochs	Student- and teacher-loss-based sample and distillation weights
Prompt tuning, "Fast Prompt Tuning" (Huang et al., 2022)	Expansion from small partial PLMs to the full PLM	Prompt recycling across stages
LoRA fine-tuning, "CoTo" (Zhuang et al., 6 Jun 2025)	Progressive adapter activation probability	Implicit balancing of layer-wise contributions through stochastic gating
Autonomous driving RL, "ThinkDrive" (Zhao et al., 8 Jan 2026)	Two-stage SFT$v_k^n$1RL and Gaussian curriculum over easy/medium/hard samples	Entropy-based advantage scaling
Remote sensing downscaling, "30-meter Land Surface Temperature from Landsat via Progressive Self-Training Downscaling" (Shen et al., 31 Mar 2026)	One pre-training and two fine-tuning stages from coarse to fine resolution	Pseudo-label refinement plus data-consistency and high-frequency losses

These instantiations differ in what is being progressed: samples, targets, subnetworks, subtasks, camera poses, or resolutions. They also differ in what is self-tuned: sample weights, teacher influence, logits, rewards, trust regions, or prompt initialization. A plausible implication is that PaST is best characterized by the coupling of these two dimensions rather than by any fixed optimization primitive.

The same pattern also appears in further specialized settings. CURVETE uses granularity-based anti-curriculum and class decomposition to address irregular medical-image class distributions (Abbas et al., 27 Oct 2025). MPFNet trains a Generic Feature Encoder, then an Advanced Feature Encoder on a motion-amplified balanced dataset, then fuses both priors in a meta-learning stage for micro-expression recognition (Ma et al., 11 Jun 2025). CPT defines competence-progressive task hardness for few-shot node classification (Yan et al., 2024). ProST progressively reveals subtasks inside agentic trajectories for multi-agent SLM systems (Bijoy et al., 2 Sep 2025). PST and PAST provide earlier continual-learning and unsupervised Re-ID formulations in which progression occurs through parameter segmentation or alternation of local-structure and global-cluster stages (Du et al., 2019, Zhang et al., 2019).

6. Empirical evidence, limitations, and open questions

The literature reports consistent empirical gains, but the magnitude and interpretation of those gains are domain-specific.

Work	Setting	Reported outcome
"Progressive Multi-Stage Learning for Discriminative Tracking" (Li et al., 2020)	OTB-2015	SRDCF $v_k^n$2 PS/AUC $v_k^n$3 SPL-DCI $v_k^n$4; $v_k^n$5 PS and $v_k^n$6 AUC
"Adaptive Class Emergence Training" (Dabounou, 2024)	MNIST; melanoma	MNIST $v_k^n$7 vs typical $v_k^n$8; melanoma $v_k^n$9 vs $v_k^n = \begin{cases} \rho_k - \dfrac{l_k+\xi c_k}{\lambda_n}\rho_k, & l_k+\xi c_k < \lambda_n,\[4pt] 0, & l_k+\xi c_k \ge \lambda_n, \end{cases}$0
"Progressive Self-Paced Distillation" (Yang et al., 2024)	ADNI, ResNet-18/50/101	Accuracy improvements over baseline: $v_k^n = \begin{cases} \rho_k - \dfrac{l_k+\xi c_k}{\lambda_n}\rho_k, & l_k+\xi c_k < \lambda_n,\[4pt] 0, & l_k+\xi c_k \ge \lambda_n, \end{cases}$1, $v_k^n = \begin{cases} \rho_k - \dfrac{l_k+\xi c_k}{\lambda_n}\rho_k, & l_k+\xi c_k < \lambda_n,\[4pt] 0, & l_k+\xi c_k \ge \lambda_n, \end{cases}$2, $v_k^n = \begin{cases} \rho_k - \dfrac{l_k+\xi c_k}{\lambda_n}\rho_k, & l_k+\xi c_k < \lambda_n,\[4pt] 0, & l_k+\xi c_k \ge \lambda_n, \end{cases}$3
"Progressive self-supervised blind-spot denoising" (Liu et al., 20 Jan 2026)	Mayo LDCT, U-Net backbone	PSNR $v_k^n = \begin{cases} \rho_k - \dfrac{l_k+\xi c_k}{\lambda_n}\rho_k, & l_k+\xi c_k < \lambda_n,\[4pt] 0, & l_k+\xi c_k \ge \lambda_n, \end{cases}$4, SSIM $v_k^n = \begin{cases} \rho_k - \dfrac{l_k+\xi c_k}{\lambda_n}\rho_k, & l_k+\xi c_k < \lambda_n,\[4pt] 0, & l_k+\xi c_k \ge \lambda_n, \end{cases}$5, RMSE $v_k^n = \begin{cases} \rho_k - \dfrac{l_k+\xi c_k}{\lambda_n}\rho_k, & l_k+\xi c_k < \lambda_n,\[4pt] 0, & l_k+\xi c_k \ge \lambda_n, \end{cases}$6
"ThinkDrive" (Zhao et al., 8 Jan 2026)	DrivingVQA	Over GMPO: $v_k^n = \begin{cases} \rho_k - \dfrac{l_k+\xi c_k}{\lambda_n}\rho_k, & l_k+\xi c_k < \lambda_n,\[4pt] 0, & l_k+\xi c_k \ge \lambda_n, \end{cases}$7 Exam, $v_k^n = \begin{cases} \rho_k - \dfrac{l_k+\xi c_k}{\lambda_n}\rho_k, & l_k+\xi c_k < \lambda_n,\[4pt] 0, & l_k+\xi c_k \ge \lambda_n, \end{cases}$8 Easy-Exam, $v_k^n = \begin{cases} \rho_k - \dfrac{l_k+\xi c_k}{\lambda_n}\rho_k, & l_k+\xi c_k < \lambda_n,\[4pt] 0, & l_k+\xi c_k \ge \lambda_n, \end{cases}$9 Accuracy
"30-meter Land Surface Temperature from Landsat via Progressive Self-Training Downscaling" (Shen et al., 31 Mar 2026)	In situ validation	Station-averaged MAE and RMSE outperform the official cubic product by approximately $$y_c(t) = t\, y_c^* + (1-t)\frac{1}{n_{\text{classes}}}\mathbf{1},$$0

The reported improvements support a broad claim: staged optimization plus adaptive supervision can improve robustness, convergence, or compute–performance trade-offs. In some cases, the gains are tied to reliability under noisy or self-generated supervision, as in tracking, self-distillation, and self-training (Li et al., 2020, Yang et al., 2024, Shen et al., 31 Mar 2026). In others, the benefit is optimization stability, as in ACET’s continuation in label space or CoTo’s gradual activation of adapters (Dabounou, 2024, Zhuang et al., 6 Jun 2025). In still others, the main advantage is Pareto efficiency: ProST improves both effectiveness and efficiency of multi-agent SLM systems, and FPT reports over $$y_c(t) = t\, y_c^* + (1-t)\frac{1}{n_{\text{classes}}}\mathbf{1},$$1 training-computation savings with comparable performance (Bijoy et al., 2 Sep 2025, Huang et al., 2022).

The main limitations are equally recurrent. Many methods retain hand-chosen schedules: ACET uses fixed increments in $$y_c(t) = t\, y_c^* + (1-t)\frac{1}{n_{\text{classes}}}\mathbf{1},$$2; CURVETE uses arbitrarily chosen $$y_c(t) = t\, y_c^* + (1-t)\frac{1}{n_{\text{classes}}}\mathbf{1},$$3-values and fixed training loops; FPT requires predefined stage durations; ThinkDrive fixes thresholds and Gaussian scheduler parameters (Dabounou, 2024, Abbas et al., 27 Oct 2025, Huang et al., 2022, Zhao et al., 8 Jan 2026). Several methods incur extra computation because progression itself is expensive: multi-stage updates in tracking, repeated masking in LDCT denoising, view generation and See3D refinement in Close-up-GS, or multiple progressive downscaling stages in Landsat LST (Li et al., 2020, Liu et al., 20 Jan 2026, Xia et al., 12 Mar 2025, Shen et al., 31 Mar 2026). Other limitations are assumption-sensitive: ACET’s convergence analysis relies on quasi-convexity and Lipschitz gradients, blind-spot denoising inherits Noise2Self-style independence assumptions, and clustering-based systems depend strongly on pseudo-label quality (Dabounou, 2024, Liu et al., 20 Jan 2026, Zhang et al., 2019, Abbas et al., 27 Oct 2025).

An important open question is therefore whether PaST should remain a manually designed curriculum family or become a genuinely adaptive training regime. The provided works already point toward the ingredients of a fuller synthesis: confidence- and loss-based weighting, equilibrium criteria, competence functions, historical-teacher fusion, trust-region estimation, and contribution-aware credit assignment. What they do not yet provide, in most cases, is a unified mechanism that learns when to expand, what to emphasize, and how strongly to regularize without hand-specified schedules.