Progressive Training Structure (PTS)

• Progressive Training Structure (PTS) is a curriculum-based optimization approach that gradually introduces complexity—whether in tasks, model capacity, or data regimes—to stabilize training and enhance efficiency.
• PTS employs modular strategies like task decomposition, capacity scaling, and teacher–student transfers to control optimization difficulty and maintain high performance across diverse domains.
• Empirical studies show PTS can reduce compute costs, improve convergence stability, and boost generalization, making it suitable for applications from speech recognition to federated learning.

Progressive Training Structure (PTS) denotes a family of staged optimization schemes in which training proceeds through progressively harder tasks, progressively larger subnetworks, progressively broader data or resolution regimes, progressively sharper targets, or progressively more expressive teacher–student transfers. Across the literature, the term is used for modular curricula in overlapped speech recognition, partial-model expansion in prompt tuning, stagewise subnetwork pretraining, progressive block freezing in federated and continual learning, progressive target evolution in classification, and hierarchical task chains in binary analysis. Taken together, these works suggest that PTS is best understood not as a single fixed algorithm but as a general curriculum-structured training pattern for controlling optimization difficulty, compute, and generalization (Chen et al., 2017, Szlendak et al., 2023, Panigrahi et al., 2024).

1. Historical emergence and conceptual scope

Early progressive training was introduced as a heuristic for growing neural networks in stages. In the formulation analyzed by Randomized Progressive Training, one fixes a sequence of blocks and, for a prescribed number of iterations, trains only the first block, then the first two blocks, then the first three, and so on, until all $B$ blocks have been trained. The two main motivations are stability and efficiency: the optimization landscape is simpler for a small network, and training a smaller subnetwork is cheaper per iteration (Szlendak et al., 2023).

This original “grow-the-model” view was extended in several directions. “Progressive Structure-conditional Generative Adversarial Networks” grows both image resolution and the resolution of the structural pose-condition map in tandem, while “Progressive Neural Networks for Image Classification” uses a multi-stage architecture with progressively increased complexity and visual recognition power and attaches a confidence-based early-exit policy (Hamada et al., 2018, Zhang et al., 2018). In unsupervised single-channel overlapped speech recognition, PTS became a curriculum over subtasks—frame-wise interpreting, utterance-level speaker tracing, and speech recognition—rather than only a schedule over layer growth (Chen et al., 2017).

Subsequent work broadened the term still further. Fast Prompt Tuning expands partial PLMs in depth and width; ProFL partitions a model into blocks and trains each block in a progressive fashion; ACET evolves the target outputs from a uniform vector to one-hot encoded vectors; ProTST organizes binary-analysis tasks as a teacher–student chain from MLM to malware classification; and PART separates within-language from cross-language alignment through stagewise freezing and unfreezing. This suggests that the essential feature of PTS is controlled progression, while the quantity being progressed is domain dependent (Huang et al., 2022, Wu et al., 2024, Dabounou, 2024, Lu et al., 2024, Zhang et al., 24 Sep 2025).

2. Recurrent architectural and algorithmic patterns

Across domains, PTS instantiations recur around a small set of design patterns. Some papers progress task difficulty; others progress model size, active parameters, or training targets; still others combine stagewise growth with distillation or alignment constraints.

Pattern	Mechanism	Representative systems
Task decomposition	Simpler subtasks precede full-task optimization	PTS for overlapped ASR, PPT, ProST
Capacity scaling	Depth, width, resolution, data volume, or active subnet size increases over stages	FPT, DT-Training, PSGAN, RaPTr, zero/one-layer PTS
Selective parameter activation	Only a subset of blocks, layers, or segments is trainable at a stage	ProFL, PST, PART
Teacher–student progression	Later stages reuse parameters, hidden states, or soft targets from earlier stages	ProGMLP, ProTST, POCL, self-transfer PTS
Target or decision progression	Labels or exit policies become sharper or more selective over time	ACET, ProgNet

The speech-recognition PTS divides training into four successive stages of increasing difficulty; PPT uses three pretext tasks—stepwise next-position prediction, destination prediction, and full trajectory prediction; and ProST introduces subtasks in their natural order via a linear schedule over epochs (Chen et al., 2017, Lin et al., 2024, Bijoy et al., 2 Sep 2025). FPT starts with a small-scale partial PLM and then progressively expands its depth and width; DT-Training scales training-data volume, model capacity, and input resolution; RaPTr progressively increases the size of subnetworks during training; and zero/one-layer progressive training expands from $N_{\text{small}}\in\{0,1\}$ to a deeper target model at a chosen iteration $\tau$ (Huang et al., 2022, Hong et al., 26 May 2025, Panigrahi et al., 2024, Bu, 7 Nov 2025).

Other variants center on structural control of parameter updates. ProFL partitions the model into contiguous blocks, first learns initialization models through progressive model shrinking, then trains the full model through progressive model growing, freezing each block after convergence. Progressive Segmented Training similarly freezes the parameters most critical to each task and leaves the remainder free for future tasks (Wu et al., 2024, Du et al., 2019). ACET, by contrast, changes neither architecture nor block structure; it progressively evolves the target itself from $\tfrac{1}{n_{\text{classes}}}\mathbf 1$ to the usual one-hot vector (Dabounou, 2024).

3. Objective functions and optimization mechanics

PTS methods are unified less by a single loss than by a common stagewise organization of losses. In ACET, the progression variable is the target: $$\mathbf{y}^c(t)=t\,\mathbf{y}^c+(1-t)\,\tfrac{1}{n_{\text{classes}}}\mathbf 1,\qquad t\in[0,1].$$ Training then minimizes the ordinary cross-entropy against $\mathbf y^c(t)$, and the schedule over $t$ controls problem difficulty (Dabounou, 2024).

In tracking, the progression variable is the training regime rather than the label. DT-Training combines a clean branch, a masked branch, and a frozen teacher: $$L_{\rm total}(f;\hat f) = L_{\rm clean}(f) +\lambda_{\rm transfer}L_{\rm transfer}(f;\hat f) +\lambda_{\rm align}L_{\rm align}(f).$$ Here Small Teacher Transfer supplies a previous-stage teacher, while Dual-Branch Alignment aligns clean and randomly masked inputs, with zero extra cost at inference time (Hong et al., 26 May 2025).

In overlapped speech recognition, progressive stages culminate in utterance-level PIT objectives and sequence-discriminative training. The joint fine-tuning criterion is

$$\mathcal{J}_{\mathrm{CE\text{-}PIT}} = \sum_u \min_{s\in S}\sum_{t,n} -\log P\bigl(l_{ut,n}^{(r)}\mid \mathbf O_u^{(m)};s\bigr),$$

and later refinements include self-transfer KLD-PIT and de-correlated LF-DC-bMMI, which penalizes cross-speaker (“swapped”) errors (Chen et al., 2017).

ProFL introduces an explicit convergence assessment metric for deciding when a block can be frozen: $EM_{t,k}^H = \frac{D_{B,k}^H}{N_{B,k}^H}.$ Large $N_{\text{small}}\in\{0,1\}$0 indicates that parameters are still making net progress; once sign changes cause cancellations and $N_{\text{small}}\in\{0,1\}$1, the block is near a local optimum and may be frozen (Wu et al., 2024). ProGMLP likewise formalizes progression through student index $N_{\text{small}}\in\{0,1\}$2, with each student minimizing

$N_{\text{small}}\in\{0,1\}$3

where Progressive Knowledge Distillation scales later-student penalties by $N_{\text{small}}\in\{0,1\}$4 and Progressive Mixup Augmentation progressively generates harder mixed samples (Lu et al., 25 Jul 2025).

These examples show that PTS commonly wraps domain-specific objectives inside a stage transition rule. The progression may alter the target, the active model, the teacher signal, the data subset, or the stopping criterion, but the underlying losses often remain standard cross-entropy, MSE, KL, or task-specific sequence criteria.

4. Representative instantiations across research domains

In overlapped speech recognition, PTS imposes a modular architecture consisting of a frame-wise interpreting module, an utterance-level speaker tracing module, and a speech recognition module. Training follows four successive stages: frame-wise PIT separation, utterance-level PIT tracing, clean-speech ASR pretraining, and full joint fine-tuning. Transfer learning leverages parallel clean speech, and LF-DC-bMMI integrates sequence-level linguistic knowledge. On artificial overlapped Switchboard and hub5e-swb, the framework achieves over 30% relative improvement of WER over both a strong jointly trained system and a separately optimized system (Chen et al., 2017).

In parameter-efficient language-model adaptation, Fast Prompt Tuning observes that soft prompts learned by different partial PLMs of various sizes are similar in the parameter space. It therefore conducts prompt tuning first on a small-scale partial PLM and then progressively expands depth and width, recycling the learned soft prompts as initialization. On five tasks, FPT could save over 30% training computations while achieving comparable performance (Huang et al., 2022).

In large-scale pretraining, Progressive Subnetwork Training maintains the full target model throughout training but restricts forward, backward, and update computations to a subnetwork whose size grows over stages. RaPTr can significantly speed up training of standard benchmarks like BERT and UL2, up to 33% compared to standard training, and also shows better downstream performance on UL2, improving QA tasks and SuperGLUE by 1.5%. A distinct depth-scaling variant, zero/one-layer progressive training, trains a model of depth $N_{\text{small}}\in\{0,1\}$5 or $N_{\text{small}}\in\{0,1\}$6 until time $N_{\text{small}}\in\{0,1\}$7, expands to the full model, and continues training; on GPT2 it can save $N_{\text{small}}\in\{0,1\}$8 compute, or equivalently accelerate $N_{\text{small}}\in\{0,1\}$9, while achieving almost the same loss (Panigrahi et al., 2024, Bu, 7 Nov 2025).

In human trajectory prediction, Progressive Pretext Task Learning is motivated by the distinction between short-term dynamics and long-term dependencies. Stage I trains stepwise next-position prediction, Stage II trains destination prediction using a learnable prompt embedding, and Stage III performs full trajectory prediction with cross-task knowledge distillation to alleviate knowledge forgetting. On SDD, ETH/UCY, and GCS, PPT reports new state-of-the-art minADE$\tau$0 / minFDE$\tau$1 results and training on SDD completes in 4.7 h versus 6–8 h for a monolithic one-stage baseline (Lin et al., 2024).

In visual tracking, DT-Training scales three orthogonal axes—training-data volume, model size, and input resolution—through a three-stage schedule: data↑ with ViT-Base@256, model↑ to ViT-Large, and resolution↑ to $\tau$2. Small Teacher Transfer and Dual-Branch Alignment are applied at each stage. The baseline B-256 reaches 59.4 mean AUC, Stage 1 reaches 62.0, Stage 2 reaches 63.6, and Stage 3 reaches 64.8 (Hong et al., 26 May 2025).

In federated and continual learning, PTS is used to control trainable capacity under resource constraints. ProFL progressively trains the front blocks, safely freezes them after convergence, and triggers training of the next block, reducing peak memory footprint by up to 57.4% and improving model accuracy by up to 82.4%. Progressive Segmented Training, in a single-network continual-learning setting, identifies important filters or neurons by Taylor-based saliency, freezes them, and leaves secondary parameters available for future tasks (Wu et al., 2024, Du et al., 2019).

Hierarchical teacher–student progression appears in binary analysis and multilingual speech. ProTST trains one transformer per node in a chain from MLM through instruction and function tasks to malware classification, yielding an average validation score improvement of 14.8% compared to traditional two-stage training. PART separates within-language alignment from cross-language alignment through three stages: adapter-only monolingual alignment, progressive encoder unfreezing, and joint cross-language and text-based fine-tuning with the LLM unfrozen (Lu et al., 2024, Zhang et al., 24 Sep 2025).

5. Reported empirical effects

The empirical profile of PTS is heterogeneous because different papers optimize different operating points: WER, BLEU, AUC, ROUGE-L, FLOPs, wall-clock time, memory footprint, or adaptive inference cost. Even so, the reported gains exhibit a consistent pattern: stagewise progression often improves either optimization stability, final quality, or compute efficiency, and frequently more than one of these simultaneously.

Setting	Reported outcome	Paper
Overlapped speech recognition	57.5% $\tau$3 35.2% WER on 50-hour; 42.2% $\tau$4 30.8% on 150-hour; over 30% relative WER reduction	(Chen et al., 2017)
Prompt tuning	FPT_CR saves $\tau$5 FLOPs and $\tau$6 wall-clock time while matching within 0.5% the average score of full PT	(Huang et al., 2022)
Human trajectory prediction	SDD: 7.03 / 10.65 px; training on SDD completes in 4.7 h versus 6–8 h	(Lin et al., 2024)
Visual object tracking	Mean AUC improves from 59.4 to 64.8 after data↑, model↑, resolution↑	(Hong et al., 26 May 2025)
Federated learning	Peak memory footprint reduced by up to 57.4%; model accuracy improved by up to 82.4%	(Wu et al., 2024)
Binary analysis	Average validation score improvement of 14.8% over traditional two-stage training	(Lu et al., 2024)

Comparable effects are reported in additional domains. ProST raises Task Goal Completion from 35.7% to 42.3% in the 14B multi-agent setting, and its Pareto analysis shows that ProST fine-tuning yields better effectiveness-efficiency trade-offs. POCL improves distilled instruction-following students across multiple white-box KD methods and model families, with validation ROUGE-L remaining above KD alone throughout training. ACET reports faster convergence, smoother loss curves on MNIST, and training-time reductions ranging from 15.2% to 79.2% depending on dataset, alongside small but consistent accuracy improvements on several tasks (Bijoy et al., 2 Sep 2025, Liu et al., 6 Jun 2025, Dabounou, 2024).

These results indicate that “improvement” under PTS should not be reduced to a single axis. Some methods primarily reduce compute, such as FPT, RaPTr, and zero/one-layer depth expansion. Others primarily improve robustness or generalization, such as ACET, POCL, and ProTST. Still others use stagewise structure to make otherwise infeasible training regimes practical, as in ProFL under heterogeneous memory budgets.

6. Theory, misconceptions, and unresolved issues

A central theoretical issue is that classical progressive training was long treated as a heuristic. Randomized Progressive Training addresses this directly by introducing a randomized surrogate of PT that fits into the Randomized Coordinate Descent framework and provides rigorous and sound theoretical guarantees for general smooth objective functions in the strongly convex, convex, and nonconvex regimes. This is a notable shift from earlier practice, where classic PT had no convergence guarantees even for quadratic objectives (Szlendak et al., 2023).

Another recurring misconception is that progressive training is synonymous with progressive growing of GAN image resolution or with stacking-only strategies. The literature is broader. PSGAN grows resolution and pose maps; FPT expands depth and width in partial PLMs; ACET evolves targets; ProFL and PST progressively restrict which parameters are trainable; ProST exposes progressively longer subtask prefixes; and PART delays LLM activation until cross-language training (Hamada et al., 2018, Huang et al., 2022, Dabounou, 2024, Wu et al., 2024, Du et al., 2019, Bijoy et al., 2 Sep 2025, Zhang et al., 24 Sep 2025). A related debate concerns dropping-based stagewise training. “Efficient Stagewise Pretraining via Progressive Subnetworks” explicitly challenges the view that stagewise dropping strategies are ineffective, arguing that, with proper design, dropping strategies can be competitive, if not better, than stacking methods (Panigrahi et al., 2024).

Theoretical analyses also clarify when stage transitions remain stable. RAPTR attributes stable stage transitions to residual connections and layer norms, while zero/one-layer progressive training analyzes initialization of newly added layers, hyperparameter transfer under muP scaling, and the timing of model expansion. In that framework, the Warmup–Stable–Decay schedule yields a smaller cumulative $\tau$7, enabling expansion as late as $\tau$8, and GPT2 experiments report 4.6–4.7$\tau$9 speedup with only 0.2%–0.5% validation-loss degradation (Panigrahi et al., 2024, Bu, 7 Nov 2025).

Open issues are method specific. ProFL depends on a freezing rule based on effective movement; PST notes that the choice of $\tfrac{1}{n_{\text{classes}}}\mathbf 1$0 is heuristic and task dependent and still relies on a modest replay buffer; ACET shows that overly small $\tfrac{1}{n_{\text{classes}}}\mathbf 1$1 loses stability and overly large $\tfrac{1}{n_{\text{classes}}}\mathbf 1$2 hurts final accuracy; ProGMLP reports no formal convergence proof; POCL depends on the quality of the difficulty measurer and scheduler; and overlapped ASR PTS derives part of its gain from transfer learning and LF-DC-bMMI rather than progression alone (Wu et al., 2024, Du et al., 2019, Dabounou, 2024, Lu et al., 25 Jul 2025, Liu et al., 6 Jun 2025, Chen et al., 2017).

A plausible implication is that PTS is best regarded as a higher-level training scaffold rather than a standalone optimizer. Its distinctive contribution is to impose order on optimization—easy to hard, small to large, local to global, or monolingual to cross-lingual—while allowing the underlying losses, architectures, and transfer mechanisms to remain domain specific. That flexibility explains both its wide applicability and the absence of a single canonical PTS algorithm.