Consistent Progressive Training

• Consistent progressive training is a staged framework that gradually increases training difficulty or capacity while ensuring the final model remains valid.
• It employs methods such as progressive block training, subnetwork updates, and adaptive target evolution to reduce instability and optimize resources.
• The approach enhances efficiency and performance across continual, federated, and multimodal learning by aligning training stages with deployment conditions.

Consistent progressive training denotes a family of staged optimization protocols in which the effective training regime is altered progressively—by changing inputs, targets, trainable blocks, active subnetworks, prompt usage, expert capacity, or reasoning context—while each stage is designed to remain compatible with the final deployment regime or with later stages. The theme appears in rehearsal-free continual learning, coarse-to-fine visual recognition, federated learning, transformer pretraining, sparse Mixture-of-Experts, large reasoning models, and multimodal spatial intelligence. Across these works, progression is used to reduce instability, memory or compute cost, catastrophic forgetting, or context-length bottlenecks, and consistency is enforced through train/test alignment, preserved feature representations, valid subnetworks of the final model, or structured curriculum and reward schedules (Gao et al., 2024, Ren et al., 2018, Panigrahi et al., 2024).

1. Conceptual scope and historical framing

Within the literature, “progressive” does not refer to a single architectural recipe. In the generalized coarse-to-fine framework, progressive training means that the fine model begins with an easier conditioning regime and gradually transitions to the condition used at test time, with the authors explicitly relating the method to curriculum learning because data difficulty increases during training (Ren et al., 2018). In the randomized formulation of progressive training, the same idea is recast as a stochastic block-update procedure over model parameters, making progressive training analyzable through Randomized Coordinate Descent rather than treating it purely as a heuristic (Szlendak et al., 2023).

A consistent pattern across later work is that progression is coupled to an explicit notion of validity. The progressively trained object is not an arbitrary intermediate system: it is either a valid subnetwork of the final model, a partial deployment regime that converges toward the test regime, or a stagewise objective that preserves information needed by later stages. This is especially explicit in progressive subnetwork training, where only subnetworks are trained early but the full model exists from the start, and in activation-sharing training, where layers are hot-switched into the inference-efficient mode during pretraining rather than only at inference time (Panigrahi et al., 2024, Karim et al., 27 Jan 2026).

This suggests that consistent progressive training is better understood as a design pattern than as a single algorithm. The common structure is a staged increase in effective difficulty, capacity, or deployment realism, paired with mechanisms that reduce discontinuity across stages.

2. Consistency as train–test and stage-to-stage alignment

The clearest formulation of consistency appears in rehearsal-free continual learning. Consistent Prompting identifies two mismatches in prompt-based continual learning: Classifier inconsistency, where test predictions use all classifiers while training optimizes only the current-task classifier, and Prompt inconsistency, where the prompt selected at test time may not be the task-associated prompt seen during training. CPrompt addresses these through Classifier Consistency Learning (CCL), which exposes the current prompt to all existing classifiers, and Prompt Consistency Learning (PCL), which trains the current classifier with prompts sampled from the entire prompt pool. Its overall objective is

$\mathcal{L}=\mathcal{L}_{CCL}+\mathcal{L}_{PCL},$

with an auxiliary classifier and multi-key prompt selection used to stabilize prompt optimization and improve prompt selection accuracy (Gao et al., 2024).

A structurally similar train–test alignment appears in coarse-to-fine recognition. The fine model should consume coarse predictions at inference, but using those predictions too early in training is unstable, while feeding only ground truth creates oracle dependence. The proposed solution is a stochastic mixture:

$$\tilde{\mathbf{y}}(\mathbf{y}^{\star},\mathbf{y}^{\mathrm{C}};t)= \begin{cases} \mathbf{y}^{\star}, & \text{if } a\sim \mathcal{U}(0,1)>t\ \mathbf{y}^{\mathrm{C}}, & \text{otherwise,} \end{cases}$$

where $t$ increases from $0$ to $1$ over training, so that the fine model ends training under exactly the condition used at test time (Ren et al., 2018).

Consistency can also be defined in output space rather than input space. Adaptive Class Emergence Training replaces static one-hot supervision with targets that evolve from a uniform vector to the final one-hot vector,

$$y_c(t)=t\cdot y_c^*+(1-t)\cdot \frac{1}{n_{\text{classes}}}\mathbf{1},$$

and performs weight updates only when loss exceeds an equilibrium threshold. The stated motivation is smoother adaptation to increasing task complexity, with bounded target evolution and convergence to a local minimum of the final loss under the paper’s assumptions (Dabounou, 2024).

3. What is progressively changed

One major branch of the literature progresses over model blocks. ProFL partitions the global model into $T$ blocks, trains them in a progressive model shrinking stage followed by a progressive model growing stage, reuses output modules and initialization parameters from shrinking in growing, and freezes each block when an effective movement criterion indicates convergence. Under its reported setting, ProFL reduces peak memory usage by up to 57.4%, improves model accuracy by up to 82.4%, and achieves a 100% participation rate in heterogeneous federated learning (Wu et al., 2024). NeuLite also divides the model into blocks, but organizes training through elastic progressive training, a Curriculum Mentor, and a Training Harmonizer; it reports peak memory reduction by up to 50.4%, model performance enhancement by up to 84.2%, and up to 1.9X faster training (Wu et al., 2024).

A second branch progresses over subnetworks or computation modes rather than static blocks. Progressive subnetwork training keeps the full model fixed from the start but updates only random subnetworks early, increasing subnetwork size in stages until full-network training; its RAPTR instantiation speeds BERT and UL2 pretraining by up to 33% and improves QA tasks and SuperGLUE by 1.5% on UL2 (Panigrahi et al., 2024). EPAS begins as a standard transformer and gradually switches deeper decoder layers into activation-sharing mode, growing the sharing region from the deep end toward the shallow end; it reports up to 11.1% improvement in training throughput, up to 29% improvement in inference throughput, and up to 10% improvement in average accuracy in continual pretraining settings (Karim et al., 27 Jan 2026).

A third branch progresses over capacity or memory representations. EMO grows the expert pool of a sparse MoE model in multiple stages, treating expert capacity as expandable memory and using a sparsity-aware scaling law for stage-wise token allocation; in large-scale experiments it matches fixed-expert performance while using ~10% lower GPU hours (Jin et al., 13 May 2026). Progressive model-family construction expands smaller LLMs into larger ones through bert2BERT, reducing compute by approximately 25% for a 1B–8B family, and by up to 31% with size-adjusted learning rates, while yielding lower KL divergence between adjacent family members (Yano et al., 1 Apr 2025). Progressive Thought Encoding addresses a different memory bottleneck: when long rollouts exceed a fixed-size cache, evicted reasoning is progressively encoded into compact representations, giving average gains of +19.3% over LoRA-based fine-tuning and +29.9% over unfine-tuned LRMs, with up to +23.4 accuracy improvement on AIME2024/2025 under the same tight cache budgets (zhang et al., 18 Feb 2026).

Progression can also be used for continual and transfer learning without full architectural replacement. EXPANSE expands pretrained layers by adding new nodes to existing layers and combines this with a two-step training regime described as “learning basics first, then adding complexities and uncertainties,” with the stated goal of avoiding both catastrophic forgetting and overly biased pre-trained models (Iman et al., 2022). PST, by contrast, retains a single static network, segments parameters into fixed, important, and secondary groups, reinforces and then freezes important parameters, and uses a fixed-size replay buffer to maintain single-head continual learning performance (Du et al., 2019).

4. Curriculum, abstraction, and hierarchical skill acquisition

Another cluster of methods makes progression explicit at the level of task hierarchy. In fine-grained visual classification, Progressive Multi-Granularity Training of Jigsaw Patches trains branches from finer to coarser granularities, uses stage-specific jigsaw inputs, and concatenates features only at the final stage; it reports state-of-the-art or competitive results on CUB-200-2011, Stanford Cars, and FGVC-Aircraft (Du et al., 2020). In visual object tracking, DT-Training scales training data volume, model size, and input resolution progressively, while using small teacher transfer and dual-branch alignment to address the optimization difficulties of naive train-big-from-scratch scaling; the paper reports a 4.7% improvement on LaSOT when scaling from ViT-Base to ViT-Large at 384 resolution, versus 2.4% for naive scaling, and a 64.8 mean AUC result on GTrack Bench (Hong et al., 26 May 2025).

Progression is especially explicit in multimodal generation and reasoning curricula. ByteLoom uses a three-stage progressive curriculum learning schedule—human pose conditioning, hand-object interaction, and full HOI—so that large weakly annotated datasets can support foundational skill acquisition before scarce high-quality HOI data are used for final adaptation; the paper states that this curriculum “effectively overcomes data scarcity while eliminating hand mesh requirements” (Liu et al., 28 Dec 2025). SpatialLadder adopts a perception-to-understanding-to-reasoning sequence: object localization, multi-dimensional spatial tasks across single-image, multi-view, and video settings, and then reinforcement learning with verifiable rewards. Its reported outcome is a 23.4% average improvement over the base model and 7.2% improvement on out-of-domain benchmarks (Li et al., 9 Oct 2025).

In generative training efficiency, Ent-Prog uses Conditional Entropy Inflation (CEI) to prioritize blocks and an adaptive progressive schedule that selects how many high-priority blocks to unfreeze by maximizing convergence efficiency. It reports up to 2.2$\times$ training speedup and 2.4$\times$ GPU memory reduction without compromising generative performance (Li et al., 26 Nov 2025). Although MovieTeller is explicitly training-free, its tool-augmented progressive abstraction pipeline—scene description, chapter summarization, and final synthesis—shows that the logic of progressive decomposition extends beyond optimization into inference-time narrative construction (Li et al., 26 Feb 2026).

5. Optimization theory and formal guarantees

The strongest formalization of progressive training appears in Randomized Progressive Training. RPT partitions the parameter space into blocks and updates a random progressive sketch,

$x^{t+1}=x^t-\gamma C_t \nabla f(x^t),$

with convergence controlled by the block-aware smoothness quantity

$$\tilde{\mathbf{y}}(\mathbf{y}^{\star},\mathbf{y}^{\mathrm{C}};t)= \begin{cases} \mathbf{y}^{\star}, & \text{if } a\sim \mathcal{U}(0,1)>t\ \mathbf{y}^{\mathrm{C}}, & \text{otherwise,} \end{cases}$$0

For $$\tilde{\mathbf{y}}(\mathbf{y}^{\star},\mathbf{y}^{\mathrm{C}};t)= \begin{cases} \mathbf{y}^{\star}, & \text{if } a\sim \mathcal{U}(0,1)>t\ \mathbf{y}^{\mathrm{C}}, & \text{otherwise,} \end{cases}$$1-strongly convex objectives and $$\tilde{\mathbf{y}}(\mathbf{y}^{\star},\mathbf{y}^{\mathrm{C}};t)= \begin{cases} \mathbf{y}^{\star}, & \text{if } a\sim \mathcal{U}(0,1)>t\ \mathbf{y}^{\mathrm{C}}, & \text{otherwise,} \end{cases}$$2, the paper gives

$$\tilde{\mathbf{y}}(\mathbf{y}^{\star},\mathbf{y}^{\mathrm{C}};t)= \begin{cases} \mathbf{y}^{\star}, & \text{if } a\sim \mathcal{U}(0,1)>t\ \mathbf{y}^{\mathrm{C}}, & \text{otherwise,} \end{cases}$$3

and it also provides convex and nonconvex guarantees. The stated contribution is that RPT is the first PT-type algorithm with rigorous guarantees for general smooth objectives (Szlendak et al., 2023).

Stage-transition stability is another recurring theoretical concern. ProFL proves convergence under standard stochastic-optimization assumptions and interprets blockwise training as converging at a rate of $$\tilde{\mathbf{y}}(\mathbf{y}^{\star},\mathbf{y}^{\mathrm{C}};t)= \begin{cases} \mathbf{y}^{\star}, & \text{if } a\sim \mathcal{U}(0,1)>t\ \mathbf{y}^{\mathrm{C}}, & \text{otherwise,} \end{cases}$$4 for each training step (Wu et al., 2024). RAPTR argues that increasing subnetwork complexity aligns with the tendency of SGD to learn lower-complexity structure first, and that residual connections and layer normalization help keep loss smooth across stage transitions (Panigrahi et al., 2024). EMO provides a scaling-law formulation,

$$\tilde{\mathbf{y}}(\mathbf{y}^{\star},\mathbf{y}^{\mathrm{C}};t)= \begin{cases} \mathbf{y}^{\star}, & \text{if } a\sim \mathcal{U}(0,1)>t\ \mathbf{y}^{\mathrm{C}}, & \text{otherwise,} \end{cases}$$5

to determine stage-wise compute-optimal token allocations during expert expansion (Jin et al., 13 May 2026). ACET, meanwhile, gives bounded-gradient and convergence arguments for progressively evolving targets rather than progressively changing architecture (Dabounou, 2024).

These analyses indicate that consistency is not merely empirical smoothness. In several lines of work it becomes a formal requirement: unbiased or controlled stochastic updates, stable loss across stage transitions, or preserved optimization geometry when stagewise constraints are changed.

6. Empirical regularities, misconceptions, and open issues

A common misconception is that progressive training is synonymous with monotonic model growth. The literature is broader. Some methods do grow capacity, such as EMO and model expansion for LLM families, but others progressively replace oracle inputs with model predictions, progressively sharpen labels, progressively enlarge random subnetworks, or progressively increase activation sharing to reduce computation (Ren et al., 2018, Dabounou, 2024, Panigrahi et al., 2024, Karim et al., 27 Jan 2026). Another misconception is that consistency requires fully end-to-end updating of the entire final model from the outset. ProFL and NeuLite rely on freezing, auxiliary output modules, and stagewise block training, yet frame these mechanisms as necessary to preserve feature representations and break information isolation (Wu et al., 2024, Wu et al., 2024).

Empirically, consistent progressive training is repeatedly associated with smoother transitions, better deployment alignment, or improved efficiency. In continual learning, CPrompt reports 6.22% Last-acc and 4.96% Avg-acc gains over previous SOTA on Split StanfordCars (20-task), plus 2.54% Last-acc and 3.33% Avg-acc gains on Split ImageNet-R (20-task), with improvements attributed to more consistent training and testing (Gao et al., 2024). In efficient transformer and language-model training, RAPTR, EPAS, model-family expansion, and PTE all report improved cost-quality trade-offs without requiring independent training of every regime (Panigrahi et al., 2024, Karim et al., 27 Jan 2026, Yano et al., 1 Apr 2025, zhang et al., 18 Feb 2026). In multimodal reasoning, SpatialLadder’s ablations indicate that the curriculum order itself matters, with perception-to-reasoning outperforming all-at-once or reversed training (Li et al., 9 Oct 2025).

Open issues remain. Many methods introduce nontrivial schedule design problems: when to freeze a block, when to expand the expert pool, how fast to evolve targets, or how large the active subnetwork should be at each stage. Several systems depend on auxiliary modules—output heads, distillation paths, auxiliary classifiers, latent summaries, or external tools—to make stage transitions workable. In generative settings, efficiency improvements may also have broader consequences: Ent-Prog explicitly notes that more efficient training might facilitate proliferation of generative models with unintended uses or biases (Li et al., 26 Nov 2025).

Taken together, the literature supports a precise but broad interpretation. Consistent progressive training is a family of staged learning strategies in which progression manages difficulty, capacity, or efficiency, and consistency prevents those stages from becoming mismatched with inference, later stages, or the final model semantics. The design space spans continual learning, federated learning, efficient pretraining, multimodal reasoning, and generative modeling, but the central technical question remains the same: how to change the training regime over time without invalidating what later stages or deployment will require.