Fast-Slow Training Strategy (FSTS)

• Fast-Slow Training Strategy (FSTS) is a design principle that divides learning into fast and slow components to enable quick adaptation and maintain long-term stability.
• Its methodology employs dual-timescale parameter updates, schedule-based resource allocation, and iterative refinement to balance efficiency with accuracy.
• Empirical studies show that FSTS accelerates training, reduces computational costs, and enhances generalization across applications like object re-identification and LLM post-training.

Searching arXiv for papers on fast-slow training strategies and closely related terminology. arXiv search query: all:"Fast-Slow Training Strategy" OR all:"fast-slow" OR ti:"Learning, Fast and Slow" Fast-Slow Training Strategy (FSTS) denotes a family of training and adaptation schemes that separate learning into components, phases, or parameter groups operating at different time scales. Across the cited literature, the fast component is used for rapid task-specific adaptation, low-latency decision-making, or inexpensive early optimization, whereas the slow component is used to preserve stability, generalizable structure, contextual reasoning, or higher-fidelity refinement. The term is used explicitly in some papers—notably in object re-identification, online preference alignment, test-time adaptation, and continual LLM post-training—but closely related formulations also appear under names such as “Fast First, Accurate Second Training (FAST),” “multirate training,” “Fast Forward,” and “Fast-in-Slow” (Wang et al., 23 Sep 2025, Qi et al., 2024, Tiwari et al., 12 May 2026, Zhang et al., 2021).

1. Definition and terminological scope

In the broadest sense represented in the literature, FSTS is not a single algorithm but a design principle: allocate different parts of learning to different temporal scales. The “fast” side may be a rapidly updated branch, a low-precision early phase, a prompt population, a local online corrector, or a low-latency encoder. The “slow” side may be an EMA teacher, model parameters updated conservatively, a larger-context branch, an offline ensemble, or an iterative solver that refines fast outputs (Wang et al., 23 Sep 2025, Tiwari et al., 12 May 2026, Moritz et al., 2024, Giuli et al., 16 Jul 2025).

The exact semantics of “fast” and “slow” vary by domain. In APC for object ReID, FSTS means a Fast Update Stream (FUS) and a Slow Update Stream (SUS), where FUS rapidly captures ReID-specific cues and SUS preserves the general visual perception inherited from CLIP (Wang et al., 23 Sep 2025). In LLM post-training, Fast-Slow Training (FST) defines slow weights as model parameters $\theta$ and fast weights as optimized textual context $\phi$ or a prompt population $\Phi$ (Tiwari et al., 12 May 2026). In FAST for DNN training, the paper does not use the label FSTS as its exact title, but it explicitly describes a “Fast First, Accurate Second Training” policy in which training starts with low-precision BFP and becomes more precise across training iterations and layer depth (Zhang et al., 2021).

This terminological variability is itself important. Some works use the expression for training proper; others apply the same fast-slow logic to test-time adaptation, online control, or inference-time regulation. “Fast-Slow Test-Time Adaptation” for VLN is an adaptation framework rather than an offline training method (Gao et al., 2023). “Gold-Switch” is explicitly a training-free deployment-time superposition method rather than a training algorithm (Lee et al., 8 Oct 2025). A plausible implication is that FSTS is best understood as a recurrent systems pattern rather than a single standardized method.

2. Genealogy of the fast-slow idea

An early explicit formulation appears in cooperative conditional learning, where a fast thinking initializer generates an output directly from the condition and latent noise, while a slow thinking solver refines that output under a conditional energy-based model by Langevin dynamics (Xie et al., 2019). The initializer learns from how the solver refines its samples, and the solver learns from the difference between refined outputs and observed data. This established a teacher-like division between amortized generation and iterative refinement.

Subsequent work specialized the same idea for transfer learning, optimization, and systems training. “Fast And Stable Task-adaptation” argued that ordinary fine-tuning is too aggressive at the beginning because a randomly initialized classifier can cause catastrophic forgetting before the target task is learned. Its remedy was fast adaptation of the appended classifier and slow, stable adaptation of the pretrained backbone through near-zero classifier initialization and asymmetric learning rates (Varno et al., 2020). “Multirate Training of Neural Networks” formalized the fast-slow split as parameter partitioning into $\theta_F$ and $\theta_S$, with fast parameters updated every step and slow parameters updated every $k$ steps, primarily for transfer learning speed-up and regularization (Vlaar et al., 2021).

In hardware-aware DNN training, FAST translated the fast-slow idea into numerical precision scheduling. Weights, activations, and gradients are represented in variable-precision block floating point (BFP), with low precision used early and higher precision introduced later and in deeper layers (Zhang et al., 2021). In online and continual settings, the same pattern was adopted for preference alignment, test-time adaptation, and continual LLM learning. OFS-DPO used two identical LoRA modules with different optimization speeds and a coupling regularizer (Qi et al., 2024). FSTTA for online VLN alternated a frequent gradient-based fast phase with a periodic parameter-trajectory slow phase (Gao et al., 2023). LLM FST separated adaptation into fast textual context and slow parametric learning (Tiwari et al., 12 May 2026).

By 2025–2026, fast-slow constructions had become increasingly architectural. FiS-VLA embedded a fast executor inside a slow VLM-based reasoning model for manipulation (Chen et al., 2 Jun 2025). JSTAR used a cascaded fast encoder for ASR and slow encoder for ST (Moritz et al., 2024). Slow4Fast-VLN introduced fast execution plus slow reflection and structured experience extraction for open-world navigation (Li et al., 14 Jan 2026). In parallel, fast-slow concepts also appeared in adaptive optimization and deployment control, as in Fast Forward for low-rank finetuning and Gold-Switch for training-free switching between fast- and slow-thinking behavior (Rahamim et al., 2024, Lee et al., 8 Oct 2025).

3. Canonical mechanisms

Despite heterogeneity across domains, several recurring mechanisms appear repeatedly.

First, many FSTS variants use dual-timescale parameter updates. In APC, FUS is updated by ordinary gradient descent, while SUS is updated by exponential moving average and periodically provides visual and attribute-aware identity prototypes that supervise FUS through contrastive loss (Wang et al., 23 Sep 2025). In multirate training, slow momentum is updated once every $k$ steps, while fast parameters are updated at each inner step and slow parameters are drifted linearly using stored slow momentum; the scheme uses coupled step sizes with $h_S = k h_F$ (Vlaar et al., 2021). In continual LLM post-training, the joint objective is

$$\max_{\theta,\,\Phi}\;\; J(\theta,\Phi) = \mathbb{E}_{x\sim\mathcal{D},\;\phi\sim U(\Phi),\;y\sim\pi_\theta(\cdot\mid x,\phi)} \!\left[r(x,y)\right],$$

so task adaptation is split between slow weights $\theta$ and fast prompt population $\phi$0 (Tiwari et al., 12 May 2026).

Second, some FSTS variants use schedule-based resource allocation rather than two persistent models. FAST defines an adaptive precision schedule between 2-bit and 4-bit mantissa BFP using

$\phi$1

and a relative improvement score

$\phi$2

If $\phi$3, 2-bit mantissa BFP is used; otherwise 4-bit is used (Zhang et al., 2021). Here, “fast” means cheaper early and shallow computation, whereas “slow” means later, deeper, more accurate computation.

Third, several formulations use fast generation plus slow refinement. In cooperative conditional learning, the generator samples

$\phi$4

while the slow solver refines by Langevin dynamics,

$\phi$5

The initializer then regresses toward the refined samples (Xie et al., 2019).

Fourth, there are optimizer-level fast-slow schemes. Fast Forward alternates ordinary Adam SGD steps with a Fast Forward stage that repeatedly applies the latest update direction

$\phi$6

until the loss on a 32-example tiny validation set stops improving (Rahamim et al., 2024). Here, the temporal split is between expensive gradient recomputation and cheap extrapolative repetition.

4. Major application families

The fast-slow pattern has been instantiated in substantially different technical settings.

Setting	Fast element	Slow element
Conditional learning (Xie et al., 2019)	Direct conditional generator	Conditional energy-based solver with Langevin refinement
Variable-precision DNN training (Zhang et al., 2021)	Low-precision BFP early and in earlier layers	Higher-precision BFP later and in deeper layers
Object ReID (Wang et al., 23 Sep 2025)	FUS trained by gradient descent	SUS updated by EMA and used as a prototype teacher
LLM post-training (Tiwari et al., 12 May 2026)	Optimized textual context or prompt population	Model parameters updated by RL
Streaming ASR+ST (Moritz et al., 2024)	Low-latency encoder for ASR	Larger-context encoder for ST
Dynamical model adaptation (Giuli et al., 16 Jul 2025)	Online GP residual correction	Offline ensemble over operating conditions

In representation learning and alignment, the central tension is usually fast specialization versus slow preservation. APC states this as discrimination versus generalization in ReID, with FUS rapidly learning identity-specific cues and SUS retaining generalizable knowledge inherited from CLIP (Wang et al., 23 Sep 2025). LLM FST states the same tension as absorbing task-specific information without catastrophic forgetting or loss of plasticity, assigning fast adaptation to optimized context and slower consolidation to model weights (Tiwari et al., 12 May 2026). OFS-DPO similarly uses fast and slow LoRA modules with different optimization speeds to improve online preference alignment and mitigate forgetting in cross-domain settings (Qi et al., 2024).

In latency-sensitive multimodal systems, the split often matches different computational horizons. JSTAR uses a fast encoder directly connected to the frontend for low-latency streaming ASR and a slow encoder with larger context for streaming ST, with separate predictor and joiner networks for the two tasks (Moritz et al., 2024). FiS-VLA uses a slow VLM/LLM System 2 for contextual reasoning and a fast embedded System 1 execution module repurposed from the final transformer blocks, operating asynchronously at a frequency ratio of $\phi$7 in the main setting (Chen et al., 2 Jun 2025). Slow4Fast-VLN uses fast end-to-end navigation plus slow LLM-based reflection that extracts structured experience

$\phi$8

and reinjects it into the fast policy through retrieval and feature fusion (Li et al., 14 Jan 2026).

In numerical training and systems optimization, the split is often between cheap approximate updates and expensive accurate ones. FAST uses low-precision early training and stochastic rounding for gradients, supported by an fMAC that processes mantissas in 2-bit chunks (Zhang et al., 2021). Multirate training updates slow parameter groups less frequently to reduce fine-tuning cost (Vlaar et al., 2021). Fast Forward extrapolates recent low-rank update directions rather than recomputing full gradients (Rahamim et al., 2024). In energy-system adaptation, the slow layer is an ensemble of GRU-based models over operating regimes and the fast layer is a sliding-window GP corrector for residual mismatch (Giuli et al., 16 Jul 2025).

5. Empirical effects

Reported benefits of FSTS are typically expressed as improved efficiency, stability, or robustness, but the specific measured quantity depends strongly on the domain.

In variable-precision DNN training, FAST reports a 2–6$\phi$9 speedup in training on a single-chip platform over prior mixed-precision or BFP approaches while maintaining similar validation accuracy; FP32 generally gives the best accuracy, but FAST-Adaptive achieves near-FP32 accuracy while using lower precision early and higher precision later (Zhang et al., 2021). In low-rank finetuning, Fast Forward reports 41–87% FLOPs reduction and 40–81% train-time reduction relative to standard Adam SGD, and one convergence experiment reports 56% FLOPs saved while reaching a slightly better loss than regular Adam SGD (Rahamim et al., 2024). Multirate training reports that transfer-learning models can be fine-tuned in almost half the time without reducing generalization performance (Vlaar et al., 2021).

In representation learning, APC provides a direct ablation of FSTS. On MSMT17, the baseline is 66.4 mAP / 84.4 R1, and adding FSTS alone raises this to 75.0 mAP / 88.8 R1; on MSMT17 $\Phi$0 Market1501, the baseline is 16.2 mAP / 32.5 R1, and adding FSTS raises this to 42.2 mAP / 67.6 R1 (Wang et al., 23 Sep 2025). In LLM post-training, FST reaches RL’s running peak with 3.0× fewer steps on CodeIO, 1.4× fewer steps on Math (Polaris), and 3.0× fewer steps on HoVer-hard, while also reaching higher fitted asymptotes and staying closer to the base policy with up to 70% less KL divergence at matched reward (Tiwari et al., 12 May 2026).

In embodied and streaming systems, FiS-VLA reports 69% mean success on RLBench versus 61% for CogACT and 55% for $\Phi$1, plus a theoretical 117.7 Hz control frequency with action chunk size 8 (Chen et al., 2 Jun 2025). JSTAR reports that, on RealConv, its first-token latency is 3.3 s versus 7.1 s for a cascaded system, with about 2 extra BLEU points on average (Moritz et al., 2024). FSTTA for online VLN reports that DUET receives a performance boost of over 5% on REVERIE/R2R-CE and that, on REVERIE validation unseen, it improves over SAR while reducing testing time by about 7% (Gao et al., 2023). In lifelong dynamical model adaptation, the combined fast+slow model reaches 94.2% FIT, compared with 69.5% for the weighted slow ensemble alone and 61.4% for the standalone online GP model (Giuli et al., 16 Jul 2025).

These results do not establish a single universal advantage. Rather, they show that the fast-slow decomposition is repeatedly used to trade off short-term adaptability against long-term stability under markedly different objectives, architectures, and hardware assumptions.

6. Conceptual boundaries, misconceptions, and limitations

A common misconception is that “slow” necessarily means longer outputs or greater token count. Gradient analysis of LLM post-training shows a narrower result: for reasoning tasks, detailed chain-of-thought produces smaller, smoother, and more layer-consistent gradients than no CoT, but on knowledge-learning tasks simply increasing response length does not reproduce the same slow-thinking gradient signature (Li et al., 2024). The same study also reports that, without CoT, correct and irrelevant responses can yield almost identical gradient behaviors, whereas detailed CoT makes the gradients more discriminative (Li et al., 2024).

Another misconception is that FSTS always corresponds to two separately trained models. Some methods do use dual branches or modules, but others do not. FAST is a precision schedule over one training system (Zhang et al., 2021). Multirate training is a partition of one parameter vector (Vlaar et al., 2021). FiS-VLA explicitly criticizes separate-model dual-system designs and instead embeds System 1 within System 2 by partial parameter sharing (Chen et al., 2 Jun 2025). Gold-Switch goes further by approximating fast-slow behavior at inference time through low-rank superposition and hard or soft switching, and it is explicitly training-free (Lee et al., 8 Oct 2025).

The limitations are correspondingly heterogeneous. Fast Forward does not work well in full-rank standard finetuning, where a single simulated step often immediately increases loss (Rahamim et al., 2024). FSCO still depends strongly on the base discriminator learning rate, reports overfitting in later stages, and failed on a $\Phi$2 network (Wang et al., 21 Apr 2025). LLM FST notes higher compute cost than RL-only because GEPA cycles add overhead (Tiwari et al., 12 May 2026). The two-fold model adaptation framework for energy systems does not provide a formal stability or convergence theory for the full two-layer architecture (Giuli et al., 16 Jul 2025). Gold-Switch requires access to both base and tuned weights and depends on a classifier for difficulty prediction; soft interpolation was reported as less effective than hard switching (Lee et al., 8 Oct 2025).

Taken together, the literature supports a restrained conclusion. FSTS is a recurrent strategy for decomposing learning across time scales, but its implementation is domain-specific: it may be realized through schedules, dual streams, multirate parameter updates, iterative refinement, memory-driven reflection, or prompt-weight co-optimization. The shared objective is consistent across these forms: make rapid adaptation possible without forcing all useful behavior into the same update mechanism or the same temporal horizon (Wang et al., 23 Sep 2025, Tiwari et al., 12 May 2026, Zhang et al., 2021).