---
title: Fast-Slow Training Strategy (FSTS)
url: https://www.emergentmind.com/topics/fast-slow-training-strategy-fsts
type: topic
---

# Fast-Slow Training Strategy (FSTS)

Searching arXiv for recent papers on fast-slow training strategies and closely related terminology.
arXiv search query: all:"Fast-Slow Training Strategy" OR all:"fast-slow" OR ti:"Learning, Fast and Slow"
Fast-Slow Training Strategy (FSTS) denotes a family of training and adaptation schemes that separate learning into components, phases, or parameter groups operating at different time scales. Across the cited literature, the fast component is used for rapid task-specific adaptation, low-latency decision-making, or inexpensive early optimization, whereas the slow component is used to preserve stability, generalizable structure, contextual reasoning, or higher-fidelity refinement. The term is used explicitly in some papers—notably in object re-identification, online preference alignment, test-time adaptation, and continual LLM post-training—but closely related formulations also appear under names such as “Fast First, Accurate Second Training (FAST),” “multirate training,” “Fast Forward,” and “Fast-in-Slow” [2509.18715] [2406.05534] [2605.12484] [2110.15456].

## 1. Definition and terminological scope

In the broadest sense represented in the literature, FSTS is not a single algorithm but a design principle: allocate different parts of learning to different temporal scales. The “fast” side may be a rapidly updated branch, a low-precision early phase, a prompt population, a local online corrector, or a low-latency encoder. The “slow” side may be an EMA teacher, model parameters updated conservatively, a larger-context branch, an offline ensemble, or an iterative solver that refines fast outputs [2509.18715] [2605.12484] [2412.15415] [2507.12187].

The exact semantics of “fast” and “slow” vary by domain. In APC for object ReID, FSTS means a Fast Update Stream (FUS) and a Slow Update Stream (SUS), where FUS rapidly captures ReID-specific cues and SUS preserves the general visual perception inherited from CLIP [2509.18715]. In LLM post-training, Fast-Slow Training (FST) defines slow weights as model parameters $\theta$ and fast weights as optimized textual context $\phi$ or a prompt population $\Phi$ [2605.12484]. In FAST for DNN training, the paper does not use the label FSTS as its exact title, but it explicitly describes a “Fast First, Accurate Second Training” policy in which training starts with low-precision BFP and becomes more precise across training iterations and layer depth [2110.15456].

This terminological variability is itself important. Some works use the expression for training proper; others apply the same fast-slow logic to test-time adaptation, online control, or inference-time regulation. “Fast-Slow Test-Time Adaptation” for VLN is an adaptation framework rather than an offline training method [2311.13209]. “Gold-Switch” is explicitly a training-free deployment-time superposition method rather than a training algorithm [2510.06750]. A plausible implication is that FSTS is best understood as a recurrent systems pattern rather than a single standardized method.

## 2. Genealogy of the fast-slow idea

An early explicit formulation appears in cooperative conditional learning, where a fast thinking initializer generates an output directly from the condition and latent noise, while a slow thinking solver refines that output under a conditional energy-based model by Langevin dynamics [1902.02812]. The initializer learns from how the solver refines its samples, and the solver learns from the difference between refined outputs and observed data. This established a teacher-like division between amortized generation and iterative refinement.

Subsequent work specialized the same idea for transfer learning, optimization, and systems training. “Fast And Stable Task-adaptation” argued that ordinary fine-tuning is too aggressive at the beginning because a randomly initialized classifier can cause catastrophic forgetting before the target task is learned. Its remedy was fast adaptation of the appended classifier and slow, stable adaptation of the pretrained backbone through near-zero classifier initialization and asymmetric learning rates [2007.01388]. “Multirate Training of Neural Networks” formalized the fast-slow split as parameter partitioning into $\theta_F$ and $\theta_S$, with fast parameters updated every step and slow parameters updated every $k$ steps, primarily for transfer learning speed-up and regularization [2106.10771].

In hardware-aware DNN training, FAST translated the fast-slow idea into numerical precision scheduling. Weights, activations, and gradients are represented in variable-precision block floating point (BFP), with low precision used early and higher precision introduced later and in deeper layers [2110.15456]. In online and continual settings, the same pattern was adopted for preference alignment, test-time adaptation, and continual LLM learning. OFS-DPO used two identical LoRA modules with different optimization speeds and a coupling regularizer [2406.05534]. FSTTA for online VLN alternated a frequent gradient-based fast phase with a periodic parameter-trajectory slow phase [2311.13209]. LLM FST separated adaptation into fast textual context and slow parametric learning [2605.12484].

By 2025–2026, fast-slow constructions had become increasingly architectural. FiS-VLA embedded a fast executor inside a slow VLM-based reasoning model for manipulation [2506.01953]. JSTAR used a cascaded fast encoder for ASR and slow encoder for ST [2412.15415]. Slow4Fast-VLN introduced fast execution plus slow reflection and structured experience extraction for open-world navigation [2601.09111]. In parallel, fast-slow concepts also appeared in adaptive optimization and deployment control, as in Fast Forward for low-rank finetuning and Gold-Switch for training-free switching between fast- and slow-thinking behavior [2409.04206] [2510.06750].

## 3. Canonical mechanisms

Despite heterogeneity across domains, several recurring mechanisms appear repeatedly.

First, many FSTS variants use **dual-timescale parameter updates**. In APC, FUS is updated by ordinary gradient descent, while SUS is updated by exponential moving average and periodically provides visual and attribute-aware identity prototypes that supervise FUS through contrastive loss [2509.18715]. In multirate training, slow momentum is updated once every $k$ steps, while fast parameters are updated at each inner step and slow parameters are drifted linearly using stored slow momentum; the scheme uses coupled step sizes with $h_S = k h_F$ [2106.10771]. In continual LLM post-training, the joint objective is
\[
\max_{\theta,\,\Phi}\;\; J(\theta,\Phi) = \mathbb{E}_{x\sim\mathcal{D},\;\phi\sim U(\Phi),\;y\sim\pi_\theta(\cdot\mid x,\phi)} \!\left[r(x,y)\right],
\]
so task adaptation is split between slow weights $\theta$ and fast prompt population $\Phi$ [2605.12484].

Second, some FSTS variants use **schedule-based resource allocation** rather than two persistent models. FAST defines an adaptive precision schedule between 2-bit and 4-bit mantissa BFP using
\[
\epsilon(l,i) = \alpha-\beta\frac{i}{I}-\beta\frac{l}{L}
\]
and a relative improvement score
\[
r(X) = \frac{\sum_{n}|BFP(X_{n},4)-BFP(X_{n},2)|}{\sum_{n}|BFP(X_{n},2)|}.
\]
If $r(X) < \epsilon(l,i)$, 2-bit mantissa BFP is used; otherwise 4-bit is used [2110.15456]. Here, “fast” means cheaper early and shallow computation, whereas “slow” means later, deeper, more accurate computation.

Third, several formulations use **fast generation plus slow refinement**. In cooperative conditional learning, the generator samples
\[
Y = g(X, C; \alpha) + \epsilon,
\]
while the slow solver refines by Langevin dynamics,
\[
Y_{\tau+1} = Y_\tau + \frac{\delta^2}{2} \frac{\partial}{\partial Y}f(Y_\tau,C;\theta) + \delta U_\tau.
\]
The initializer then regresses toward the refined samples [1902.02812].

Fourth, there are **optimizer-level fast-slow schemes**. Fast Forward alternates ordinary Adam SGD steps with a Fast Forward stage that repeatedly applies the latest update direction
\[
\Delta_{\mathbf{W}} = \mathbf{W}_t - \mathbf{W}_{t-1}, \qquad \mathbf{W}_t^{(\tau)} = \mathbf{W}_t + \tau \Delta_{\mathbf{W}},
\]
until the loss on a 32-example tiny validation set stops improving [2409.04206]. Here, the temporal split is between expensive gradient recomputation and cheap extrapolative repetition.

## 4. Major application families

The fast-slow pattern has been instantiated in substantially different technical settings.

| Setting | Fast element | Slow element |
|---|---|---|
| Conditional learning [1902.02812] | Direct conditional generator | Conditional energy-based solver with Langevin refinement |
| Variable-precision DNN training [2110.15456] | Low-precision BFP early and in earlier layers | Higher-precision BFP later and in deeper layers |
| Object ReID [2509.18715] | FUS trained by gradient descent | SUS updated by EMA and used as a prototype teacher |
| LLM post-training [2605.12484] | Optimized textual context or prompt population | Model parameters updated by RL |
| Streaming ASR+ST [2412.15415] | Low-latency encoder for ASR | Larger-context encoder for ST |
| Dynamical model adaptation [2507.12187] | Online GP residual correction | Offline ensemble over operating conditions |

In **representation learning and alignment**, the central tension is usually fast specialization versus slow preservation. APC states this as discrimination versus generalization in ReID, with FUS rapidly learning identity-specific cues and SUS retaining generalizable knowledge inherited from CLIP [2509.18715]. LLM FST states the same tension as absorbing task-specific information without catastrophic forgetting or loss of plasticity, assigning fast adaptation to optimized context and slower consolidation to model weights [2605.12484]. OFS-DPO similarly uses fast and slow LoRA modules with different optimization speeds to improve online preference alignment and mitigate forgetting in cross-domain settings [2406.05534].

In **latency-sensitive multimodal systems**, the split often matches different computational horizons. JSTAR uses a fast encoder directly connected to the frontend for low-latency streaming ASR and a slow encoder with larger context for streaming ST, with separate predictor and joiner networks for the two tasks [2412.15415]. FiS-VLA uses a slow VLM/LLM System 2 for contextual reasoning and a fast embedded System 1 execution module repurposed from the final transformer blocks, operating asynchronously at a frequency ratio of $1\!:\!4$ in the main setting [2506.01953]. Slow4Fast-VLN uses fast end-to-end navigation plus slow LLM-based reflection that extracts structured experience
\[
E = [S_t, C_s, R_s, T_n, n_s, f]^\top
\]
and reinjects it into the fast policy through retrieval and feature fusion [2601.09111].

In **numerical training and systems optimization**, the split is often between cheap approximate updates and expensive accurate ones. FAST uses low-precision early training and stochastic rounding for gradients, supported by an fMAC that processes mantissas in 2-bit chunks [2110.15456]. Multirate training updates slow parameter groups less frequently to reduce fine-tuning cost [2106.10771]. Fast Forward extrapolates recent low-rank update directions rather than recomputing full gradients [2409.04206]. In energy-system adaptation, the slow layer is an ensemble of GRU-based models over operating regimes and the fast layer is a sliding-window GP corrector for residual mismatch [2507.12187].

## 5. Empirical effects

Reported benefits of FSTS are typically expressed as improved efficiency, stability, or robustness, but the specific measured quantity depends strongly on the domain.

In variable-precision DNN training, FAST reports a **2–6$\times$ speedup** in training on a single-chip platform over prior mixed-precision or BFP approaches while maintaining similar validation accuracy; FP32 generally gives the best accuracy, but FAST-Adaptive achieves near-FP32 accuracy while using lower precision early and higher precision later [2110.15456]. In low-rank finetuning, Fast Forward reports **41–87% FLOPs reduction** and **40–81% train-time reduction** relative to standard Adam SGD, and one convergence experiment reports **56% FLOPs saved** while reaching a slightly better loss than regular Adam SGD [2409.04206]. Multirate training reports that transfer-learning models can be fine-tuned in **almost half the time** without reducing generalization performance [2106.10771].

In representation learning, APC provides a direct ablation of FSTS. On MSMT17, the baseline is **66.4 mAP / 84.4 R1**, and adding FSTS alone raises this to **75.0 mAP / 88.8 R1**; on MSMT17 $\rightarrow$ Market1501, the baseline is **16.2 mAP / 32.5 R1**, and adding FSTS raises this to **42.2 mAP / 67.6 R1** [2509.18715]. In LLM post-training, FST reaches RL’s running peak with **3.0× fewer steps** on CodeIO, **1.4× fewer steps** on Math (Polaris), and **3.0× fewer steps** on HoVer-hard, while also reaching higher fitted asymptotes and staying closer to the base policy with **up to 70% less KL divergence** at matched reward [2605.12484].

In embodied and streaming systems, FiS-VLA reports **69%** mean success on RLBench versus **61%** for CogACT and **55%** for $\pi_0$, plus a theoretical **117.7 Hz** control frequency with action chunk size 8 [2506.01953]. JSTAR reports that, on RealConv, its first-token latency is **3.3 s** versus **7.1 s** for a cascaded system, with about **2 extra BLEU points on average** [2412.15415]. FSTTA for online VLN reports that DUET receives a performance boost of **over 5%** on REVERIE/R2R-CE and that, on REVERIE validation unseen, it improves over SAR while reducing testing time by about **7%** [2311.13209]. In lifelong dynamical model adaptation, the combined fast+slow model reaches **94.2% FIT**, compared with **69.5%** for the weighted slow ensemble alone and **61.4%** for the standalone online GP model [2507.12187].

These results do not establish a single universal advantage. Rather, they show that the fast-slow decomposition is repeatedly used to trade off short-term adaptability against long-term stability under markedly different objectives, architectures, and hardware assumptions.

## 6. Conceptual boundaries, misconceptions, and limitations

A common misconception is that “slow” necessarily means longer outputs or greater token count. Gradient analysis of LLM post-training shows a narrower result: for reasoning tasks, detailed chain-of-thought produces smaller, smoother, and more layer-consistent gradients than no CoT, but on knowledge-learning tasks simply increasing response length does not reproduce the same slow-thinking gradient signature [2410.23743]. The same study also reports that, without CoT, correct and irrelevant responses can yield almost identical gradient behaviors, whereas detailed CoT makes the gradients more discriminative [2410.23743].

Another misconception is that FSTS always corresponds to two separately trained models. Some methods do use dual branches or modules, but others do not. FAST is a precision schedule over one training system [2110.15456]. Multirate training is a partition of one parameter vector [2106.10771]. FiS-VLA explicitly criticizes separate-model dual-system designs and instead embeds System 1 within System 2 by partial parameter sharing [2506.01953]. Gold-Switch goes further by approximating fast-slow behavior at inference time through low-rank superposition and hard or soft switching, and it is explicitly training-free [2510.06750].

The limitations are correspondingly heterogeneous. Fast Forward does **not** work well in full-rank standard finetuning, where a single simulated step often immediately increases loss [2409.04206]. FSCO still depends strongly on the base discriminator learning rate, reports overfitting in later stages, and failed on a **$512\times 512$** network [2504.15099]. LLM FST notes higher compute cost than RL-only because GEPA cycles add overhead [2605.12484]. The two-fold model adaptation framework for energy systems does not provide a formal stability or convergence theory for the full two-layer architecture [2507.12187]. Gold-Switch requires access to both base and tuned weights and depends on a classifier for difficulty prediction; soft interpolation was reported as less effective than hard switching [2510.06750].

Taken together, the literature supports a restrained conclusion. FSTS is a recurrent strategy for decomposing learning across time scales, but its implementation is domain-specific: it may be realized through schedules, dual streams, multirate parameter updates, iterative refinement, memory-driven reflection, or prompt-weight co-optimization. The shared objective is consistent across these forms: make rapid adaptation possible without forcing all useful behavior into the same update mechanism or the same temporal horizon [2509.18715] [2605.12484] [2110.15456].

Source: https://www.emergentmind.com/topics/fast-slow-training-strategy-fsts