Parallel Scheduled Sampling in ML

• Parallel Scheduled Sampling is a training strategy that replaces sequential token-by-token history with full-sequence parallel passes to mitigate exposure bias.
• It is applied in auto-regressive models, transformer-based speech recognition, and diffusion models, adapting the mixing process to each domain's structure.
• Empirical results demonstrate that moderate mixing probabilities and limited passes can significantly improve performance while preserving parallel computation efficiency.

Parallel Scheduled Sampling denotes a family of training procedures that preserve the central purpose of scheduled sampling—reducing the train–test mismatch induced by teacher forcing—while avoiding the fully sequential roll-in that makes conventional scheduled sampling expensive on parallel hardware. In the auto-regressive formulation, the method replaces token-by-token history construction with one or more parallel passes over the entire target sequence (Duckworth et al., 2019). In transformer-based speech recognition, the same label is used for a token-level mixer that combines ground truth with precomputed or online hypotheses before a single parallel decoder pass (Zhou et al., 2019). In diffusion models, the idea is transplanted to the denoising time axis by replacing states drawn only from the forward process with states obtained partly from short model rollouts (Deng et al., 2022). This suggests that “Parallel Scheduled Sampling” is best understood as a general strategy for exposure-bias mitigation under parallelizable training constraints, rather than as a single invariant algorithm.

1. Definition, rationale, and scope

Teacher forcing trains an auto-regressive model on the conditional likelihood

$$p_\theta(y \mid x)=\prod_{t=1}^T p_\theta(y_t \mid y_{1:t-1}, x),$$

while always conditioning on the gold history during training. At inference, the same model conditions on its own previous predictions. Scheduled sampling addresses this discrepancy by replacing some ground-truth history with model predictions during training, but standard scheduled sampling is inherently sequential because the prediction used at step $t$ depends on the mixed history realized at steps $<t$ (Duckworth et al., 2019).

In the transformer setting, the difficulty is sharper because the decoder is normally trained on the full target sequence in one masked parallel pass. The speech-recognition formulation states the problem directly: “vanilla transformer, when applied with scheduled sampling, is slower than teacher-forcing training due to the destructed parallelism and duplicated calculations” (Zhou et al., 2019). The diffusion formulation makes the same point in different language: diffusion denoising is sequential in time but parallel in space, and training only on states sampled from the forward process creates an exposure-bias analogue because inference follows the model’s own reverse chain (Deng et al., 2022).

Setting	Parallel unit	Core mechanism
Auto-regressive sequence models	All positions within each pass	Run one or more full passes, sample predictions for all positions, then mix in parallel (Duckworth et al., 2019)
Transformer ASR	Decoder inputs across all positions	Mix ground truth with pre-existing or self-decoded hypotheses before the decoder pass (Zhou et al., 2019)
Diffusion models	Full latent state at a timestep	Train on a mixture of forward-process states and short model-rollout states (Deng et al., 2022)

A common misconception is that scheduled sampling becomes “parallel” only if autoregression is removed. The cited literature does not make that claim. Instead, it restructures training so that the stochastic replacement of gold context is performed at the level of full-sequence passes, full decoder inputs, or full denoising states, while the underlying model class remains auto-regressive or iterative.

2. Canonical formulation in auto-regressive sequence models

The canonical formulation appears in “Parallel Scheduled Sampling” (Duckworth et al., 2019). Let $y_{0,t}=y_t$ initialize the conditioning sequence with the ground-truth target. For pass $k=1,\dots,K$, the method first performs a parallel sampling step: $$\hat{y}_{k,t} \sim p_\theta(y_t \mid y_{k-1,1:t-1}, x),$$ for all $t$ simultaneously under a fixed conditioning sequence $y_{k-1,1:T}$. It then performs a parallel mixing step: $$y_{k,t}= \begin{cases} y_{k-1,t} & t<k,\ y_t & \text{with probability } 1-p,\ \hat{y}_{k,t} & \text{with probability } p. \end{cases}$$ After $K$ passes, training uses

$t$0

The key structural change is that the model is never asked to generate the mixed history token by token inside a single forward pass. Instead, each pass sees a complete conditioning sequence, so all positions can be processed in parallel. The method therefore trades a sequential scan over time for a small number of full-sequence passes. In practice, the paper reports that most experiments use $t$1 (Duckworth et al., 2019).

The same paper proves a limiting equivalence result: when the mixing probability is $t$2 and the number of passes satisfies $t$3, the distribution over the final conditioning sequence $t$4 equals the distribution of full sample decoding from the model. This result is important conceptually because it places teacher forcing, scheduled sampling, and free-running sampling on a continuum parameterized by $t$5 and $t$6. It does not imply that practical deployments use $t$7 or $t$8; the reported experiments do not.

The computational argument is central. Within a pass, PSS uses the same parallelism profile as teacher-forced training: one full forward computation over the sequence, followed by an elementwise Bernoulli mix. The sequential dependence is only across passes. This is why the method is practical on accelerators whereas conventional scheduled sampling is not.

3. Transformer-specific realizations

A closely related transformer-specific line appears in “Scheduled Sampling for Transformers” (Mihaylova et al., 2019). That work introduces a two-pass decoding strategy rather than the pass-indexed construction above, but the underlying design principle is the same: generate model predictions for all positions in a first pass, construct a mixed reference sequence in parallel, and train a second pass on that sequence. The first pass is teacher forced; the second pass consumes embeddings that are either gold embeddings or prediction-derived embeddings. The paper evaluates argmax, top-k mix, softmax mix, Gumbel-softmax mix, and sparsemax mix, with a linear decay schedule

$t$9

It reports that mixed-embedding variants achieve performance close to a teacher-forcing baseline, while argmax replacement is substantially worse, and backpropagation through both passes underperforms the baseline.

In speech recognition, “Improving Generalization of Transformer for Speech Recognition with Parallel Schedule Sampling and Relative Positional Embedding” formalizes a different transformer realization (Zhou et al., 2019). Here PSS does not generate the replacement tokens inside the current decoder pass. Instead, it assumes a pre-existing hypothesis sequence $<t$0 that approximates the decoder’s inference-time error distribution, and constructs the decoder input in advance: $<t$1 The teacher-force rate follows a piecewise linear decay: $<t$2

That paper distinguishes two ways of simulating the error distribution used to form $<t$3. PSS-CM uses an external Kaldi TDNN-LSTM chain model. PSS with self-decoding uses the transformer itself, either offline from a teacher-forcing baseline or online within training. The online variant states that “the hypothesized results are obtained by greedy search decoding with ground truth $<t$4 as decoder input.” The token-level formulation is emphasized because “the token level is closer to the real scheduled sampling.”

The ASR paper also places its online variant in relation to the multi-round formulation of Duckworth et al. In its terminology, teacher forcing corresponds to $<t$5, its PSS-online corresponds to $<t$6, and conventional scheduled sampling is recovered in the limit $<t$7, at which point the time cost becomes approximately the same as conventional scheduled sampling. The significance is that transformer PSS can interpolate between a single parallel perturbation stage and a progressively more faithful simulation of stepwise roll-in.

4. Diffusion-model adaptation

“Markup-to-Image Diffusion Models with Scheduled Sampling” redefines Parallel Scheduled Sampling for diffusion training (Deng et al., 2022). The paper treats diffusion denoising as a sequential decision process. Training draws $<t$8 from the fixed forward process $<t$9, while inference traverses the learned reverse chain, so the state distribution encountered at test time can differ substantially from the one used in training. The paper explicitly attributes layout mistakes to “error propagation in the sequential Markov chain.”

The adaptation replaces states drawn purely from the forward process with an approximate model-based state distribution. For a finite rollout horizon $y_{0,t}=y_t$0, the paper defines

$y_{0,t}=y_t$1

When $y_{0,t}=y_t$2, this reduces to standard diffusion training. When $y_{0,t}=y_t$3 and the rollout begins from the terminal Gaussian state, it recovers the true model marginal at time $y_{0,t}=y_t$4. In experiments, the paper uses $y_{0,t}=y_t$5.

The training rule is then simple: for some fraction of training instances, sample the training input state from $y_{0,t}=y_t$6 instead of $y_{0,t}=y_t$7, and keep the usual denoising target anchored to the ground-truth clean image $y_{0,t}=y_t$8. The schedule acts over training epochs, not diffusion timesteps: the paper states, “We linearly increase the rate of applying scheduled sampling from 0% to 50% from the beginning of the training to the end.” Gradients through $y_{0,t}=y_t$9 are ignored for efficiency.

This diffusion usage is parallel in a different sense from the auto-regressive case. The denoising network updates an entire image state at once, so scheduled sampling chooses whether the full noisy image at time $k=1,\dots,K$0 comes from the expert forward process or from a short model rollout. The sequential axis is diffusion time, while the network evaluation remains spatially parallel. The paper therefore imports the scheduled-sampling idea without reverting to tokenwise autoregression.

5. Empirical behavior across domains

The original auto-regressive PSS paper evaluates dialog response generation, summarization, image generation, and machine translation (Duckworth et al., 2019). Its headline results are task-dependent. In dialog response generation, PSS achieves “1.6 BLEU score (11.5%) improvement over teacher-forcing.” In image generation, it achieves “20% and 13.8% improvement in Frechet Inception Distance (FID) and Inception Score (IS) respectively.” On CIFAR-10, teacher forcing gives FID $k=1,\dots,K$1 and IS $k=1,\dots,K$2, while PSS with $k=1,\dots,K$3 and one pass gives FID $k=1,\dots,K$4 and IS $k=1,\dots,K$5. In summarization, PSS improves or matches teacher forcing in several settings, especially under greedy decoding. In WMT’14 En–De, teacher forcing yields BLEU $k=1,\dots,K$6 and PSS yields BLEU $k=1,\dots,K$7, so no improvement is observed. The same paper reports that teacher forcing and PSS are “two orders of magnitude faster” than sequential scheduled sampling in dialog, and that a single pass of PSS is approximately $k=1,\dots,K$8 slower than teacher forcing.

The transformer-ASR formulation reports results on a 10,000-hour Chinese dictation corpus (Zhou et al., 2019). The transformer baseline with absolute positional embedding gives CER $k=1,\dots,K$9 on short utterances and $$\hat{y}_{k,t} \sim p_\theta(y_t \mid y_{k-1,1:t-1}, x),$$0 on long utterances. PSS alone improves short-utterance CER to the $$\hat{y}_{k,t} \sim p_\theta(y_t \mid y_{k-1,1:t-1}, x),$$1 range, which the paper summarizes as about a $$\hat{y}_{k,t} \sim p_\theta(y_t \mid y_{k-1,1:t-1}, x),$$2 relative improvement, but long-utterance gains are modest: for example, PSS-offline with $$\hat{y}_{k,t} \sim p_\theta(y_t \mid y_{k-1,1:t-1}, x),$$3 gives LU CER $$\hat{y}_{k,t} \sim p_\theta(y_t \mid y_{k-1,1:t-1}, x),$$4. The major long-utterance improvement comes from combining PSS with Relative Positional Embedding. RPE alone with encoder+decoder RPE gives SU $$\hat{y}_{k,t} \sim p_\theta(y_t \mid y_{k-1,1:t-1}, x),$$5 and LU $$\hat{y}_{k,t} \sim p_\theta(y_t \mid y_{k-1,1:t-1}, x),$$6, while combining RPE with PSS-online ($$\hat{y}_{k,t} \sim p_\theta(y_t \mid y_{k-1,1:t-1}, x),$$7, $$\hat{y}_{k,t} \sim p_\theta(y_t \mid y_{k-1,1:t-1}, x),$$8) gives SU $$\hat{y}_{k,t} \sim p_\theta(y_t \mid y_{k-1,1:t-1}, x),$$9 and LU $t$0. The abstract summarizes the combined effect as “a 7% relative improvement for short utterances and a 70% relative gain for long utterances.”

The diffusion adaptation reports consistent improvements across four markup-to-image datasets: mathematical formulas, table layouts, sheet music, and molecular images (Deng et al., 2022). On Math, the best baseline (+Attn,+Pos) gives DTW $t$1 and RMSE $t$2, while scheduled sampling gives DTW $t$3 and RMSE $t$4. On Table Layouts, the corresponding change is DTW $t$5 and RMSE $t$6. The improvements are smaller on Sheet Music and Molecules but remain consistent. The paper also reports that training speed is almost unchanged on Math: $t$7 per epoch with scheduled sampling versus $t$8 without it.

Taken together, these results show a recurring pattern. Parallel scheduled sampling often improves robustness under greedy or self-conditioned decoding, but its effect size depends strongly on the source of generalization failure. In the ASR study, PSS addresses history noise, whereas RPE addresses positional extrapolation; only the combination yields the large long-utterance gain. In machine translation, where the paper did not observe a gain, teacher forcing already remained competitive.

6. Interpretation, limitations, and relation to adjacent ideas

The literature uses the same phrase for several non-identical mechanisms. In auto-regressive generation, PSS is pass-based mixing over discrete histories (Duckworth et al., 2019). In transformer ASR, it is token-level mixing between gold text and an externally or internally supplied hypothesis stream (Zhou et al., 2019). In diffusion, it is state-level replacement along the denoising trajectory (Deng et al., 2022). A useful interpretation is therefore architectural rather than nominal: parallel scheduled sampling preserves the scheduled-sampling objective of training under error-like contexts, but relocates the stochastic intervention to a level that can be processed in parallel.

A second misconception is that PSS is equivalent to conventional scheduled sampling in ordinary use. The sources do not support that claim. The general auto-regressive formulation matches full sample decoding only in the limit $t$9 and $y_{k-1,1:T}$0 (Duckworth et al., 2019). The ASR formulation states that conventional scheduled sampling is recovered only in the limit $y_{k-1,1:T}$1, with approximately the same time cost as the conventional method (Zhou et al., 2019). Practical regimes typically use $y_{k-1,1:T}$2 or $y_{k-1,1:T}$3, moderate mixing probabilities, and substantial teacher-forcing warm-up.

The limitations are also consistent across papers. The original PSS paper finds that very high mixing probability can degrade performance severely; in dialog, $y_{k-1,1:T}$4 yields much worse BLEU than moderate $y_{k-1,1:T}$5 (Duckworth et al., 2019). The ASR paper states that schedule sensitivity matters, that very low $y_{k-1,1:T}$6 can cause under-fitting, and that PSS alone does not dramatically fix long-utterance issues (Zhou et al., 2019). The diffusion paper notes that scheduled sampling reduces but does not eliminate failure modes, and that improvements on Sheet Music remain modest (Deng et al., 2022).

Finally, Parallel Scheduled Sampling should not be conflated with the later literature on parallel sampling algorithms. “Parallel simulation for sampling under isoperimetry and score-based diffusion models” studies time-parallel Picard methods for simulation depth reduction (Zhou et al., 2024). “Parallel Sampling via Counting” and “Parallel Sampling via Autospeculation” study parallel inference-time sampling from counting or denoising oracles, including exact or approximate samplers with sublinear round complexity (Anari et al., 2024, Anari et al., 11 Nov 2025). These works concern parallel sampling itself rather than train-time exposure-bias mitigation. This suggests a useful conceptual split: scheduled sampling alters the training distribution of contexts or states, whereas parallel sampling algorithms alter the runtime structure of generation or simulation.