Alignment Drift in Single-Model Speculative Decoding for ASR: Mechanism, Correction, and Cost

Abstract: Speculative decoding speeds up generation by letting a cheap draft propose several tokens that a target model checks in one pass. In the single-model form, the draft is a lightweight module attached to the target rather than a separate model. Applying this design to Automatic Speech Recognition (ASR) introduces an extra problem. The draft can read the whole audio at every step, yet its proposals get worse as it runs on its own. Access is not localization. The accepted text keeps the transcript position explicit, but the draft must also track the changing audio position. In the primary matched comparison, per-step audio access changes the first proposal modestly but roughly doubles later-proposal acceptance. Fixed-width windows show that the audio position explains part of this gap. A correctly placed window recovers continuation, while an equally narrow window at the wrong position reduces it. Late-draft median error reaches 21 frames in the hardest reported condition, while target attention during verification stays within a 2-frame median. We test two ways to reduce this drift. The first reads the audio position from verification attention and uses it to guide the next draft round. It saves time only when the extra accepted tokens offset the readout cost. The second is AnchorDraft, which teaches the draft to track the audio position during training without changing the inference graph. The trained draft improves end-to-end speed at both tested target scales. These results show that ASR self-speculation depends on token prediction, audio-position tracking, and draft cost.

• The paper identifies alignment drift as a major cause of continuation failures, showing that correct audio-window positioning raises depth-two conditional acceptance by 0.254 and improves survival.
• The authors demonstrate that verification-attention readouts and AnchorDraft training partially correct drift, delivering mean speed gains of up to 5.0% and 7.2% respectively without changing the inference graph.
• The results show that speculative ASR performance depends on balancing token quality, audio-position tracking, and draft cost, while remaining sensitive to model architecture, batch size, audio duration, and numerical precision.

Single-model speculative decoding attaches a lightweight draft module to a frozen ASR target, reusing its tokenizer, encoder output, and KV cache instead of serving a second recognizer. This paper identifies and characterizes a failure mode specific to this design: the draft reads the entire encoded audio at every step, but its attention drifts away from the audio frame corresponding to the next token as it runs autonomously between target passes. The authors term this alignment drift and show that it is a measurable, partially causal, and partially correctable source of acceptance loss (2608.12703).

Problem setup and evaluation discipline

The system follows the EAGLE-3 recipe: the draft combines frozen target features with the current token embedding through a causal block containing self-attention, cross-attention to the frozen audio representation, and a feed-forward layer. Drafts are trained for two Qwen3-ASR scales (0.6B and 1.7B targets) and Voxtral-Mini, with Whisper as a cross-architecture check. Evaluation covers LibriSpeech clean/other, TED-LIUM, GigaSpeech, and FLEURS, and all headline latency numbers come from a real cached loop on a single A100 at batch size one in bfloat16, starting from the waveform.

A notable methodological contribution is the demonstration that cheaper evaluation protocols overstate performance. Mean accepted length falls from 9.4 under offline fixed-prefix scoring to 4.83 under chained survival and 4.35 in the real cached loop; restart acceptance drops from 0.92 to 0.635. The paper also concedes that lossless verification holds only under exact arithmetic: in bfloat16, token agreement with a separately computed target trace is 0.9705–0.9966, and float32 recomputation changes the argmax in 0.14–0.31% of positions. WER differences remain small and sign-mixed, but the "lossless" claim is qualified rather than absolute.

Localizing the loss: restart versus continuation

The central diagnostic splits each speculative round into the restart proposal (immediately after verification) and continuation proposals (subsequent draft-only steps). In matched comparisons differing only in per-step audio cross-attention, restart acceptance changes modestly while continuation acceptance roughly doubles. For the Qwen 1.7B target, continuation acceptance rises from 0.25–0.32 without audio access to 0.54–0.58 with it, and speedup from 1.14–1.30× to 1.41–1.55×; for Voxtral 3B, continuation rises from 0.22–0.25 to 0.71–0.78, with speedup from roughly parity to 1.42–1.69×. The benefit of audio access therefore develops between target passes, not at the first proposal.

A cost boundary sharpens the picture: a full Qwen3-ASR-0.6B recognizer used as a draft achieves near-perfect acceptance (restart 0.941–0.973, continuation 0.963–0.978) yet runs at only 0.59–0.70× autoregressive speed. Proposal quality alone does not pay for its computation, which motivates studying the shallow draft's failure rather than simply strengthening the draft.

Mechanism: anchor error grows with draft depth

The paper operationalizes the draft's audio position as the peak of its cross-attention and validates this measure against the MMS forced aligner (MMS-FA), finding pooled correlations of 0.74–0.78 for Qwen-1.7B across evaluation sets. Voxtral is a boundary case: its correlations are higher (0.93–0.95) but its attention peak is systematically late by roughly 165–205 ms, so correlation does not imply exact localization.

During free-running continuation, anchor error grows while survival falls: in the official-split runs, unrestricted draft error increases at +3.18 frames per step (95% CI [+2.99, +3.37]) with conditional acceptance falling at −0.118 per step. In the hardest reported condition, late-draft median error reaches 21 frames (roughly 1.7 seconds at ~80 ms/frame), whereas target verification attention stays within a 2-frame median. Rejected proposals show larger anchor error than accepted ones at every depth, with large-error rejection odds ratios of 3.67 at depth one, decaying to 1.43 (interval including zero) at depth four. The authors are explicit that this association does not explain every rejection.

Linear probes locate position information in intermediate decoder layers (peaking at layers 18/28, 17/28, and 13/30 for Qwen-1.7B, Qwen-0.6B, and Voxtral-3B, then dropping 64–70% toward the output), and post-verification states carry substantially more readable position than draft states at matched restart events (paired probe difference +0.158 R²). The paper correctly notes that probe readability does not establish causal use.

Causal test: matched window-position interventions

The key experiment fixes the visible window width (±400 ms half-width) and changes only its center across five conditions: unrestricted, oracle (correct, from forced alignment), pointer (extrapolated from verified frames), random, and shifted. At depth two, the correct-versus-shifted conditional-acceptance contrast is +0.254 [0.241, 0.268]. Correct centering raises cumulative survival $s_2$ by +0.105 and mean accepted length $L$ by +0.262, while the equally wide shifted window lowers $s_2$ by −0.122 and $L$ by −0.370. The direction holds for both seeds on every evaluation set.

Two qualifications matter. First, survival remains well below one under correct centering, so position explains only part of continuation failure. Second, the effect is checkpoint-dependent: one 0.6B comparison checkpoint responds weakly to any window, and a feature-noise control does not account for the gap, leaving the cause unidentified. Voxtral provides a second boundary: it distinguishes correct from wrong centers, yet restricting attention reduces acceptance even at the correct position. Position sensitivity and loss of position during continuation are therefore separable properties, and a useful correction requires both.

Alternative draft changes fail to close the gap: later target layers (−0.037 continuation), a two-layer draft (≈0), soft KL training (−0.111), and tree drafting all lose to the default one-layer cross-entropy draft on the quality-cost tradeoff; scheduled sampling gains +0.054 continuation but hurts restart and two-step survival.

Correction one: verification-attention readout

Because forced alignment is unavailable at inference, the paper screens runtime position sources. A speech-rate prior reproduces only 17–32% of correct-window behavior; a previous-round frame reproduces 64–66% but costs 0.8–5.3% in speed; the target's audio attention during verification reproduces 83–86% and is the only source that clears the cost threshold, yielding a +3.4% five-set mean speed gain at 1.7B (clean-set interval includes zero). The implementation averages attention over heads at layer 21—selected on development data and consistent across both independently trained 1.7B drafts—reading from the existing cache with no extra target pass.

The cost condition is made explicit: with target verification at 86–89% of wall-clock and drafting at 6.3–9.0%, a correction pays only when $\Delta L/(1+L)$ exceeds $\Delta C_{\text{corr}}/(1+K\rho)$, with $\rho = 0.0275$ at batch one. This rule matches the sign of 10/10 deployed evaluation-set point estimates across the two Qwen scales, but it tests sign, not magnitude. Net gains are +3.4% (1.7B) and +5.0% (0.6B) on the five-set mean, with four of five per-set intervals positive at each scale. The scope is explicitly bounded: the draft-to-verification cost ratio rises from 0.0275 at batch one to 0.1274 at batch 64, and speedup falls as audio duration grows from ~18 to ~104 seconds, so the result describes short-form, batch-one serving.

Correction two: AnchorDraft

AnchorDraft adds a training-only guided-attention loss (in the spirit of Tachibana et al.'s guided attention) matching the draft's first-layer head-averaged audio attention to a Gaussian centered on the forced-alignment frame of the next token, with $\lambda = 0.1$ and $\sigma = 5$ fixed on a holdout. The inference graph is unchanged, and verification remains greedy. Results concentrate exactly where the diagnostic located the problem: continuation acceptance improves by +0.111 (0.6B) and +0.050 (1.7B) versus restart improvements of +0.021 and +0.009, with maximum absolute WER change of 0.01 percentage points. Five-set speed gains span +6.4–7.2% (0.6B seeds) and +3.9–5.1% (1.7B seeds). The Voxtral control, whose position error is stable during continuation, changes by −0.011, consistent with the mechanism claim.

An overlap experiment ties the two corrections to the same mechanism: after AnchorDraft training, the residual correct-position accepted-length gain falls by 24–30% (intervals excluding zero) but remains positive. Stacking the runtime readout on top yields absolute speedups of 1.472× (control), 1.531× (AnchorDraft), and 1.562× (stacked) over autoregressive decoding, with the stacked increment of +1.8% having CI [+0.9, +2.7]. The two corrections partially overlap but remain complementary.

Limitations and open questions

The paper is careful about what its evidence does not establish. The position intervention explains only part of rejected proposals; the causal contrast at depth four is unsupported because fewer than 2% of rounds survive that far. Checkpoint sensitivity of the position response is observed but not explained, and the feature-noise control does not resolve it. AnchorDraft requires token-to-frame supervision from a forced aligner, limiting applicability to settings where such supervision is available. The runtime correction is validated only on two Qwen scales under batch-one, short-form decoding with a specific readout layer; Voxtral and other architectures fail at least one screening condition. Extension to other continuous-memory tasks is stated as a hypothesis, not a result. Finally, low-precision verification is not token-exact, so the lossless property holds only approximately in deployment.

Conclusion

The paper decomposes single-model speculative decoding for ASR into three interacting factors—token prediction, audio-position tracking, and draft cost—and shows that the second factor, alignment drift, accounts for a substantial and partially recoverable share of continuation failure. The matched window interventions provide a clean causal estimate (+0.254 conditional acceptance at depth two), the verification-attention readout supplies a cost-effective runtime position source, and AnchorDraft demonstrates that the drift can be trained away without changing the inference graph. The residual open questions—checkpoint-dependent position sensitivity, the unexplained portion of rejections under correct positioning, and transfer to batched or long-form serving—are clearly delimited by the paper's own measurements.