Self-Speculative Biased Decoding

• Self-Speculative Biased Decoding is an inference paradigm for streaming translation that reuses prior outputs as a draft to efficiently update translations under latency constraints.
• Its core mechanism employs biased verification to selectively accept draft tokens, achieving up to 1.7× speedup while notably reducing output flicker.
• Experimental results on multiple language directions demonstrate SSBD’s plug-and-play, model-agnostic approach, with critical trade-offs tuned via the bias coefficient (β ≈ 0.2).

Searching arXiv for the cited paper and closely related self-speculative decoding work. arXiv search query: (Zeng et al., 26 Sep 2025) Self-Speculative Biased Decoding for Faster Live Translation Self-Speculative Biased Decoding (SSBD) is an inference paradigm for simultaneous re-translation in streaming LLM applications, especially live translation, in which the input stream grows over time and the output must be continually updated under a latency constraint. Instead of regenerating the translation from scratch after each input update, SSBD reuses the most recent output as a draft for the current, longer input, verifies that draft in one forward pass, biases verification toward draft-token acceptance, and then hands over to conventional autoregressive decoding at the first point of divergence. In simultaneous text-to-text re-translation, this design is reported to be model-agnostic and plug-and-play, to avoid separate draft computations, to achieve up to $1.7\times$ speedup over conventional autoregressive re-translation without compromising quality, and, with display-only mask-$k$, to reduce flickering by up to $80\%$ (Zeng et al., 26 Sep 2025).

1. Problem setting and conceptual definition

SSBD is defined in the setting of simultaneous re-translation for streaming applications. An ASR component incrementally provides source tokens, and the MT or LLM system must continually update translations as the source context expands. The objective is high quality at low latency. In this regime, conventional autoregressive re-translation is expensive because it generates one token per step, underutilizes parallel compute, is memory-bound, and re-generates the output from scratch after each input update. Standard speculative decoding addresses autoregressive latency by drafting and verifying multiple tokens per step, but it requires a separate drafter model or a tuned subnet, must compute the draft anew each time, and introduces alignment issues between drafter and target model (Zeng et al., 26 Sep 2025).

SSBD reexamines re-translation under the observation that streaming outputs often preserve long stable prefixes across successive input updates. Its central move is therefore not to construct a new draft model, but to reuse the previous output $Y^{T-1}$, produced for input $X^{T-1}$, as the draft for the next input $X^T$. This makes the draft effectively free: there is no auxiliary drafter, no subnet selection, and no per-step draft recomputation. The method is consequently described as model-agnostic and plug-and-play for latency-sensitive streaming applications (Zeng et al., 26 Sep 2025).

A key point is that SSBD is not merely self-speculative decoding in the lossless sense used by many speculative-decoding systems. Its verification stage is explicitly biased toward draft-token acceptance. That bias is introduced to increase the draft acceptance rate and to reduce flicker, meaning unnecessary revision of already displayed output. This distinguishes SSBD from lossless self-speculative frameworks such as LayerSkip, Kangaroo, and CLaSp, which preserve the base model’s output distribution by requiring exact agreement under standard verification (Elhoushi et al., 2024, Liu et al., 2024, Chen et al., 30 May 2025).

2. Core mechanism and mathematical formulation

For the $T$-th streaming update, conventional autoregressive decoding is described in the paper as

$$Y^T = (y_0, \ldots, y_n) = \prod_{i=0}^{n} F_{\text{decode}}\left(P(y_i \mid X^T, Y^T_{<i})\right).$$

SSBD approximates this process by splitting inference into two phases. First, it verifies the previous output $Y^{T-1}$ against the current input $X^T$. Second, once the model disagrees with the draft, it switches to standard autoregressive decoding from that divergence point onward. The paper expresses the approximation as verifying positions up to the draft length $k$0, then decoding autoregressively from the first disagreement position $k$1 (Zeng et al., 26 Sep 2025).

The defining modification is biased verification. For each draft token $k$2, the verification distribution is changed to

$k$3

where $k$4 is the bias coefficient and $k$5 denotes a unit mass at the draft token. Operationally, this is a convex combination of the model distribution and a point mass centered on the draft token. The acceptance rule is greedy: let

$k$6

If $k$7, the token is accepted; otherwise, the system sets $k$8, stops verification, discards the remainder of the draft, and hands off to autoregressive decoding from that position (Zeng et al., 26 Sep 2025).

This design couples latency reduction with stability of user-visible output. Because the method prefers the existing draft whenever the model remains sufficiently supportive, stable prefixes can be accepted in bulk, and already displayed tokens are less likely to be rewritten. The same bias that increases acceptance also reduces flicker, but the paper shows that this must be tuned carefully because excessive bias degrades translation quality (Zeng et al., 26 Sep 2025).

The paper further introduces display-only mask-$k$9. In ordinary mask-$80\%$0, one trims up to $80\%$1 tokens from the draft suffix to avoid unstable regions, but this reduces acceptance and speed. Display-only mask-$80\%$2 retains the full draft for verification while masking the last $80\%$3 tokens only in the UI. The result is unchanged acceptance and speed, but lower user-visible flicker (Zeng et al., 26 Sep 2025).

3. Algorithmic realization and runtime behavior

Algorithm 1 in the paper makes the control flow explicit. The prompt is constructed from the current input $80\%$4 and the previous output $80\%$5. The model then runs a forward pass over that prompt. For each draft position $80\%$6, the probability vector is modified by line 6 of the algorithm,

$80\%$7

after which the system takes

$80\%$8

If $80\%$9 differs from the draft token, the loop breaks. The algorithm then calls kv_cache_clear(i, -1) and resumes conventional step-wise model forward passes until EOS (Zeng et al., 26 Sep 2025).

Several implementation properties follow directly from that procedure. Verification processes up to $Y^{T-1}$0 draft positions in parallel in one forward pass. Divergence handling is explicit: once mismatch occurs at position $Y^{T-1}$1, cached states from that point onward are cleared before autoregressive continuation, preventing mismatched draft states from contaminating subsequent decoding. Experiments used greedy sampling for both verification and autoregressive continuation, although the paper notes that SSBD is compatible with beam search if biasing and acceptance checks are defined consistently (Zeng et al., 26 Sep 2025).

The latency behavior is tied to draft-token acceptance. The paper defines two acceptance measures: A/D, the share of accepted draft tokens among draft tokens, and A/O, the share of accepted draft tokens among all output tokens. Higher A/D and A/O reduce the number of autoregressive steps after verification and thereby increase output tokens per second (TPS). Since SSBD eliminates separate draft computations, only a single verification forward pass is required to test the reused draft, and no external drafter or subnet must be run (Zeng et al., 26 Sep 2025).

Flicker is quantified through erasure. For update $Y^{T-1}$2,

$Y^{T-1}$3

where LCP is the longest common prefix of two sequences. The paper states that normalized erasure (NE) aggregates erasures across updates and normalizes them by the final output length; lower NE indicates reduced flicker (Zeng et al., 26 Sep 2025).

4. Experimental results and ablations

The reported evaluation covers the ACL 60/60 speech translation evaluation set and the Flores-101 text benchmark. ACL 60/60 uses English audio segmented with Azure real-time speech-to-text API for streaming transcriptions; segments with long pauses were omitted, and the remaining English text streams were used as inputs. Flores-101 emulates streaming behavior with a dummy test-time lag-$Y^{T-1}$4, with $Y^{T-1}$5, feeding input prefixes in increments of $Y^{T-1}$6 words. Language directions are English$Y^{T-1}$7German, English$Y^{T-1}$8Chinese, and English$Y^{T-1}$9Japanese. The models are Qwen3-4B and Tower+ 2B, implemented in llama.cpp, with experiments on Apple Silicon (MacBook Pro 14", M2 Pro). Metrics are COMET (wmt22-comet-da), Normalized Erasure (NE), output TPS, speedup versus autoregressive re-translation, and the acceptance metrics A/D and A/O (Zeng et al., 26 Sep 2025).

The main table in the paper uses $X^{T-1}$0. For Tower+ 2B on Flores, SSBD matches or nearly matches autoregressive COMET while reducing NE and increasing TPS. The reported figures are summarized below.

Setting	Quality / flicker / speed	Acceptance
English$X^{T-1}$1German	COMET 0.877 both; NE 0.77 vs 1.17; TPS 101 vs 60; speedup 1.69	A/D 79.0%; A/O 63.1%
English$X^{T-1}$2Chinese	COMET 0.880 both; NE 1.02 vs 1.72; TPS 89 vs 59; speedup 1.50	A/D 71.3%; A/O 56.6%
English$X^{T-1}$3Japanese	COMET 0.911 vs 0.912; NE 1.49 vs 2.06; TPS 82 vs 60; speedup 1.36	A/D 60.4%; A/O 48.5%

Qwen3-4B shows the same qualitative pattern, with speedups of $X^{T-1}$4 to $X^{T-1}$5. The paper attributes the weaker gains in English$X^{T-1}$6Japanese to lower monotonicity between SVO English and SOV Japanese, which yields lower A/D, higher NE, and smaller speedups than English$X^{T-1}$7Chinese or English$X^{T-1}$8German (Zeng et al., 26 Sep 2025).

The ablation on bias coefficient $X^{T-1}$9, reported for Flores English$X^T$0Chinese with Tower+ 2B, makes the bias-quality trade-off explicit. At $X^T$1, COMET is 0.882, NE 1.53, A/D 53.8%, and speedup 1.26. At $X^T$2, COMET is 0.880, NE 1.02, A/D 71.7%, and speedup 1.42. At $X^T$3, COMET declines to 0.870, NE improves to 0.83, A/D rises to 77.3%, and speedup reaches 1.51. For $X^T$4, NE approaches 0, A/D approaches 100%, and speedup reaches 2.00, but COMET drops sharply to 0.741. The paper therefore recommends $X^T$5 as a practical starting point (Zeng et al., 26 Sep 2025).

The mask-$X^T$6 ablation on the same setting shows the distinction between masking the draft itself and masking only the display. Baseline SSBD at $X^T$7 gives COMET 0.880, NE 1.02, A/D 71.7%, A/O 57.0%, and speedup 1.42. Adding mask-$X^T$8 with $X^T$9 changes the figures to COMET 0.881, NE 0.52, A/D 80.6%, A/O 52.3%, and speedup 1.35. Adding display-only mask-$T$0 with $T$1 yields COMET 0.880, NE 0.53, A/D 71.7%, A/O 57.0%, and speedup 1.42. This is the basis for the paper’s claim that display-only mask-$T$2 can reduce flicker substantially without speed loss (Zeng et al., 26 Sep 2025).

5. Relation to speculative decoding and the meaning of “biased”

SSBD belongs to the broader family of speculative decoding, but it occupies a distinct point in that family. Classical speculative decoding accelerates generation by having a fast drafter propose several future tokens and a target model verify them, while preserving the target model’s sampling distribution through the verification rule. Self-speculative decoding removes the external drafter by deriving the draft path from the target model itself, for example by early exit, shallow subnet decomposition, or layer skipping (Elhoushi et al., 2024, Liu et al., 2024, Chen et al., 30 May 2025).

LayerSkip uses early layers to draft and remaining layers to verify and correct, with a shared LM head and exact greedy agreement under verification, yielding exact greedy equivalence to the full model under the stated assumptions (Elhoushi et al., 2024). Kangaroo uses a shallow sub-network plus a lightweight adapter and a second early-exit mechanism during drafting, while maintaining losslessness by ensuring that any emitted token is either accepted by or replaced from the base model (Liu et al., 2024). CLaSp constructs the draft by in-context layer skipping and also preserves the original distribution of generated text because verification remains standard speculative verification (Chen et al., 30 May 2025).

SSBD differs from these methods in two ways. First, its draft is not an internal subnetwork computation but the previously emitted output in a streaming re-translation loop. Second, its verification distribution is intentionally modified by the bias coefficient $T$3. The method therefore does not pursue losslessness relative to the base decoding distribution; instead, it introduces a controlled bias toward already displayed output in order to improve acceptance, reduce flicker, and accelerate re-translation (Zeng et al., 26 Sep 2025).

This distinction matters because the term “self-speculative” can otherwise suggest exactness by analogy with LayerSkip, Kangaroo, or CLaSp. In SSBD, the word “biased” is literal and central. The paper’s empirical trade-offs show that increasing $T$4 raises acceptance and speed while lowering NE, but eventually harms COMET. A plausible implication is that SSBD should be understood not as a substitute for lossless speculative decoding in general-purpose generation, but as a streaming-specific re-translation mechanism in which stability of displayed output is part of the optimization target (Zeng et al., 26 Sep 2025).

6. Limitations, failure modes, and future directions

The principal failure mode is over-biasing. Large $T$5 can force acceptance of incorrect draft tokens, harming quality; the paper reports sharp COMET degradation for $T$6–0.5, and extreme bias yields zero erasure and maximum speed but poor quality (Zeng et al., 26 Sep 2025). This makes SSBD’s main control knob simultaneously its main risk factor.

A second limitation concerns the trade-off between flicker and responsiveness. Display-only mask-$T$7 hides unstable suffix tokens from the UI and thereby lowers visible NE, but the paper notes that users may perceive slower confirmation of final words. Tuning $T$8 therefore trades visible progress against visible stability (Zeng et al., 26 Sep 2025).

A third limitation is linguistic and domain dependence. Lower monotonicity, exemplified by SOV versus SVO order differences, reduces draft acceptance and speed gains. Domain shifts can also increase divergence between the reused draft and the current optimal output, reducing the value of self-speculation based on previous outputs (Zeng et al., 26 Sep 2025).

The paper identifies several research directions: prefix training to improve monotonicity and raise acceptance; combining SSBD with self-speculative subnet approaches or other speculative frameworks to further accelerate verification; and investigating biased beam search together with adaptive $T$9 and adaptive $$Y^T = (y_0, \ldots, y_n) = \prod_{i=0}^{n} F_{\text{decode}}\left(P(y_i \mid X^T, Y^T_{<i})\right).$$0 policies (Zeng et al., 26 Sep 2025). This suggests that SSBD is best viewed as a member of a broader design space in which verification can be deliberately altered for application-specific objectives such as latency, stability, or UI smoothness, rather than only for exact reproduction of a base distribution.

In that sense, SSBD formalizes a streaming-specific compromise. It reuses the previous output as the draft, biases verification toward that draft, switches to autoregressive decoding at the first disagreement, and combines these steps with display-only masking to suppress visible instability. The resulting system departs from the lossless tradition of speculative decoding, but does so transparently and with measurable control over the speed-quality-flicker trade-off (Zeng et al., 26 Sep 2025).