FLy: Training-Free Speculative Decoding

Updated 5 December 2025

The paper introduces FLy as a training-free acceleration method that leverages entropy gating and deferred verification to enhance speculative decoding efficiency.
FLy employs techniques like layer skipping, cosine similarity-based pruning, and low-bit quantized substitutes to boost inference speed in autoregressive and diffusion models.
FLy achieves state-of-the-art speedups with over 99% accuracy recovery across domains, offering robust plug-and-play enhancements for varied model architectures.

Training-Free Loosely Speculative Decoding (FLy) is a family of lossless or nearly-lossless inference acceleration methods for LLMs that remove the requirement for model retraining or auxiliary draft models. FLy-type approaches enable multi-token speculative verification while remaining entirely plug-and-play across model architectures, domains, and data distributions. The key innovation is loosening the rigid exact-match verification constraint used in classical speculative decoding, instead leveraging entropy-driven gating, self-corrective behavior, and dynamic draft acceptance rules to maximize throughput at minimal accuracy loss. FLy achieves substantial acceleration on transformer-based autoregressive and diffusion models without compromising distribution fidelity, providing state-of-the-art practical speedup on both in-domain and out-of-distribution tasks (Li et al., 28 Nov 2025, Xia et al., 9 Oct 2024, Wang et al., 22 Sep 2025, Metel et al., 1 Oct 2024, Zhang et al., 2023, Agrawal et al., 22 Sep 2025).

1. Conceptual Foundations and Evolution

The speculative decoding (SPD) paradigm accelerates autoregressive and diffusion-based generation by having a fast draft model propose several candidate tokens (or states), then verifying them in parallel with a slower, higher-fidelity target LLM, bypassing the need for sequential single-token model invocations (Li et al., 28 Nov 2025, Xia et al., 9 Oct 2024). Classic SPD uses strict exact-match verification, discarding all draft proposals after the first mismatch.

Training-free speculative approaches—collectively referred to here as FLy—arose from the need to circumvent two limitations: the requirement for a separately trained/compatible draft model and the tendency of exact-match verification to reject plausible continuations, thus capping speedups. FLy leverages structural, probabilistic, or semantic insights—such as dynamic layer-skipping, quantized surrogate layers, adaptive context-based subnet pruning, and self-corrective deferred acceptance rules—to yield higher token acceptance, broader generalization, and higher throughput without auxiliary training.

FLy encompasses several variants, notably:

Self-Speculative Decoding with Layer Skipping, adaptively removing transformer layers in the proposed draft (Xia et al., 9 Oct 2024, Zhang et al., 2023).
Substitute Speculative Decoding (SubSpec), replacing CPU-offloaded layers with low-bit quantized GPU-resident substitutes to reduce memory transfer bottlenecks (Wang et al., 22 Sep 2025).
Adaptive Drafting via Cosine Similarity, pruning attention/MLP layers contextually by measuring hidden state redundancy (Metel et al., 1 Oct 2024).
Loosely Speculative Decoding, which introduces entropy-based gates and deferred semantic verification for drafts (Li et al., 28 Nov 2025).
Block-Graph-Based Speculative Decoding in dLLMs, using auto-speculation and directed graph verification (Agrawal et al., 22 Sep 2025).

2. Mathematical Mechanisms and Verification Schemes

FLy approaches define a two-stage process: drafting and verification.

Drafting: Use either a pruned subnetwork of the target model (layer skipping or adaptive similarity pruning), surrogate quantized layers, or fast n-gram drafters to generate token blocks or state sequences.
Verification: Employ a parallel target model pass to validate drafts, using acceptance criteria that may range from strict (exact match) to loose (entropy-gated and deferred).

A canonical FLy algorithm (Li et al., 28 Nov 2025) employs:

Entropy-Level Gate: For each draft-target mismatch at position $j$ , compute the normalized entropy

$h_j = \frac{H(p_{\mathcal M_T,j})}{\log|\mathcal V|}$

where $H(p_{\mathcal M_T,j})$ is the token-level entropy. If $h_j < \theta$ (strong certainty), strict rejection occurs; for $h_j \geq \theta$ (ambiguous), provisional acceptance with deferred semantic window is enforced.

Token-Level Deferred Window: For an ambiguous mismatch, accept provisionally, then check for further mismatches in the next $W$ positions. If no additional disagreements occur, the initial draft is considered semantically valid.

Both staged acceptance criteria are designed to capture the model’s self-corrective behavior: distinguishing true errors from plausible paraphrases. This principle enables the algorithm to accept more tokens per speculative round while maintaining near-perfect accuracy.

Mathematically, the speedup factor ( $S_{\mathrm{FLy}}$ ) is bounded by: $S_{\mathrm{FLy}} \approx \frac{\mathbb{E}[s_{\mathrm{FLy}}]}{\frac{C_D}{C_T}\mathbb{E}[s_{\mathrm{FLy}}] + 1}$ where $\mathbb{E}[s_{\mathrm{FLy}}]$ is the average token acceptance per round, $C_D$ is the amortized draft cost, and $C_T$ the target verification cost (Li et al., 28 Nov 2025).

3. Draft Model Construction and Acceleration Strategies

Deployment of FLy-type methods can use several draft construction techniques:

Layer Skipping: Selectively skip layers in the target model to create a lightweight draft. Optimization can be static (offline Bayesian search) or dynamic (on-the-fly random/Bayesian search in a context window), maximizing alignment (“matchness”) to the full LLM’s output (Xia et al., 9 Oct 2024, Zhang et al., 2023).
Cosine Similarity-Based Pruning: Measure redundancy by cosine similarity of hidden states across attention layers using the input context, pruning layers with high similarity and periodically pruning MLP layers, while safeguarding final layers (Metel et al., 1 Oct 2024).
Low-Bit Quantized Substitute Layers: In offloading scenarios, replace CPU-resident full-precision layers by low-bit (e.g., INT4) quantized versions resident on GPU, maintaining a shared KV-cache. This approach ("SubSpec") greatly reduces offload latency while achieving high token acceptance (Wang et al., 22 Sep 2025).
n-Gram Prompt Lookup Decoding: Use fast, parameter-free $n$ -gram retrieval to massively accelerate draft proposal in conjunction with looser FLy verification (Li et al., 28 Nov 2025).

The acceleration is further amplified by multi-level stacks that simultaneously speed up both the draft and verification stages. Prompt Lookup Decoding, for example, achieves order-of-magnitude faster draft generation compared to parametric drafters.

4. Empirical Performance and Generalization Characteristics

FLy methods consistently achieve high speedups while maintaining $>$ 99% accuracy recovery across model families and domains (Li et al., 28 Nov 2025, Xia et al., 9 Oct 2024, Wang et al., 22 Sep 2025). Representative results include:

Target Model	FLy Speedup	Accuracy Recovery	Draft Tokens Accepted $\tau$
Llama-3.1-70B-Instruct	2.81×	$\geq$ 99%	$\approx$ 12
Llama-3.1-405B-Instruct	5.07×	$\geq$ 99%	$\approx$ 17
Qwen2.5-7B (SubSpec, 8GB)	10.1×	100%	$\approx$ 27
Qwen2.5-32B (SubSpec, 24GB)	12.5×	100%	$\approx$ 27

FLy demonstrates robustness to out-of-distribution (OOD) task and data shifts, significantly outperforming training-based speculative decoding (e.g., EAGLE-3, speedup ratio improvement of $1.62\times$ ) (Li et al., 28 Nov 2025). Hyperparameters such as entropy threshold $\theta$ , window size $W$ , and draft block size $K$ exhibit insensitivity across models and datasets.

In ablation studies, deferred acceptance and entropy gating are shown to contribute substantial incremental speedup without loss of semantic fidelity. Adaptive drafting techniques further enhance performance under domain shift compared to offline-tuned static subnetworks (Metel et al., 1 Oct 2024, Xia et al., 9 Oct 2024).

5. Implementation Guidelines, Limitations, and Extensions

FLy deployment is straightforward due to its training-free nature:

Zero additional parameters; integrate into standard SPD frameworks.
Entropy computation and deferred matching imposed negligible runtime overhead (entropies derived from logits via softmax).
Dynamic drafting performs robustly with minimal tunable parameters.
Batch inference and multi-GPU support extend applicability to production-scale workloads (Li et al., 28 Nov 2025, Xia et al., 9 Oct 2024).

Limitations may arise in highly specialized or low-redundancy models, where layer-pruning yields few skips or low acceptance (Metel et al., 1 Oct 2024). Additional work may be needed for non-transformer or mixture-of-expert networks (Wang et al., 22 Sep 2025).

Potential extensions include auto-tuned threshold schedules, sophisticated similarity metrics (e.g., subspace-angle), joint MLP pruning guided by activation statistics, and integration with parallel unmasking/KV-caching methods in diffusion LLMs (Agrawal et al., 22 Sep 2025). Combining graph-structured speculative proposals further amplifies acceleration in non-autoregressive regimes.

6. Comparative Analysis and Theoretical Guarantees

FLy approaches distinguish themselves by balancing speed and semantic fidelity. Deterministic tokens (low entropy) trigger exact-match verification, guaranteeing no deviation from the model’s output distribution (Li et al., 28 Nov 2025). Looser criteria (entropy-gated and deferred window) enable the acceptance of semantically correct, non-exact alternatives, leveraging the target model’s own self-corrective generation.

Theoretical analysis demonstrates that, under the loose speculative criteria, the expected per-token cost approaches that of the idealized acceptance ratio, bounded above by the mean tokens accepted per round; practical gains depend critically on the relative cost of draft and verification stages, alignment of the draft model, and acceptance dynamics (Li et al., 28 Nov 2025, Wang et al., 22 Sep 2025, Zhang et al., 2023, Xia et al., 9 Oct 2024).

Losslessness is preserved on deterministic tokens and nearly perfect accuracy recovery is observed empirically, supporting the notion that the entropy and deferred acceptance mechanism filters genuine semantic errors from harmless gap variants. The plug-and-play nature of FLy allows composability with legacy and contemporary inference stacks throughout research and applied settings.

References:

"Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match" (Li et al., 28 Nov 2025) "SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration" (Xia et al., 9 Oct 2024) "Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding" (Wang et al., 22 Sep 2025) "Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similarity" (Metel et al., 1 Oct 2024) "Draft & Verify: Lossless LLM Acceleration via Self-Speculative Decoding" (Zhang et al., 2023) "Spiffy: Multiplying Diffusion LLM Acceleration via Lossless Speculative Decoding" (Agrawal et al., 22 Sep 2025)