---
title: Lossy Speculative Decoding
url: https://www.emergentmind.com/topics/lossy-speculative-decoding
type: topic
---

# Lossy Speculative Decoding

Lossy speculative decoding is a class of algorithms for accelerating autoregressive language model inference by accepting partial or approximate solutions from a draft ("lossy") model, followed by verification and correction using the original ("target") model. Unlike strictly lossless speculative decoding, which preserves the exact output distribution of the base model at all times, lossy speculative decoding introduces controlled approximation—in either the drafting model, the acceptance policy, or the computational process—to improve throughput and utilization, potentially at the cost of a small, tunable accuracy drop. This approach addresses fundamental bottlenecks in large language model decoding workflows, especially under resource or latency constraints.

## 1. Core Principles and Definitions

Lossy speculative decoding generalizes the standard speculative decoding framework by tolerating certain mismatches or errors in the draft phase to gain computational efficiency. Standard speculative decoding uses a draft model to predict a block of future tokens, then verifies them in parallel against the full model; only predictions that exactly match the base model are accepted, reverting to autoregressive evaluation on mismatch. This strict acceptance policy is distribution-preserving (lossless) but limits maximal speedup, particularly when draft and target models are only loosely aligned or when draft outputs degrade due to quantization or approximation [2510.18525] [2310.08461] [2502.02493].

Lossy speculative decoding relaxes draft quality and/or modifies the acceptance rule. Drafts may be generated by quantized, aggressively pruned, or layer-parallel approximations of the target model. Acceptance policies can employ lenience parameters, permitting the system to accept more draft tokens by trading strict fidelity for throughput. Sources of "lossiness" include:

- Use of draft models constructed through quantization or architectural changes that reduce representational accuracy (e.g., FP16→FP4 quantization).
- Relaxed statistical criteria during the accept/reject phase (lenience functions/temperatures).
- Structural computation approximations in the draft pass, such as layer-parallel execution that breaks true inter-layer dependencies for increased parallelism.

These strategies are integrated with a verification phase by the target model that mitigates large errors and controls overall output quality.

## 2. Algorithmic Techniques

Several algorithmic innovations have been developed to facilitate efficient lossy speculative decoding, each exploiting a unique axis of the draft/verification pipeline:

- **Bit-Sharing Quantization with Remapping (SPEQ):** The draft model is formed by decomposing the full model’s FP16 weights into a 4(–5) bit quantized representation (E3M0), sharing exponent/mantissa/sign bits, and remapping exponents to minimize quantization error. Draft computation uses only these compressed weights, and full-precision weights are recomposed on-the-fly for verification. Group-wise scales are applied per 128-weight segment to minimize MSE [2510.18525].
- **Lossy Acceptance Ratios via Lenience Functions:** The acceptance computation in the verification phase is generalized from $r(y) = \min(1, p(y)/q(y))$ to $r(y) = \min(1, f(p(y),\epsilon)/q(y))$, where $f$ is an increasing function of $p$ and $\epsilon$ is a lenience parameter. Common choices include $f_\mathrm{lin}(p,\epsilon)=p/\epsilon$, $f_\mathrm{sq}(p,\epsilon)=p/\epsilon^2$, and $f_\mathrm{exp}(p,\epsilon)=p^\epsilon$. The $\epsilon$ knob controls the trade-off between fidelity and throughput [2310.08461].
- **Layer-Parallel Drafting (EasySpec):** In distributed inference, the draft model's layers are grouped and run in parallel across available GPUs. Within a group, attention sublayers process approximated hidden states simultaneously, yielding "fuzzy" layer outputs. Verification remains strict, but per-layer approximation errors are reset via periodic key-value (KV) cache recalibration passes [2502.02493].

Algorithmic structure generally follows: (i) draft generation (possibly approximate/quantized/layer-parallel), (ii) blockwise verification against the base model, (iii) acceptance or rollback, and (iv) optional calibration to correct drift or model mismatch.

## 3. Hardware and Systems Integration

Lossy speculative decoding methods have prompted specialized hardware and distributed system adaptations:

- **Algorithm-Hardware Co-Design (SPEQ Accelerator):** SPEQ targets a 28nm custom accelerator comprising a reconfigurable PE array. Each PE can switch between 4-bit quantized (draft) and FP16 (full) compute, sharing arithmetic/logical units for area efficiency. The draft mode computes three times as many partial sums per cycle compared to FP16. The remapping decoder is implemented in hardware with negligible area/power (<4%) overhead. Both computation modes operate at 500 MHz with total power ~0.5 W, and memory utilization is identical for draft and full weights due to the bit-sharing approach [2510.18525].
- **Efficient Multi-GPU Utilization (EasySpec):** Layer-parallel speculation assigns multiple layers of the draft model to different GPUs, synchronizing only at window boundaries. This removes idle time present in standard tensor-parallel speculative decoding when the draft model is much smaller than the base model. Approximation errors are controlled via periodic KV cache calibration [2502.02493].

A practical implication is that these hardware and system-level optimizations can substantially raise single-step and end-to-end throughput, and are compatible with common LLM deployment backends.

## 4. Error Analysis and Empirical Performance

The error and speed-accuracy trade-off in lossy speculative decoding are controlled by the quality of the draft, the lenience function/temperature, and the calibration frequency. Key findings from recent work include:

- **Token Acceptance Rate and Correctness:** The quantized/remapped draft model in SPEQ achieves an average token accept-rate $r \approx 0.976$ across diverse models and tasks. This enables draft lengths $L \approx 5$–$8$ and block acceptance rates $0.97$–$0.99$ with no deviation from final output, as mispredicted tokens are never accepted (speculation correctness is maintained by verification) [2510.18525].
- **Empirical Latency and Throughput:** In DistillSpec's lossy SD with distilled draft and $f_\mathrm{sq}$ acceptance, increasing lenience $\epsilon$ from $1.0$ to $0.0001$ yields speedups from $1.53\times$ to $2.80\times$ with accuracy drops from $0.2\%$ to $4.6\%$ (GSM8K, block size $7$). Practically, speedup increases as acceptance is made more lenient, with a direct, tunable hit to answer quality [2310.08461].
- **Layer-Parallelization Errors:** Layer-parallel EasySpec reports per-layer approximation errors with cosine similarities $\geq 0.82$ (for $N=4$ layers grouped). After periodic cache calibration, cumulative error is bounded. Peak speedup for Llama-3-70B is $4.17\times$ with a maximum end-to-end accuracy drop $\leq 7\%$ [2502.02493].

| Method (Paper)       | Speedup Range | Acceptance/Accuracy Drop |
|----------------------|---------------|-------------------------|
| SPEQ [2510.18525]    | 1.45–2.07×    | $\leq 0.03$ (acceptance)|
| DistillSpec [2310.08461] | 1.53–2.80× | $<5\%$ (accuracy)      |
| EasySpec [2502.02493]| 3.38–4.17×    | $\leq 7$ p.p.           |

Draft quality depends on weight distribution (draft outliers require per-tensor scaling) and approximation granularity; overly aggressive quantization/remapping or excessive window size in layer-parallel schemes cause larger acceptance drops.

## 5. Representative Algorithms and Pseudocode

Each methodology formalizes the lossy speculative decoding pipeline using clear algorithmic structure:

### SPEQ (Remapped Quantized Draft):

```python
function SPEQ_decode(prefix, max_decode_T):
    kv_cache_full, kv_cache_draft = {}, {}
    output = prefix
    while |output| < |prefix|+max_decode_T:
        draft_tokens = []
        for i in range(L_max):
            logits, kv_cache_draft = draft_forward(output + draft_tokens, kv_cache_draft)
            p, t = sample_top1(logits)
            if p < gamma: break
            draft_tokens.append(t)
        logits_full, kv_cache_full = full_forward(output, kv_cache_full)
        accept = []
        for j, t in enumerate(draft_tokens):
            if t maximizes logits_full·softmax:
                accept.append(t)
            else: break
        if accept:
            output += accept
        else:
            next_logit, kv_cache_full = full_forward(output, kv_cache_full)
            t = sample_top1(next_logit)
            output.append(t)
        kv_cache_draft.clear()
    return output
```
[2510.18525]

### DistillSpec (Lossy Acceptance):

```python
function lossy_speculative_decode(p, qθ, x, γ, f, ε):
    y ← []
    while not end_of_sequence(y):
        draft_batch = []
        for i in 0..γ-1:
            q_dist = qθ(x ∥ y)
            y_candidate = sample(q_dist)
            draft_batch.append((y_candidate, q_dist[y_candidate]))
            y.push(y_candidate)
        p_dists = p(x ∥ y[-γ:])
        r = []
        for i in 0..γ-1:
            p_i = p_dists[i][draft_batch[i].token]
            q_i = draft_batch[i].prob
            r_i = min(1, f(p_i, ε) / q_i)
            r.append(r_i)
        n = 0
        for i in 0..γ-1:
            u = Uniform(0,1)
            if u ≤ r[i]: n += 1
            else: break
        remove_last_γ_minus_n_tokens(y)
        p_next = p(x ∥ y)
        y_next = sample(p_next)
        y.push(y_next)
    return x ∥ y
```
[2310.08461]

### EasySpec (Layer-Parallel Draft with Calibration):

```python
for each speculative iteration:
    # Drafting: run N layers of M_d in parallel on G GPUs (fuzzy)
    # Verification: run M_b on drafted tokens (strict)
    # Accept tokens if u_i ≤ min(1, p_i(x'_i)/p'_i(x'_i))
    # Periodically, run M_d on accepted+bonus tokens to recalibrate KV cache
```
[2502.02493]

## 6. Generalization, Practical Guidance, and Limitations

Lossy speculative decoding methods are extensible to various quantization formats (e.g., BF16→FP4, INT formats) by bit-sharing and remapping, with potential for further bitwidth and exponent code optimization. The techniques can be integrated into inference libraries (e.g., cuBLAS/TensorRT) by incorporating quantized GEMM kernels and compact decode LUTs to accelerate drafting without accuracy regression [2510.18525].

Practical deployment involves:

- Tuning lenience parameters ($\epsilon$) to find a knee point where speed-up is maximized for an acceptable quality drop ($<1$–$2\%$ typical). Theoretical proxies, such as $\alpha(\epsilon) \approx \mathbb{E}[\min(1,f(p,\epsilon)/q)]$ and expected block acceptance size $\tau(\epsilon)$, aid in predicting latency gains [2310.08461].
- Layer-parallel size ($N$) and attention width ($w$) trade off parallelism vs. error; empirically $N=4$ is optimal for most workloads [2502.02493].
- Draft quality, dictated by weight distribution and approximation granularity, places practical lower bounds on quantization or parallelization aggressiveness.
- Hardware choices, such as shared-memory layouts or reconfigurable compute arrays, materially affect cost, memory, and energy efficiency.

Limitations include increased sensitivity of the acceptance rate to draft outliers, and reduced draft quality if quantization or layer-parallel grouping is too aggressive. Overly low-bitwidth quantization (<4-bit) or large window sizes can degrade acceptance to the point that speed-accuracy trade-off is no longer favorable. Additional memory for KV caches and communication bandwidth for large-window parallelism must also be considered.

## 7. Outlook and Future Directions

The future development of lossy speculative decoding is expected along several axes:

- Adaptive remapping or quantization per layer/tensor, guided by online statistics or learned heuristics, for more flexible quality/performance control [2510.18525].
- Integration with activation quantization or hybrid acceleration paradigms (sparsity, pruned attention) for further speedup.
- On-the-fly learning-based selection of draft/verify hyperparameters using runtime feedback to optimize acceptance rate vs. quality [2510.18525].
- Incorporation of lossy speculative decoding into mainstream multi-GPU and cloud-inference backends, aided by minimal hardware and memory requirements.
- Exploration of advanced knowledge distillation protocols and divergence choices to further align distill-draft models with targets, boosting acceptance with negligible quality cost [2310.08461].

Collectively, lossy speculative decoding leverages targeted approximations in the drafting and acceptance stages to achieve significant latency reductions in LLM inference, with principled mechanisms to bound and control the resulting output deviation. The approach is validated across model architectures, datasets, and hardware backends, and continues to evolve to address scaling and deployment challenges in large-scale generative AI.

Source: https://www.emergentmind.com/topics/lossy-speculative-decoding