SpecTr: Fast Speculative Decoding via Optimal Transport

Abstract: Autoregressive sampling from LLMs has led to state-of-the-art results in several natural language tasks. However, autoregressive sampling generates tokens one at a time making it slow, and even prohibitive in certain tasks. One way to speed up sampling is $\textit{speculative decoding}$: use a small model to sample a $\textit{draft}$ (block or sequence of tokens), and then score all tokens in the draft by the LLM in parallel. A subset of the tokens in the draft are accepted (and the rest rejected) based on a statistical method to guarantee that the final output follows the distribution of the large model. In this work, we provide a principled understanding of speculative decoding through the lens of optimal transport (OT) with $\textit{membership cost}$. This framework can be viewed as an extension of the well-known $\textit{maximal-coupling}$ problem. This new formulation enables us to generalize the speculative decoding method to allow for a set of $k$ candidates at the token-level, which leads to an improved optimal membership cost. We show that the optimal draft selection algorithm (transport plan) can be computed via linear programming, whose best-known runtime is exponential in $k$. We then propose a valid draft selection algorithm whose acceptance probability is $(1-1/e)$-optimal multiplicatively. Moreover, it can be computed in time almost linear with size of domain of a single token. Using this $new draft selection$ algorithm, we develop a new autoregressive sampling algorithm called $\textit{SpecTr}$, which provides speedup in decoding while ensuring that there is no quality degradation in the decoded output. We experimentally demonstrate that for state-of-the-art LLMs, the proposed approach achieves a wall clock speedup of 2.13X, a further 1.37X speedup over speculative decoding on standard benchmarks.

• The paper introduces a novel framework for speculative decoding using optimal transport with a membership cost to maximize accepted tokens from multi-candidate drafts.
• It derives an LP formulation and proposes a greedy algorithm that achieves a (1-1/e) approximation with near-linear time complexity.
• Empirical results demonstrate up to 2.13X speedup over standard decoding while preserving target distribution consistency.

Speculative decoding accelerates autoregressive sampling from LLMs by using a smaller, faster draft model to propose sequences (drafts) of tokens, which are then verified in parallel by the larger target LLM. Only a prefix of the draft sequence statistically consistent with the target model's distribution is accepted. "SpecTr: Fast Speculative Decoding via Optimal Transport" (2310.15141) introduces a novel framework for speculative decoding based on Optimal Transport (OT) with a membership cost, generalizing existing methods and leading to improved performance.

Theoretical Framework: Optimal Transport with Membership Cost

Standard speculative decoding aims to sample from a target distribution $p(x_{t+1} | x_{1:t})$ using proposals from a draft distribution $q(x'_{t+1} | x_{1:t})$. A sequence $x'_{t+1}, \dots, x'_{t+\gamma}$ is sampled from $q$. Then, for each position $i=1, \dots, \gamma$, the token $x'_{t+i}$ is accepted if a random variable $U \sim \text{Uniform}(0, 1)$ satisfies $$U \le \frac{p(x'_{t+i} | x_{1:t+i-1})}{q(x'_{t+i} | x_{1:t+i-1})}$$. If rejected, a correction token is sampled from a modified distribution derived from $p$ and $q$, and the process restarts.

SpecTr reformulates this process through the lens of optimal transport. The goal is to find a coupling (a joint probability distribution) $q(x'_{t+1} | x_{1:t})$0 between the draft distribution $q(x'_{t+1} | x_{1:t})$1 and the target distribution $q(x'_{t+1} | x_{1:t})$2 (conditioned on the history $q(x'_{t+1} | x_{1:t})$3, omitted for brevity) that maximizes the expected number of accepted tokens. This is framed as an OT problem with a specific "membership cost".

The key generalization introduced by SpecTr is allowing the draft model to propose a set of $q(x'_{t+1} | x_{1:t})$4 candidate tokens $q(x'_{t+1} | x_{1:t})$5 at each position, instead of just one. Let $q(x'_{t+1} | x_{1:t})$6 be the distribution over sets of $q(x'_{t+1} | x_{1:t})$7 candidates generated by the draft mechanism. The objective is to find a transport plan $q(x'_{t+1} | x_{1:t})$8, where $q(x'_{t+1} | x_{1:t})$9, that defines the probability of selecting the target token $x'_{t+1}, \dots, x'_{t+\gamma}$0 given the proposed set $x'_{t+1}, \dots, x'_{t+\gamma}$1. This plan must satisfy the marginal constraint $x'_{t+1}, \dots, x'_{t+\gamma}$2 for all $x'_{t+1}, \dots, x'_{t+\gamma}$3, ensuring the final output follows the target distribution $x'_{t+1}, \dots, x'_{t+\gamma}$4.

The goal is to maximize the probability that the selected token $x'_{t+1}, \dots, x'_{t+\gamma}$5 is one of the proposed candidates, i.e., maximize $x'_{t+1}, \dots, x'_{t+\gamma}$6. This corresponds to maximizing the expected number of accepted tokens from the draft set. This formulation extends the maximal-coupling problem by considering a set of candidates ($x'_{t+1}, \dots, x'_{t+\gamma}$7) rather than just a single proposal.

Optimal Draft Selection and Computational Complexity

The paper demonstrates that the optimal transport plan $x'_{t+1}, \dots, x'_{t+\gamma}$8 maximizing the acceptance probability can be formulated and solved as a Linear Program (LP). The variables in the LP are $x'_{t+1}, \dots, x'_{t+\gamma}$9 for all possible target tokens $q$0 and all possible candidate sets $q$1. The objective is to maximize $q$2 subject to the marginal constraint $q$3 for all $q$4, and the probability constraints $q$5 and $q$6 for all $q$7.

While theoretically solvable via LP, the number of possible candidate sets $q$8 grows combinatorially with $q$9 and the vocabulary size $i=1, \dots, \gamma$0, making the LP formulation computationally intractable for practical values of $i=1, \dots, \gamma$1 and $i=1, \dots, \gamma$2. The best-known runtime for solving such LPs is exponential in $i=1, \dots, \gamma$3.

The SpecTr Algorithm and Approximation Guarantee

To overcome the computational bottleneck of solving the exact LP, SpecTr introduces an efficient, greedy algorithm for constructing a valid transport plan $i=1, \dots, \gamma$4 that achieves a strong approximation guarantee. This algorithm provides a $i=1, \dots, \gamma$5-multiplicative approximation to the optimal acceptance probability achievable by any valid transport plan. Crucially, this approximate algorithm runs in time nearly linear in the vocabulary size $i=1, \dots, \gamma$6, specifically $i=1, \dots, \gamma$7 or potentially $i=1, \dots, \gamma$8 with appropriate data structures, making it highly practical.

The core idea of the greedy algorithm resembles online bipartite matching or ad allocation algorithms. It iteratively assigns probability mass from the target distribution $i=1, \dots, \gamma$9 to candidate sets $x'_{t+i}$0 containing $x'_{t+i}$1, prioritizing assignments that yield the highest "gain" in terms of acceptance probability, while carefully managing the remaining probability mass of $x'_{t+i}$2 and $x'_{t+i}$3 to ensure the marginal constraints are met.

The overall SpecTr sampling procedure at each step $x'_{t+i}$4 involves:

1. Candidate Generation: Use the draft model $x'_{t+i}$5 to generate a set of $x'_{t+i}$6 candidate tokens $x'_{t+i}$7 for position $x'_{t+i}$8. This might involve beam search or multiple sampling passes from $x'_{t+i}$9.
2. Target Probability Calculation: Query the target model $U \sim \text{Uniform}(0, 1)$0 in parallel to obtain the probabilities $U \sim \text{Uniform}(0, 1)$1 for all candidates $U \sim \text{Uniform}(0, 1)$2. Note that obtaining the full distribution $U \sim \text{Uniform}(0, 1)$3 might be needed for the selection algorithm, depending on its specifics.
3. Draft Selection: Apply the efficient $U \sim \text{Uniform}(0, 1)$4-approximate algorithm using $U \sim \text{Uniform}(0, 1)$5, $U \sim \text{Uniform}(0, 1)$6, and the candidate set $U \sim \text{Uniform}(0, 1)$7 to determine the acceptance/rejection probabilities for the candidates. This yields a sampling distribution over $U \sim \text{Uniform}(0, 1)$8.
4. Sampling: Sample an outcome according to the distribution computed in step 3.
   • If a token $U \sim \text{Uniform}(0, 1)$9 is sampled, it is accepted. Increment $$U \le \frac{p(x'_{t+i} | x_{1:t+i-1})}{q(x'_{t+i} | x_{1:t+i-1})}$$0 and repeat from step 1 for the next position.
   • If 'reject' is sampled, discard all candidates. Sample a correction token from the residual distribution required to maintain consistency with $$U \le \frac{p(x'_{t+i} | x_{1:t+i-1})}{q(x'_{t+i} | x_{1:t+i-1})}$$1. Restart the process from step 1 for position $$U \le \frac{p(x'_{t+i} | x_{1:t+i-1})}{q(x'_{t+i} | x_{1:t+i-1})}$$2.

This process allows multiple tokens (potentially the entire sequence proposed by the draft model, depending on acceptance) to be decoded per single invocation of the selection logic, amortizing the cost and leveraging the parallel computation capability for target model evaluations.

Implementation Considerations

Implementing SpecTr involves several key components and choices:

• Draft Model Selection: The choice of the draft model $$U \le \frac{p(x'_{t+i} | x_{1:t+i-1})}{q(x'_{t+i} | x_{1:t+i-1})}$$3 is critical. It needs to be significantly faster than the target model $$U \le \frac{p(x'_{t+i} | x_{1:t+i-1})}{q(x'_{t+i} | x_{1:t+i-1})}$$4 but also provide distributions reasonably close to $$U \le \frac{p(x'_{t+i} | x_{1:t+i-1})}{q(x'_{t+i} | x_{1:t+i-1})}$$5 to ensure a high acceptance rate. Smaller versions of the target model or distilled models are common choices. The method used to generate the $$U \le \frac{p(x'_{t+i} | x_{1:t+i-1})}{q(x'_{t+i} | x_{1:t+i-1})}$$6 candidates (e.g., beam search with width $$U \le \frac{p(x'_{t+i} | x_{1:t+i-1})}{q(x'_{t+i} | x_{1:t+i-1})}$$7, diverse beam search, multiple independent samples) also impacts performance.
• Target Model Parallelism: The primary speedup comes from evaluating the target model $$U \le \frac{p(x'_{t+i} | x_{1:t+i-1})}{q(x'_{t+i} | x_{1:t+i-1})}$$8 on multiple candidate tokens $$U \le \frac{p(x'_{t+i} | x_{1:t+i-1})}{q(x'_{t+i} | x_{1:t+i-1})}$$9 in parallel. Efficient batching and hardware utilization (GPUs/TPUs) are essential.
• Parameter $p$0: Selecting the number of candidates $p$1 involves a trade-off. Higher $p$2 increases the theoretical upper bound on the acceptance rate via the $p$3-optimal algorithm, potentially leading to greater speedup. However, it also increases the cost of generating candidates (step 1) and potentially the complexity of the selection algorithm (step 3), though the proposed algorithm is efficient. The optimal $p$4 is likely task- and model-dependent and requires empirical tuning. Values like $p$5 or $p$6 might be practical starting points.
• Selection Algorithm Implementation: The efficient $p$7-approximate algorithm needs careful implementation. It likely involves sorting or using priority queues based on probability ratios like $p$8 or similar metrics derived from the OT formulation. The exact details would require referring to the algorithm description in the paper (2310.15141). Ensuring numerical stability when dealing with small probabilities is important.
• Distribution Handling: The algorithm requires access to probabilities $p$9 and potentially $q$0 or the distribution $q$1. Efficiently querying these distributions and handling the residual distribution for rejection sampling are necessary.
• System Integration: Integrating SpecTr into an LLM inference framework requires modifying the generation loop. Instead of sampling one token, it generates $q$2 candidates, runs the parallel verification, executes the SpecTr selection logic, and handles acceptance/rejection/correction. This involves managing the state across multiple potential future tokens.

Below is a high-level conceptual pseudocode for the SpecTr sampling loop:

$q$8

Experimental Results

The paper reports significant empirical speedups. On standard benchmarks using state-of-the-art LLMs, SpecTr achieved a wall clock speedup of 2.13X compared to standard autoregressive decoding. Furthermore, it demonstrated a 1.37X speedup over the baseline speculative decoding method (which corresponds to the case $q$3). These results highlight the practical benefit of the OT formulation and the multi-candidate ($q$4) approach enabled by SpecTr. The authors emphasize that this speedup is achieved while mathematically guaranteeing that the output distribution remains identical to that of the target model $q$5, thus incurring no degradation in generation quality.

Conclusion

SpecTr provides a principled extension to speculative decoding by framing it as an optimal transport problem with membership cost. This allows generalizing the mechanism to handle multiple ($q$6) draft candidates per position. While the exact optimal solution is computationally expensive, SpecTr introduces an efficient approximation algorithm with a $q$7-optimality guarantee and near-linear time complexity. This leads to substantial wall-clock speedups in practice (reported as 2.13X overall, 1.37X over standard speculative decoding) without compromising the output quality, making it a promising technique for accelerating inference in LLMs. Implementing SpecTr requires careful integration with existing inference systems, focusing on efficient candidate generation, parallel target model evaluation, and the implementation of the novel selection algorithm.