CLaSp: Dynamic Programming for LLM Decoding

• Dynamic Programming (CLaSp) is a method that adaptively skips transformer layers to accelerate LLM decoding without retraining.
• It employs a DP table to choose between skipping or keeping layers by maximizing cosine similarity between model hidden states.
• Empirical evaluations show 1.3×–1.7× speedups with minimal loss in fidelity by exploiting redundancy in deep transformer architectures.

CLaSp (In-Context Layer Skip for Self-Speculative Decoding) is a dynamic programming-based strategy designed to optimize layer skipping in LLMs for accelerated speculative decoding. CLaSp leverages hidden state feedback from the latest verification stage, enabling adaptive construction of a compressed draft model by skipping intermediate layers in a plug-and-play manner, without requiring extra drafting modules or retraining. Central to its approach is a dynamic programming (DP) algorithm that determines the set of layers to skip by maximizing the alignment between the draft and verify model’s hidden states, achieving substantial generation speedup while maintaining output fidelity (Chen et al., 30 May 2025).

1. Problem Formulation and Objective

Let $M_p$ denote the full (“verify”) transformer model with $L$ layers, and $M_d$ the compressed draft model obtained by skipping exactly $M$ of these layers. For any token input, the hidden state at each layer $i$ in $M_p$ is $h_i\in\mathbb{R}^d$. The goal is to select a skip set $S\subset\{1,\dots,L\}$, $|S|=M$, such that the top hidden state $g_S(L)$ of the draft model $L$0 is maximally similar—measured by cosine similarity—to $L$1, the top-layer output of $L$2 on the last accepted token. The global optimization can be stated as:

$L$3

where $L$4 represents the hidden state after skipping precisely the layers in $L$5.

2. Dynamic Programming Recurrence Structure

The core of CLaSp is the DP table $L$6, which records the maximal achievable cosine similarity between $L$7 and any draft hidden state at layer $L$8 when $L$9 skips have been used among the first $M_d$0 layers. Let $M_d$1 record the actual hidden state vector at this point. The recursion proceeds as follows:

• Base case: $M_d$2 (embedding output), $M_d$3.
• For $M_d$4, $M_d$5:
  • Skip layer $M_d$6: $M_d$7, score $M_d$8 if $M_d$9.
  • Keep layer $M$0: $M$1, score $M$2.
  • $M$3, and $M$4 set to the argument achieving the maximum.

Upon filling $M$5, backtracking through $M$6 recovers the skip pattern $M$7.

3. Utilization of Verify Model Hidden States

CLaSp integrates the complete stack of hidden representations $M$8 from $M$9’s most recent verification for the accepted token. This feedback provides local reward signals at each DP step, aligning the in-progress draft state at each layer with the gold verify-model state. The incremental DP process thus makes skip/keep choices that maintain maximal alignment with the verify model at every layer, guided by real activation trajectories rather than fixed skip sets or pre-calibrated proxy signals.

4. Computational Complexity and Implementation

The DP table is of size $i$0, with each entry requiring either storage of a $i$1-dimensional vector or its cosine similarity score. The total forward computation across all entries is bounded by $i$2 for models with $i$3-dimensional hidden states. Notably, evaluation of both skip and keep options per table entry admits parallelization over $i$4 (the number of skips) for each layer $i$5, which enables sequence-level parallelism on typical hardware accelerators; the effective wall-clock cost is thus approximately $i$6 forward passes. Memory footprint can be reduced by only retaining two consecutive rows and requisite backpointers for reconstruction.

5. Integration with Self-Speculative Decoding Loop

CLaSp is incorporated as a control block within the self-speculative decoding workflow:

• Draft stage: Employ current skip pattern $i$7 to generate $i$8 tokens from $i$9 with skipped layers.
• Verify stage: $M_p$0 processes the $M_p$1 tokens, identifying the first rejection and recording all hidden states $M_p$2 for the last accepted token.
• Layer-skipping optimization: Feed $M_p$3 into the DP algorithm to produce new skip set $M_p$4 for the next draft pass.
• Optionally, to exploit the temporal persistence of optimal skip patterns across sequential tokens, CLaSp allows lower-frequency DP recomputation ("Sparse Persistence") by reusing $M_p$5 for $M_p$6 verification rounds.

6. Empirical Performance and Theoretical Guarantees

CLaSp does not offer exact optimality due to the lack of strict Markov independence between layers in deep transformers, but empirical evaluation demonstrates that the resulting draft states achieve at least $M_p$7 cosine alignment with exhaustive search solutions. Experimental results on LLaMA3 models show wall-clock speedups of $M_p$8 to $M_p$9 across diverse tasks, when skipping approximately $h_i\in\mathbb{R}^d$0–$h_i\in\mathbb{R}^d$1 of layers. The method induces negligible change in token output distribution relative to full-model decoding. This suggests substantial inherent redundancy in model layers, a property crucial for high-fidelity layer skipping. Adjusting $h_i\in\mathbb{R}^d$2 provides direct control over the trade-off between decoding speed and acceptance rate, with more aggressive skipping yielding faster drafts but lower verify acceptance (Chen et al., 30 May 2025).

7. Relationship to Related Methods and Limitations

Unlike prior speculative decoding frameworks requiring additional drafting modules or retraining, CLaSp is fundamentally “in-context” and agnostic to model architecture; it operates entirely through runtime access to the verify model’s hidden activations. The DP-based skip pattern selection leverages per-token feedback for dynamic adaptation, contrasting with approaches predicated on statically pre-optimized skip sets. The main practical limitation is the only approximate optimality achieved—the skip decisions are locally but not globally optimal within the nonlinear composition of transformer layers. Nonetheless, the suboptimality is minor in high-redundancy settings commonly present in large LLMs, as empirically confirmed. A plausible implication is that layer-skipping dynamic programming of this form will continue to be effective as model depth and overparameterization scale further (Chen et al., 30 May 2025).