Speculative Decoding for Packed NPU Serving

Build and evaluate a composition of K-token speculative decoding with the static, mask-packed NPU serving mode, including correctness, throughput, cache management, and context-window effects.

Background

The packed NPU serving mode uses a static sequence dimension and host-generated attention masks to execute multiple isolated requests in one inference. Measurements show that packing substantially reduces marginal per-request decode cost on NPU stages.

Because the packed path has low marginal row cost, the paper suggests that speculative verification could be especially attractive there. Nevertheless, the interaction between speculative draft tokens, packed slot masks, static KV rings, and stage synchronization has not been implemented.

References

The 0.2--0.3~ms marginal row cost would also make a $K$-token speculative verify nearly free on this path, a composition we have not yet built.

— Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets  (2608.19147 - Berenbaum et al., 19 Aug 2026) in Section 6.6, “Continuous Batching on CPU, GPU, and NPU”