Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference

Abstract: Long-context inference is increasingly common in LLM serving, driven by retrieval-augmented generation and agentic systems. In disaggregated inference, these workloads require transferring large Key-Value (KV) caches across the network, where decoding cannot begin until the transfer completes. Recent KV quantization techniques reduce data volume and alleviate this bottleneck, but existing schemes fail to achieve both low network-exposed latency and high inference accuracy. We challenge the assumption that the KV cache is an indivisible unit that must be fully received before use. We leverage the observation that different bits in the KV cache contribute unequally to attention computation and inference precision: the most significant bits capture the coarse structure of attention and the least significant bits refine precision. This property enables partial use of the KV cache during decoding. We present Lynx, a system that enables progressive, split-stream KV transfer by partitioning the KV cache into a high-priority Anchor stream carrying the most significant bits and a low-priority Residual stream carrying remaining precision. Decoding begins upon receipt of the Anchor stream and proceeds speculatively while the Residual stream is transferred concurrently, followed by verification that ensures equivalence to higher-precision decoding. Across multiple models and serving workloads, Lynx achieves Time-to-First-Token (TTFT) comparable to aggressive 4-bit KV quantization, while matching the accuracy of high-precision (BF16) inference, improving TTFT over standard 8-bit KV quantization by up to $1.43\times$ and improving accuracy over state-of-the-art by up to $5.1\%$.

• The paper introduces a split-stream KV transfer method that partitions the cache into Anchor (MSBs) and Residual (LSBs), enabling early speculative decoding.
• The paper applies a hierarchical, per-channel non-linear quantization using logarithmic mapping to address outlier distortion and maintain accuracy.
• The paper demonstrates up to 1.43× reduction in time-to-first-token and 30.5% lower latency across benchmarks with high acceptance rates of speculative tokens.

Lynx: Progressive Speculative Quantization for Accelerating KV Transfer in Long-Context Inference

Motivation and Problem Landscape

Scaling LLM inference to long contexts in retrieval-augmented generation, agentic architectures, and code-centric applications imposes acute bottlenecks in the transfer of KV caches across network boundaries during disaggregated inference. As the sequence length grows, the KV cache scales linearly, becoming a prohibitive source of latency that dominates the time-to-first-token (TTFT) and cascades into elevated token generation times. Despite advances through quantization (INT8/INT4), prior methods treat the KV cache as a monolithic tensor and fail to address the fundamental serialization barrier: decoding does not commence until complete KV transfer, even if aggressive quantization is used.

Key Insights: Exploiting Bitwise Asymmetry and Speculative Execution

This paper introduces Lynx, predicated on the critical observation that not all bits in KV cache values contribute equally to attention computation and model output. The dominance of MSBs in determining attention structure, with LSBs serving as precision refinements, motivates the design of a split data transfer protocol: the KV cache is partitioned into an Anchor (MSBs) and Residual (LSBs) stream. Decoding proceeds speculatively as soon as the Anchor stream is received, with subsequent verification and correction after Residual arrival.

This decoupling of transfer and computation eliminates the strict blocking dependency between communication and compute, enabling systematic overlap and early speculative decoding.

Figure 1: Inference timelines showing that Lynx achieves both high accuracy and low inference latency by overlapping the network communication of KV transfer with the computation of decoding.

Quantization Challenges: Outliers and Non-Uniform Distributions

Standard linear quantization is destabilized by the presence of extreme outliers in the KV cache, which are channel-local and skew the value distribution. Most entries concentrate sharply around zero, while a minority of outlier channels possess values $> 100 \times$ larger. Utilizing a global quantization scale squanders precision for the information-dense region, yielding severe loss in effective bit-width for the majority of the data. INT4 quantization, in particular, introduces high-magnitude errors in the attention output.

Figure 2: Visualization of the attention output (left) and the corresponding INT4 quantization error (right). The error magnitude is significant relative to the feature scale, indicating substantial precision loss.

Figure 3: Visualization of KV cache values at different layers (Qwen 32B on the MMLU-Pro dataset). The persistence of extreme values in specific outlier channels (visible as dark horizontal stripes) highlights the need for a per-channel quantization approach.

Empirical data after per-channel normalization confirm that 50% of KV cache elements are concentrated within the middle 17% of the value range, showing the inefficiency of uniform quantization and motivating finer-grained, non-linear strategies.

Figure 4: The KV cache data distribution after tokenwise (per-channel) normalization that effectively isolates extreme outliers.

System Architecture and Hierarchical Quantization

Lynx integrates a domain-specific, hierarchical, per-channel quantization algorithm with non-linear (logarithmic) mapping. KV cache is normalized and chunked at the channel level, followed by a logarithmic $\alpha$-law transform that aligns bit allocation with data density; the high-entropy, information-rich center receives higher granularity, while tails receive coarse quantization. The top 4 bits (Anchor stream) deterministically encode exponent-like information, sufficient to approximate attention structure for speculative decoding. The Residual (LSB, bottom 4 bits) linearly interpolate within anchor bins, enabling precise post-hoc refinement and lossless reconstruction.

Figure 5: System overview of Lynx.

Figure 6: The Lynx's split stream pipelining architecture, where MSB 4b denotes the anchor stream $Q_{anc}$.

Data is serialized for efficient DMA into anchor and residual streams, allowing immediate execution of prefill/anchor transfer, and asynchronous background transfer of residuals. The architecture enforces strict priority draining of anchor buffers to further reduce TTFT.

Progressive Execution: Prefill and Decode Workflow

Lynx modifies the inference serving contract across compute/storage nodes. The progressive execution engine instantiates three pipelined mechanisms: (1) quantization, (2) prioritized KV transmission, and (3) speculative decoding and verification. Immediately upon anchor stream receipt, the decoder initiates speculative token generation using the partial KV. Once the residuals are available and dequantized, the draft sequence is verified by a single forward pass—accepting valid prefixes and correcting divergences per the formal guarantee of speculative decoding.

Figure 7: The Lynx's prefill and decode workflows described in §bit_stream_pipe.

Empirical Evaluation and Numerical Results

The evaluation comprises three model families (LLaMA, Qwen, Mistral) and three benchmarks (MMLU-Pro, Needle, QMSum) under realistic long-context (up to 128K tokens) and varying bandwidth regimes. Lynx's hierarchical quantization method is compared against uncompressed (BF16), uniform INT8/INT4, and delta-encoded CacheGen baselines.

Figure 8: End-to-end inference accuracy across different LLM inference workloads. Lynx hierarchical quantization schemes achieves similar accuracy across datasets when compared to both BF16 and Int8.

Lynx attains BF16-equivalent accuracy (within $\pm 0.3\%$ random variation) while reducing TTFT and TT32T by up to 1.43× and 30.5% over INT8, respectively, and surpasses the accuracy of CacheGen and INT4 by up to 5.1%. Notably, INT4 and CacheGen experience substantial accuracy drops ($>8\%$ on MMLU-pro in some settings).

Figure 9: End-to-end inference latency in terms of Time-to-Kth-token (TTKT) across different workloads. Lynx incurs only marginal overhead compared to its standard quantization counterparts.

Acceptance rates of speculative tokens generated during anchor-only execution are high: in the MMLU-Qwen workload, 64.8% of speculative decode sequences are fully accepted, with an average of 19.38 out of 21.43 speculative tokens retained. This high acceptance is crucial for hiding residual transfer time and achieving consistent latency gains.

Figure 10: Evaluation of Lynx's speculative tokens acceptance rate over the MMLU Qwen workload.

For extremely long contexts (e.g., 128K), Lynx maintains its accuracy advantage while the latency reduction benefits compound—outperforming vanilla INT8 by increasingly larger margins as context length grows, and retaining low TTKT even under restricted bandwidth.

Figure 11: TT64T for the MMLU LLaMA workload across varying context length and bandwidth. Lynx incurs only marginal overhead compared to its standard quantization counterparts.

Figure 12: The inference accuracy varying the context length from 32K tokens to 128K tokens, running LLaMA 8B over the MMLU dataset.

Theoretical Implications and Future Directions

Lynx's design demonstrates that the serialization dependency in KV transfer is not fundamental: progressive, bit-aware streaming and speculative execution can decouple compute/communication bottlenecks without accuracy degradation. The effectiveness is rooted in the heterogeneity of bitwise significance in the KV cache and the formal verification step that preserves output equivalence.

Potential future directions include:

• Dynamically tuning anchor/residual bit widths (INT2/INT6, mixed precision) to balance acceptance rates, quality, and transfer bandwidth.
• Extension to more aggressive context-parallel and expert-parallel systems with overlapping token and layer-wise speculative execution.
• Generalization across accelerator architectures and further optimization of SerDes and memory layouts for highly concurrent settings.

Conclusion

Lynx redefines the long-context inference system stack through a progressive, prioritized KV streaming model and hierarchical non-linear quantization—delivering high accuracy and low-latency inference for modern LLM workloads. By leveraging split-stream speculative quantization and lossless verification, it fundamentally overcomes the communication bottleneck of disaggregated inference without incurring accuracy compromise, pointing toward a communication-aware future for scalable LLM deployment.

(2607.01831)