Runtime-Certified Bounded-Error Quantized Attention

Abstract: KV cache quantization reduces the memory cost of long-context LLM inference, but introduces approximation error that is typically validated only empirically. Existing systems rely on average-case robustness, with no mechanism to detect or recover from failures at runtime. We present a tiered KV cache architecture that enables runtime-certified attention: INT8 keys and INT4 values are stored in GPU memory, while FP16 originals are retained in system RAM for deterministic fallback. A two-term error decomposition yields per-head, per-step bounds on (i) attention distribution distortion from key quantization and (ii) value reconstruction error. These bounds are computed online and used to drive adaptive precision selection and a multi-stage fallback ladder, which guarantees recovery to the exact dense attention output when required. Across PG-19, NIAH, and RULER benchmarks on LLaMA~3.1-8B with contexts up to 128K, the system matches dense FP16 KV quality within noise for language modelling and retrieval tasks, while recovering catastrophic failures observed in naive INT8/INT4 baselines. Value-sensitive tasks at short context expose a controlled trade-off between compression and fidelity, which can be eliminated via tighter value tolerances or FP16-value fallback. The certification is local (per-head, per-step) and does not guarantee end-to-end model correctness, but ensures that each attention computation is either bounded relative to an FP16 reference or exactly recovered via fallback. This reframes KV cache quantization as a runtime-verified computation rather than a fixed approximation. The goal is not raw speedups, but enabling safe deployment of aggressive KV compression under strict quality constraints.

• The paper introduces runtime-computable key and value error bounds, adaptive block promotion, and a four-level fallback ladder that returns exact dense attention when certification fails.
• The certified system matches dense FP16 quality on PG-19 and NIAH, while naive INT8/INT4 quantization causes severe retrieval failures; RULER remains near parity at 64K and 128K but shows value-sensitive losses at shorter contexts.
• The approach reduces VRAM use by up to 28% at 128K contexts, but adds 2.73–4.76× latency, requires substantial pinned CPU memory, and provides only per-head, per-step—not end-to-end—guarantees.

Overview

"Runtime-Certified Bounded-Error Quantized Attention" (2605.20868) addresses a gap in KV cache quantization for long-context LLM inference: existing systems (KVQuant, KIVI, QServe, PQCache) validate compression empirically and provide no per-step mechanism to detect or recover from approximation failures. The paper proposes a tiered architecture in which per-channel INT8 keys and per-group INT4 values reside in GPU memory while FP16 originals are retained in pinned system RAM, together with a two-term error decomposition whose bounds are computed online at per-head, per-step granularity. These bounds drive adaptive precision selection and a four-rung fallback ladder terminating in exact dense attention via torch.scaled. The stated goal is not raw speedup but safe deployment of aggressive KV compression under strict quality constraints; the paper is explicit that the guarantee is local (per-head, per-step) and does not imply end-to-end model correctness.

The evaluation uses LLaMA-3.1-8B on a single RTX PRO 6000 WS across PG-19, NIAH, and a seven-subtask RULER subset at contexts from 8K to 128K. The headline empirical finding is that the certified system matches dense FP16 quality within noise on language modelling and retrieval, while an identical naive INT8K/INT4V baseline without certification collapses catastrophically — NIAH accuracy drops to 5–10% versus 39–66% dense, and RULER falls by 44–56pp with variable tracking scoring 0% everywhere. This positions the certification machinery as the load-bearing component of quality preservation rather than monitoring overhead.

System architecture

The design separates storage into two tiers. Tier 1 (VRAM) holds INT8 keys quantized per channel after RoPE application, INT4 values quantized per group of $g{=}16$ elements, quantization metadata, and per-block value-error annotations $\eta_b$ computed at cache-write time. Tier 2 (pinned CPU RAM) retains the FP16 originals as ground truth for promotion and fallback. The metadata cost is substantial: per-channel key scales and offsets add 64 bytes/token, so total Tier-1 storage is 288 bytes/token/head, or 56% of dense FP16 — not the 50%/25% that raw INT8/INT4 codes would suggest. The system trades system RAM for VRAM: total footprint including Tier-2 is 156% of dense, which the author argues is acceptable because VRAM is the scarcer resource.

Attention executes directly on compressed data via a fused kernel that dequantizes in registers, following the compressed-domain execution approach of HACK and VecInfer. Execution proceeds in two phases: a lightweight Phase-1 INT8 scoring pass computes per-block log-masses $\ell_b$, an adaptive selector promotes the top-$K^*$ blocks to FP16 keys until estimated coverage reaches $\tau_{\mathrm{cov}}{=}0.995$ (clamped to $[K_{\min}{=}2, K_{\max}{=}128]$), and Phase-2 performs a single fused attend pass with branchless mask-gated key precision over both representations. The selector's $K^*$ adapts to attention concentration: it saturates near the cap at short contexts where coverage demands most blocks, and falls well below the cap at longer contexts where mass concentrates.

A notable architectural decision is that appended tokens accumulate in a trailing partial block kept in FP16 until the block fills, then are quantized atomically. This eliminates stale-scale problems and makes the per-block error bound $\Delta_b$ strictly block-local, supporting precondition P1 (metadata currency) deterministically by construction.

Formal error bounds

The central analytical contribution decomposes $\|O_{\mathrm{quant}} - O_{\mathrm{ref}}\|_2$ through a bridge term into two independent components: $E_{\mathrm{key}}$ (attention-distribution distortion from key quantization, measured against exact values) and $\eta_b$0 (value reconstruction error under the quantized-key distribution). The bounds certify distance to $\eta_b$1 — the same kernel on unquantized FP16 KV — not to $\eta_b$2; the arithmetic-path gap between the certified kernel and Flash Attention is characterized separately and empirically.

Value error admits a tight convexity bound: $\eta_b$3, the worst-case per-token reconstruction error, independent of sequence length because softmax weights form a convex combination. The blockwise corollary $\eta_b$4 is computed exactly at runtime from cached attention masses and pre-stored annotations, avoiding any circular dependency on FP16 originals at attend time.

Key error is bounded via total variation: $\eta_b$5, with TV bounded tightly by $\eta_b$6 where $\eta_b$7 aggregates per-channel scale vectors via Cauchy–Schwarz. The appendix proves this bound is tight, length-independent, and extends to the tail-restricted case relevant to mixed-precision execution: $\eta_b$8 on the INT8 tail. A separate theorem bounds one-sided mass estimation error from INT8 scores by $\eta_b$9, ensuring the true tail mass satisfies $\ell_b$0 regardless of ranking correctness within the promoted set. At the operating point ($\ell_b$1, $\ell_b$2), the key term evaluates to roughly $\ell_b$3, or $\ell_b$4 under the implementation's conservative $\ell_b$5 substitution that accounts for optional INT8×INT8 tensor-core query scoring.

The paper is careful about what these bounds do and do not certify. The mass bound is one-sided and does not guarantee the INT8 ranking is correct — misranking wastes page-in bandwidth but introduces no error, since the guarantee that matters is aggregate tail mass. Four preconditions (P1–P4: current metadata, deterministic reconstruction bounds, no overflow, Tier-2 availability) are distinguished by how each is guaranteed versus monitored, and if P4 fails the system reports an error rather than returning uncertified output. The overall contract is that each head-step either returns a bounded output or the exact dense result.

Instability detection and the ranking-consistency check

Beyond the formal bounds, three defence-in-depth mechanisms run at runtime: a score-consistency canary comparing FP16 scores against declared bounds (zero violations observed, consistent with P2's deterministic guarantee), a 1–5% exploration budget spot-checking low-mass blocks, and value-error annotations driving Rung-2 promotion.

The most consequential mechanism is the ranking-consistency check, which addresses a failure mode the formal bounds cannot see. The adaptive selector ranks blocks using INT8 scores; when two blocks have similar true mass, ~0.4% mean per-channel scoring noise can swap their order. For retrieval tasks this matters acutely: promoting tail blocks to FP16 corrects their scores upward, increasing their softmax share at the needle token's expense — an "attention dilution" effect the paper shows is non-monotonic in the FP16 key budget (the development ablation found uncapped promotion worsened NIAH accuracy from −10pp to −16pp). The check compares top-$\ell_b$6 block rankings under FP16 and INT8 log-masses over the promoted set, plus an interval-bound boundary verification ensuring no tail block's upper-bounded FP16 log-mass could enter the top-$\ell_b$7. On violation, the affected head alone escalates to full FP16 recomputation returning $\ell_b$8. Boundary checks fire at non-trivial rates (8.0% of head-steps on PG-19 at 64K, 0.7% on NIAH), confirming score-order changes at the promote/tail boundary occur and are caught; Rung-3 itself triggers on 1.2–2.2% of head-steps and adds only 1.7% of step time. Rung-4 (all-heads dense recomputation) never fired in evaluation.

Empirical results

PG-19 perplexity shows parity at every context length: $\ell_b$9ppl ranges from $K^*$0 to $K^*$1 across 8K–128K (20 chunks each), with all 95% CIs including zero and ratios within 0.00014 of 1.0.

NIAH (10 needles/trial, 100 paired trials per context) shows no statistically significant difference anywhere testable: identical 39% accuracy at 8K (McNemar $K^*$2), 51% vs 49% at 32K ($K^*$3), identical 66% at 64K ($K^*$4). Discordant pairs are perfectly balanced at 8K and 64K, indicating noise rather than systematic degradation. The paper notes an instructive statistical point: an earlier 20-trial run showed an apparent −5pp gap that vanished entirely at $K^*$5. 128K NIAH is omitted because the base model scores 0% on dense.

RULER reveals the system's one genuine quality trade-off. At 64K and 128K the certified system matches or slightly exceeds dense ($K^*$6pp and $K^*$7pp). But at 8K there is a $K^*$8pp gap (95% CI $K^*$9, 19 critical failures) localized entirely to three value-sensitive subtasks — VT ($\tau_{\mathrm{cov}}{=}0.995$0pp), CWE ($\tau_{\mathrm{cov}}{=}0.995$1pp), FWE ($\tau_{\mathrm{cov}}{=}0.995$2pp) — while all four retrieval variants remain perfect. A decisive ablation isolates the cause: sweeping $\tau_{\mathrm{cov}}{=}0.995$3 from 64 to 512 leaves the gap unchanged, whereas replacing INT4 values with FP16 values closes it to $\tau_{\mathrm{cov}}{=}0.995$4pp with zero critical failures. The mechanism analysis is candid about why the value certificate did not prevent this: the default $\tau_{\mathrm{cov}}{=}0.995$5 policy is a local promotion heuristic, not a global budget enforcement, and small per-step errors compound through the residual stream across layers and decode steps, flipping marginal logits on tasks demanding exact character-level reconstruction. A mild residual ($\tau_{\mathrm{cov}}{=}0.995$6pp, driven by CWE) persists at 32K with a CI nearly excluding zero; both FP16 values and tightening $\tau_{\mathrm{cov}}{=}0.995$7 to 0.02 eliminate it there, at a 9% latency cost. This is the clearest demonstration in the paper of the fundamental limitation of per-step certification: proximity to $\tau_{\mathrm{cov}}{=}0.995$8 at each step does not compose into end-to-end task accuracy over autoregressive chains.

Certificate telemetry distinguishes candidate pre-fallback bounds from returned-output bounds. Candidate $\tau_{\mathrm{cov}}{=}0.995$9 maxima grow with context (13.6 at 64K, 18.1 at 128K) with heavy tails (~46–70× median-to-max), but these extreme values are precisely where Rung-3 intercepts and returns $[K_{\min}{=}2, K_{\max}{=}128]$0; returned outputs are either within the candidate p95 envelope or exactly dense. The paper concedes that post-fallback per-head certificates are inferred rather than logged, flagging instrumentation as a planned improvement.

Runtime performance

The cost of certification is substantial and honestly reported. At a realistic 2048-block scratch cache, certified latency is $[K_{\min}{=}2, K_{\max}{=}128]$1 dense at 8K rising monotonically to $[K_{\min}{=}2, K_{\max}{=}128]$2 at 128K, driven by host-to-device page-in traffic (644 MB/step at 128K over PCIe 5.0). The phase breakdown at 64K attributes 28.3% of step time to the Python-orchestrated ranking-consistency check — identified as the largest single optimization target — and 11.1% to H2D page-in, against only 8.4% for the actual FP16 attend kernel and 2.4% for INT8 scoring. Non-attention model computation accounts for 39%.

Memory economics depend strongly on context length. Because the FP16 scratch buffer is fixed-size (~4.3 GB symmetric), the system uses more VRAM than dense below 64K. With asymmetric cache tuning (2048 key / 256 value blocks at 64K; 2048/512 at 128K, exploiting a sharp value-cache knee below which split-$[K_{\min}{=}2, K_{\max}{=}128]$3 fallback doubles latency), savings reach ~17% at 64K and ~28% at 128K at no latency cost relative to symmetric configurations. GQA union analysis explains why: with 32 query heads independently selecting blocks, the per-layer working-set union covers ~87% of position blocks at 64K but only 64% at 128K, so VRAM savings become meaningful only at long context. Tier-2 retention costs ~17 GB of system RAM at 128K — the explicit price of the unconditional fallback guarantee.

Limitations and open questions

The paper is unusually forthright about scope. The certification is per-head, per-step and bounds distance to $[K_{\min}{=}2, K_{\max}{=}128]$4, not $[K_{\min}{=}2, K_{\max}{=}128]$5; end-to-end quality must be assessed empirically per deployment, and the 8K/32K RULER gaps demonstrate precisely how per-step bounds fail to compose. The arithmetic-path difference between the certified kernel and Flash Attention contributes a residual ~4pp variance source on retrieval-adversarial benchmarks in development diagnostics; whether a tensor-core-native kernel eliminates it remains untested. The Tier-2 requirement doubles total memory footprint relative to Tier-1 alone. There is a group-size cliff for INT4 values at $[K_{\min}{=}2, K_{\max}{=}128]$6 ($[K_{\min}{=}2, K_{\max}{=}128]$7 near-lossless, $[K_{\min}{=}2, K_{\max}{=}128]$8 catastrophic at +26.4 Δppl) with no intermediate operating point, and INT8 values are not implemented in the kernel. The value-promotion policy enforces no global error budget — greedy budget enforcement is noted as a straightforward extension but not adopted. Finally, the operating regime is narrow: the system's value proposition is 64K+ inference where dense KV strains VRAM and quality regressions are unacceptable; below that, fixed scratch overhead dominates.

Conclusion

This work reframes KV cache quantization as a runtime-verified computation: each attention computation is either accompanied by a computed error bound relative to an FP16 reference or replaced by the exact dense output through a deterministic fallback path. The formal contribution — independent, runtime-computable bounds on key-distribution distortion and value reconstruction error, coupled to adaptive per-block precision selection — is validated empirically by the stark contrast between the certified system (parity with dense on PG-19 and NIAH, near-parity on RULER at 64K+) and the naive baseline with identical quantization but no certification (near-total retrieval collapse). The honest accounting of what remains unresolved — composition of per-step bounds into end-to-end guarantees, the kernel-path numerical gap, the latency overhead dominated by orchestration rather than attention itself, and the confined 64K+ operating regime — delineates the specific questions subsequent work on certified compressed-domain inference must address.