Hardware validation of low-bit KV-compression latency overhead

Validate the cost-normalized comparison between tensor parallelism and KV-cache compression on physical hardware by measuring the decode-kernel dequantization overhead of low-bit KV-cache compression and determining whether that overhead reverses the reported cost ordering.

Background

The study models the memory benefits of KV-cache quantization and eviction but does not model their kernel-level latency consequences because the authors did not have GPU access. Its sensitivity analysis estimates that compression would need to lose 16.8%, 33.5%, or 47.0% of throughput at the examined relief levels before tensor parallelism became cheaper. The paper therefore presents the cost comparison as robust to published overhead estimates, but treats direct hardware confirmation as unresolved.

Resolving this problem would close the most consequential empirical gap in the comparison: measuring actual low-bit KV decode behavior, including dequantization work and batching interactions, rather than relying on an assumed zero-overhead compression arm. Such measurements would determine whether the simulated cost advantage of KV compression persists in deployed systems.