Efficient Kernelization of Output-Aware KV-Cache Eviction

Determine whether output-aware KV-cache eviction approaches that incorporate value representations and quantify attention-output perturbation can be efficiently kernelized for production inference.

Background

The paper discusses recent KV-cache eviction methods that use value representations in addition to attention scores to estimate the perturbation caused by removing cached tokens. These methods formulate eviction as an optimization problem intended to minimize deviation from the original attention output, but the resulting joint token-selection problem is combinatorial and NP-hard.

Although the paper proposes DropKV as a tractable alternative based on decoupled per-token perturbation scores and supplies a fused Triton implementation, it explicitly identifies the efficient kernelization of more intricate output-aware eviction policies as unresolved. Resolving this question would help determine whether such methods can be deployed practically in production inference systems without prohibitive implementation or memory costs.

References

Furthermore, as policy designs become increasingly intricate, it remains unclear whether these approaches can be efficiently kernelized, limiting their viability in production inference.

— DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction  (2610.22130 - Zhang et al., 15 Oct 2026) in Section 1, Introduction, paragraph beginning “More recent methods have sought to transcend attention-score-only criteria”