Joint Cross-Tier Memory-Budget Allocation

Determine whether a single allocation policy can jointly choose prompt-compression ratios, retained or offloaded KV states, the number of retrieved items, and the injection form within a request under the budgeted cost model for long-context inference.

Background

The survey argues that current systems generally use fixed compression ratios, retrieval counts, cache budgets, or summary granularities, even though optimal memory allocation depends on task requirements and available resources. Prompt compression, KV retention and offloading, retrieval, post-retrieval compression, and injection form are also typically optimized separately despite their interactions.

The unresolved problem is to learn or construct a unified policy that makes these decisions jointly while accounting explicitly for inference cost. The survey notes that existing learned memory policies do not include explicit cost terms and do not cover KV-cache or serving-level decisions.