CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving

Abstract: Prefix caching avoids prefill only when a repeated request returns to a server that still holds the prefix KV. Cache-blind balancing disperses that reuse; fixed affinity preserves it but can overload a server. CacheRoute resolves this tradeoff with a periodic routing plan. It admits high-rate keys to a stable warm set and places their assignments by expected load. Hot keys may use more than one destination, although every key in our primary semi-synthetic aggregate uses exactly one. On Llama-3.3-70B in fp8 across 60 H100 GPUs, CacheRoute sustains 176+/-11 QPS at a 3.5-s p99 SLO, 2.3x the strongest of five baselines. Served KV-cache hit rate rises from 64.1+/-1.3% under cache-blind balancing to 93.2+/-0.5%. A second semi-synthetic aggregate and controlled 8B and burst experiments separate the effects of affinity and placement. Two 32B workloads provide the counterexamples: when affinity recovers too little KV work, its residual load skew reduces or erases the improvement. We therefore recommend gating any deployment with a shadow replay rather than enabling affinity from workload statistics alone.

• The paper introduces a periodic, rate-aware routing planner that combines stable prefix affinity, load-sized destination assignments, and longest-processing-time-first placement to balance cache reuse with queue stability.
• CacheRoute achieved a 93.2% KV-cache hit rate and sustained 176 ± 11 QPS at a 3.5-second p99 SLO on a 70B model, delivering 2.3× the capacity of the strongest baseline and 4.2× Flat-LB.
• The paper shows that affinity is workload-dependent: it can reduce capacity when reusable prefixes are small or skewed, so deployments should use shadow replay gates rather than rely on hit-rate or analytic cache-residency predictions alone.

CacheRoute addresses a specific tension in large-scale LLM serving: prefix caching only pays off when recurring requests return to the server that still holds their KV state, but cache-blind load balancing disperses that reuse, while naive affinity concentrates skewed traffic onto overloaded servers. The paper proposes resolving this with a periodic, rate-aware routing plan rather than per-request reactive decisions, and it grounds the proposal in an unusually careful hardware evaluation that includes negative results and a failed analytic predictor (2608.19677).

Problem setting and workload

The target scenario is a multi-tenant conversational assistant in which each tenant carries a stable, reusable context across multi-turn conversations. The evaluation uses semi-synthetic aggregates derived from de-identified operational telemetry: the primary distribution contains 128,824 opaque business keys with a Gini coefficient of 0.756, where roughly 4% of keys account for 47% of requests but no key exceeds 0.3% share. Prompts average about 1.2K input tokens, but the routing-relevant stable segment is only the per-business context (~180 tokens, ~15% of the prompt); a shared template warms everywhere, and retrieved few-shot blocks break exact-prefix reuse in 67% of consecutive pairs. This decomposition matters: the router's opportunity is confined to a small fraction of prefill work, which foreshadows why affinity can fail on other distributions.

The paper quantifies both failure modes. Under uniform spraying across $R$ destinations, mean revisit time scales as $R/\lambda_b$, so scaling out the fleet can make prefixes colder despite growing total capacity. Pure consistent hashing restores locality but maps key skew directly onto queues, so tail latency follows the hottest destination. Reactive cache-aware policies (Preble-style, DualMap-style) spill requests to cold destinations once caps bind.

Design

CacheRoute separates a periodic offline plan from fast dispatch. Given per-key rates $\lambda_b$, $R$ destinations, a per-destination load cap $q_{\mathrm{cap}}$ calibrated from a single-destination latency knee, and a warm-slot allocation $C = RW$, the planner (1) assigns each key $$k_b = \max(1, \lceil \lambda_b / q_{\mathrm{cap}} \rceil)$$ destinations as a load-control rule, (2) admits keys in decreasing rate while $\sum k_b \le C$, and (3) places each admitted job via longest-processing-time-first list scheduling against expected load, initialized with cold-tail shares. During the control interval, admitted traffic picks the least-loaded member of its fixed destination set; unadmitted traffic falls back to power-of-two choices. The planner runs off the request path in 345 ms at $R=30$ for 128,824 keys.

Two design caveats are stated explicitly. First, CacheRoute offers no analytic residency guarantee: admitted and cold-tail prefixes share the engine's native eviction policy, so cache residency is measured, not planned. Second, the equal-slot admission model assumes roughly uniform reusable-prefix sizes; heterogeneous long contexts would require byte-aware admission, which the paper does not implement. Notably, in the primary distribution every key satisfies $\max_b \lambda_b < q_{\mathrm{cap}}$, so $R/\lambda_b$0 throughout—the flagship result measures admission plus stable single-copy affinity plus balanced placement, with replication exercised only in synthetic-whale mechanism studies.

Flagship results

The primary experiment serves Llama-3.3-70B in fp8 across 30 tensor-parallel-2 destinations (60 H100 GPUs), comparing six policies under paired seeds with capacity defined as the highest offered-QPS ladder point meeting p99 ≤ 3.5 s and ≤5% failures.

Policy	KV hit	[email protected]	p99@100 QPS
Flat-LB	64.1±1.3%	42±20	5.7 s
Sticky hashing	87.3±2.4%	30	8.5 s
CHWBL	75.6±0.8%	64±11	3.8 s
DualMap-style	88.7±1.9%	58±22	5.3 s
Preble-style	72.0±0.7%	76±11	3.8 s
CacheRoute	93.2±0.5%	176±11	1.8 s

The headline claim is that CacheRoute sustains $R/\lambda_b$1 QPS at the 3.5-s SLO, $R/\lambda_b$2 the strongest baseline (Preble-style) and $R/\lambda_b$3 Flat-LB. At a tight 2-s SLO, CacheRoute is the only policy passing any tested load (120 QPS). The mechanism explanation is that CacheRoute occupies the useful middle ground: sticky and DualMap recover high reuse (87–89% hit) but suffer queue-dominated tails, while Preble and CHWBL keep queues even but lose reuse (72–76%). A wider top-$R/\lambda_b$4256 confirmation with eight seeds narrows the advantage over DualMap to $R/\lambda_b$5, and on a second independent distribution the three balanced cache-aware policies tie at top-$R/\lambda_b$6128, with CacheRoute reaching $R/\lambda_b$7 only once the active set outgrows the warm allocation at top-$R/\lambda_b$8256. These are honest scope statements: the multiplier depends on active-set breadth relative to warm capacity.

Mechanism ablation

An 8B testbed isolates locality from balance using injected synthetic whales. Affinity alone raises served KV hit from 56% to 88% but worsens load imbalance to $R/\lambda_b$9, leaving capacity unchanged at 240 QPS; adding load-proportional replication reduces imbalance to $\lambda_b$0 without helping; only LPT placement, bringing imbalance to $\lambda_b$1, moves the knee past the 500-QPS sweep ceiling. The conclusion is that affinity recovers cached work but placement makes it usable at saturation—both components are necessary. Across 8B active-set sizes, CacheRoute never falls below Flat-LB, though several knees are right-censored at the sweep maximum and should not be read as equality.

Operating envelope and negative results

The paper's most valuable contribution may be its boundary conditions. On two Qwen3-32B aggregate workloads, affinity raises KV hit by only 8–11 percentage points; on workload A this residual skew reduces capacity to $\lambda_b$2–$\lambda_b$3 Flat-LB, and on workload B the policies tie. Model size alone does not distinguish outcomes. Consequently, the authors recommend gating any deployment with a shadow replay—at one load below and one near the current knee, requiring p99 or capacity improvement rather than merely a hit-rate increase—and re-running the gate after any change to model, precision, batching, or cache allocation.

Supporting this gate, an analytic single-characteristic-time occupancy model fails badly against the 70B fp8 engine despite passing seven instrumentation isolation checks: prediction misses served hit rate by 14.3 points at the median and 44.7 points at p90, with an observed drop-then-plateau residency curve outside the tested model family. Physical block counts and aggregate rates are therefore insufficient for pre-deployment sizing on this engine.

Burstiness and replanning sensitivities are measured but scoped narrowly. Gamma arrivals matched to CV=1.9 cost CacheRoute one ladder step (180→160) while Flat-LB stays at 30; a block-bootstrap trace with empirical CV=2.73 leaves CacheRoute unchanged. Under lognormal rate drift ($\lambda_b$4), recomputing LPT from scratch changes 94.5% of assignment sets while only 1.1% of keys change count; stale plans cost about 1 point of KV hit and 192 ms of p99, whereas installing fresh plans causes a transient 13.6-point rewarming dip. The operational guidance—preserve placements until measured staleness penalty exceeds warm-up penalty—is sound, but no churn-aware replanner is implemented.

Limitations

The paper is candid about scope. Baselines are common-harness reimplementations, not reproductions of the original Preble, DualMap, or CHWBL systems, so absolute comparisons reflect routing behavior under matched hardware only. Several 8B knees are right-censored; three-seed studies carry wide Student-$\lambda_b$5 intervals. Equal-slot admission ignores heterogeneous prefix sizes, cold-tail traffic can evict admitted prefixes, and two aggregate distributions cannot represent all markets. The fleet-size sweep shows non-monotonic behavior at fixed absolute load (at $\lambda_b$6, targeted prefixes go cold), which limits generalization to deployments whose load scales with fleet size. Timestamp-exact replay is left open, as is a churn-aware replanning algorithm and a byte-aware admission value function.

Conclusion

CacheRoute demonstrates that a periodic, rate-planned routing table—top-rate admission, load-sized assignment counts, and LPT placement—can simultaneously achieve high served KV-hit rate and balanced queues, yielding $\lambda_b$7 the SLO capacity of the strongest baseline on a 70B fp8 fleet. Equally important, the measured loss and tie regimes establish that prefix affinity is not universally beneficial: when the recoverable key-specific prefix work is small, residual skew dominates. The paper's durable recommendation is procedural rather than algorithmic—validate any candidate plan with shadow replay before enabling it, since neither workload statistics nor analytic residency models predict the outcome reliably.