LongCache Mechanism in TokenLake LLMs

• LongCache Mechanism is a segment-level prefix cache pooling system that enhances load balancing and deduplication in TokenLake.
• It divides long-context inputs into fixed-size segments (~500–600 tokens) to enable asynchronous, efficient cache reorganization and resource management.
• The design minimizes communication overhead and load imbalance, achieving throughput improvements up to 2.6× compared to traditional methods.

The LongCache Mechanism refers to the unified segment-level prefix cache pool introduced as part of TokenLake, a system designed for fine-grained, elastic serving of long-context LLMs on GPU clusters. TokenLake's architecture and mechanisms address pathological inefficiencies of traditional prefix caching schemes at cluster scale when serving multi-turn, long-context interactions. By segmenting prefix caches and pooling them asynchronously across peer @@@@1@@@@, TokenLake’s LongCache Mechanism achieves improved load balance, deduplication, defragmentation, and communication minimization, while fully decoupling cache management from the request scheduler (Wu et al., 24 Aug 2025).

1. Motivation and System Architecture

Conventional prefix caching in LLM serving systems employs imperative APIs (put/get/transfer) that couple cache management tightly to request scheduling. This entanglement induces three principal pathologies as context and KV-cache sizes grow:

• Load Imbalance: Hot prefixes concentrate computation and KV-bandwidth on specific GPUs, overloading them.
• Redundancy: Shared prefixes (e.g., common system prompts) must be replicated across many GPUs to maintain compute balance, consuming excess memory.
• Fragmentation: With "strict locality" (one prefix per GPU), memory is divided into small unusable slots, which cannot be coalesced for large prefixes.

TokenLake restructures this model through a single, peer-to-peer, segment-level prefix cache pool. Prefixes are divided into fixed-size segments (typically 500–600 tokens) that may reside on any GPU. A declarative cache interface enables TokenLake to execute reorganization, balancing, deduplication, and defragmentation of cache segments without exposing these operations to the computation scheduler.

2. Declarative Cache Interface and Key Properties

TokenLake introduces two primary abstractions:

• Query tensors ($q$): Per-layer $Q$ projections driving prefix attention.
• Fixed-Size Prefix Cache Segments: Segments of $C$ tokens per segment, with $C \approx 500$–$600$.

The system presents distinct APIs for control and data planes:

• Control-plane: get_prefix_tree, get_cache_load, gen_plans.
• Data-plane: init_query, init_transfer, query, put.

All segments are sized $C$ and address-aligned. Theoretical constraints for segment size ensure efficiency, where

$$k_{\mathrm{comp}} = \max(4d/F, 4d/B_{\mathrm{mem}})$$

$T_{\mathrm{comp}}(C) = k_{\mathrm{comp}} \cdot C$

$$T_{\mathrm{comm}}(C) = 2\alpha_{\mathrm{net}} + \frac{4d}{B_{\mathrm{net}}}$$

with $d$ as hidden size; $C$ is chosen such that

$$C \geq \frac{2\alpha_{\mathrm{net}}}{k_{\mathrm{comp}}} + \frac{4d}{k_{\mathrm{comp}}\cdot B_{\mathrm{net}}} \approx 568$$

3. Segment-Level Cache Operations

Each prefix in a multi-turn interaction is partitioned into $\lceil\mathrm{prefix\_len}/C\rceil$ segments, tracked globally via a directory mapping segment-IDs to the set of hosting GPU instances and last access times.

Key operations (expressed in pseudo-code in the canonical description (Wu et al., 24 Aug 2025)):

• Insertion: Upon generation, segments are registered in the global directory with their hosting instance(s) and access time.
• Lookup: For a given batch, required segments' owners are determined, and an optimal replica is selected for each segment (see Section 4).
• Eviction: When capacity is exceeded, global LRU is invoked—prioritizing eviction of redundant replicas before unique copies.

4. Heavy-Hitter-Aware Load Balancing

A breadth-first search on the global prefix trie identifies "heavy hitter" segments—the top $O(N \log N)$ segments by query access—but only these frequently accessed segments are selectively replicated. Normal segments are mapped via hash-based placement ($$\mathrm{seg\_id} \mapsto \mathrm{hash}(\mathrm{seg\_id}) \bmod N$$).

When selecting from available segment replicas during batch servicing, TokenLake employs the "power-of-two choices" strategy, choosing the less-loaded GPU from two random replicas. The objective is to minimize the variance $\mathrm{Var}(\{L_1, ..., L_N\})$ of per-GPU loads, where $L_k$ is the current load on GPU $k$. Theoretical guarantees (Fan-Lim-Andersen) indicate that absorbing the top $O(N \log N)$ heavy hitters suffices for low load variance with high probability.

5. Deduplication and Defragmentation

The global directory enforces deduplication by identifying and coalescing identical segments via unique IDs. On eviction, TokenLake prefers discarding surplus replicas before removing the last copy. Segment sizing inherently caps internal fragmentation at $<C$ tokens per segment.

Pooling segments cluster-wide means that any available memory slot, regardless of its GPU, can satisfy a segment allocation, eliminating fragmented "islands" of memory. Quantitatively:

• Before: $$\mathrm{frag\_before} = 1 - \frac{\mathrm{max~contiguous~free~on~any~GPU}}{\mathrm{total\_free}}$$
• After: $$\mathrm{frag\_after} \approx \frac{C-1}{\mathrm{total\_free}} \ll \mathrm{frag\_before}$$

6. Communication Minimization Framework

All communication of query tensors and KV-writes occurs at the segment granularity ($\sim4d$ per segment). Communication and local self-attention are overlapped, minimizing end-to-end data transfer.

Assignment of batch processing is formulated as a weighted bipartite matching problem: for each batch node $u_i$ and GPU $v_j$,

$$e(u_i, v_j) = -\left[\sum_{v_k \in Q(u_i), v_k \neq v_j} 4d + \sum_{v_k \in P(u_i), v_k \neq v_j} 4d \cdot N_p(u_i, v_k)\right]$$

where $Q(u_i)$ is the set of GPUs holding required segments, $P(u_i)$ the destination GPUs for KV-writes, and $N_p$ the number of KV-write operations. An $O(N^3)$ Hungarian algorithm computes the assignment that minimizes total communication.

7. Scheduler Decoupling and API Integration

The scheduler within TokenLake interacts only with stateless, high-level APIs (get_prefix_tree, get_cache_load, gen_plans), with all prefix cache placement, replica management, and memory operations abstracted away. For each scheduling iteration, the scheduler:

1. Queries the projected cache load ($L$) via get_cache_load.
2. Adjusts batch sizes and/or parallelism such that $L + \mathrm{compute\_load} \leq 1$.
3. Generates batch-specific query and transfer plans.
4. Invokes fully asynchronous data-plane calls: init_query, init_transfer, query, put.

This design enables elastic request scheduling, unburdened by cache coupling.

8. Performance Outcomes and Outlook

Empirical evaluation on real-world datasets (LooGLE, SCBench, ShareGPT) yields:

• Throughput Speedup: Up to $2.6\times$ (versus SGLang-Router) and $2.0\times$ (versus MoonCake PD disaggregation).
• Hit Rate: $2.0\times$–$2.1\times$ higher relative to both prior approaches.
• Load Imbalance: TokenLake reduces per-GPU coefficient of variation to approximately $11\%$, versus $99\%$ (SGLang-Router) and $62\%$–$122\%$ (MoonCake PD, depending on configuration).

The architectural innovations underpinning the LongCache Mechanism—segment-based global pooling, heavy-hitter-aware load balancing, deduplication, and communication minimization—support near-perfect resource utilization and operational elasticity. Potential extensions include online adaptive segment sizing, hierarchical (multi-tier) pooling (GPU:host), and integration with on-GPU KV-compression or selective overwrite to further accommodate extreme context lengths (Wu et al., 24 Aug 2025).