---
title: 'SGLANG-LSM: Scalable KV-Cache for LLMs'
url: https://www.emergentmind.com/topics/sglang-lsm
type: topic
---

# SGLANG-LSM: Scalable KV-Cache for LLMs

Searching arXiv for the relevant SGLANG-LSM papers and surrounding context.
SGLANG-LSM denotes a database-inspired KV-cache management system for large language model inference that applies Log-Structured Merge-tree design to disk-resident KV caches in the SGLang ecosystem. It was introduced to address the scalability collapse of file-per-object KV-cache layouts, which incur substantial file-system metadata overhead, random I/O, and poor spatial locality once KV state spills beyond memory. The system combines a prefix-preserving storage engine, an adaptive controller for LSM-tree configuration, and runtime services for batch operation and automatic resource management, with the stated objective of scaling disk-resident KV caches to hundreds of millions of tokens while improving cache hit rate and reducing time-to-first-token (TTFT) latency [2511.16138]. In the broader SGLang stack, this design is naturally interpreted as an extension of the RadixAttention-based KV-cache reuse mechanisms introduced by SGLang’s runtime, which already exploits prefix structure for reuse across inference requests [2312.07104].

## 1. Origin and problem setting

SGLANG-LSM emerged from the observation that modern LLM inference systems can be highly optimized on GPU yet still encounter a severe scalability wall when KV cache spills to local disk. In the characterization given for the system, traditional file-per-object layouts are described as suffering from “massive metadata overhead, random I/O, and shattered spatial locality,” which degrades cache effectiveness and inference latency under large-scale workloads [2511.16138].

The motivating context is SGLang, a system for efficient execution of structured language model programs that couples a frontend language with a high-performance runtime, SGVM. That runtime includes RadixAttention, which maintains prefix-structured KV-cache state and reuses shared prefixes across requests, yielding substantial throughput gains on prefix-sharing workloads [2312.07104]. SGLANG-LSM addresses a complementary systems problem: preserving prefix-aware reuse and efficient lookup when the KV cache is too large to remain solely in GPU or CPU memory and must be managed on local flash storage [2511.16138].

A plausible implication is that SGLANG-LSM should be understood less as a standalone inference engine than as a storage substrate for production-scale KV reuse in SGLang-like serving pipelines. That interpretation is consistent with the paper’s service interface being explicitly aligned with “exactly the operations RadixAttention expects” [2511.16138].

## 2. Architectural decomposition

The system is organized as a three-layer architecture comprising a prefix-preserving storage engine, an adaptive controller, and runtime services [2511.16138].

| Component | Stated role | Key mechanisms |
|---|---|---|
| Prefix-Preserving Storage Engine | Maintain token-sequence locality while storing large KV tensors efficiently | LSM-tree index, key-value separation, append-only tensor-log |
| Adaptive Controller | Optimize LSM-tree configuration under changing workloads | Sliding-window workload monitoring, cost model, lazy parameter transitions |
| Runtime Services | Support production deployment | Batch compression/decompression, automatic tensor-file merging, concurrency tuning |

The prefix-preserving storage engine uses an LSM-tree index whose key encoding ensures that lexicographic order coincides with prefix order. This enables range scans over a prefix to return all of its extensions, which is central to prefix-oriented KV reuse [2511.16138]. The design further separates metadata from tensor payloads: the LSM index stores only fixed-size records of the form $\langle\mathit{encoded\_prefix},\mathit{file\_id},\mathit{offset}\rangle$, whereas the large KV tensors reside in an append-only tensor-log. The tensor sizes are given as 40–120 KB [2511.16138].

The adaptive controller addresses workload nonstationarity. The paper distinguishes “population” phases dominated by cache inserts from “serving” phases dominated by reads and probes, and formulates a total I/O cost objective
$$
C(T,K) = w\cdot W(T,K) + s\cdot S(T,K) + r\cdot R(T,K) + z\cdot Z(T,K),
$$
where $w,s,r,z$ are observed workload fractions for writes, successful reads, scans, and empty probes, respectively. The controller solves
$$
\min_{T,K} C(T,K)
$$
over the size ratio $T$ and max-runs parameter $K$, and applies changes lazily during subsequent compactions rather than by a disruptive reconfiguration step [2511.16138].

The runtime-services layer adds operational features needed for production deployment: batch tensor compression and decompression, background tensor-file merging once file counts exceed a threshold, and concurrency tuning using multi-threaded RocksDB I/O together with asynchronous flush and compaction [2511.16138].

## 3. Prefix-preserving storage engine

The storage engine’s defining principle is key-value separation. The LSM index stores metadata only, while bulk tensors are written once into append-only log files. The paper states that compaction never rewrites the tensor payload $s$, so tensor-log write amplification is effectively $1\times$ [2511.16138]. This is a direct response to the inefficiency of rewriting large KV objects during conventional compaction or file-level reorganization.

Prefix locality is encoded in the key space itself. Keys are byte-encodings of token sequences augmented by a sentinel such that lexicographic order equals prefix order. Consequently, a range scan starting at prefix $P$ returns all keys corresponding to extensions of $P$ [2511.16138]. This is the storage-side counterpart of prefix matching in RadixAttention, which searches for the longest matching token prefix in a radix tree before extending the cache with only the nonshared suffix [2312.07104].

For a `put_batch` of tokens $[t_1,\dots,t_k]$, the engine serializes $\{K_i,V_i\}_{i=1}^k$ into the tensor-log at a single offset and writes a RocksDB `WriteBatch` containing the corresponding prefix-to-location metadata records [2511.16138]. The service interface exposes `put_batch`, `probe`, and `get_batch`, reflecting the operational pattern of batch insertion, existence testing, and batch retrieval expected by a prefix-sharing inference runtime [2511.16138].

The paper provides several analytical cost expressions for the LSM-tree component. The number of levels is modeled as
$$
L = \log_T \frac{N\cdot e}{M},
$$
where $N$ is the number of entries, $e$ is the average metadata size, and $M$ is the memtable size [2511.16138]. Update amplification is given by
$$
\mathit{WA}(T,K)=\frac{T\,L}{B\,K},
$$
with $B$ the page size and $K$ the max runs per level, reflecting amortized sequential writes [2511.16138]. For point lookups, the cost of an absent key is stated as
$$
\mathrm{Cost}_{\rm miss} = O\bigl(K\,L\,p\bigr),
$$
using Bloom filters with false-positive rate $p$, while a hit incurs one extra I/O:
$$
\mathrm{Cost}_{\rm hit}=O(KLp+1).
$$
These expressions frame the controller’s optimization objective and clarify why index-only compaction, Bloom filters, and prefix-preserving ordering are central to the design [2511.16138].

## 4. Adaptive control and online reconfiguration

The adaptive controller is designed for workload regimes in which the optimal LSM parameters shift over time. The paper’s algorithm sketch maintains counters over a sliding window, estimates the observed fractions of writes, successful reads, scans, and empty probes, and triggers re-optimization when the current workload mix diverges from the previous window by more than a threshold [2511.16138].

The search space described in the paper is a small grid: $T\in\{2,4,8\}$ and $K\in\{1,\dots,T-1\}$. For each candidate pair $(T',K')$, the controller evaluates the cost model $C(T',K')$, chooses the minimizer, and then performs a lazy transition to the new settings [2511.16138]. The laziness is explicit: rather than rebuilding the store, the system incrementally adjusts $T$ at each flush and changes $K$ during compaction, without halting service.

This design reflects a database-style control loop rather than a fixed storage policy. The stated rationale is that LLM workloads “ebb and flow,” alternating between write-heavy cache population and read-heavy serving phases [2511.16138]. A plausible implication is that the controller’s value lies not only in reducing average cost, but also in preventing the cache layer from being tuned for the wrong phase for long periods, which would otherwise degrade TTFT and flash endurance.

The paper also notes an important limitation: because transitions are lazy, it takes several compaction cycles to fully realize a new $(T,K)$ configuration, so short-term misconfigurations can persist [2511.16138]. That caveat is materially relevant for bursty workloads whose phase duration is shorter than the reconfiguration horizon.

## 5. Runtime services and integration with SGLang

The runtime-services layer contains three classes of mechanisms: batch operations, automatic resource management, and concurrency control [2511.16138].

Batch operations are exposed through `put_batch` and `get_batch`. `put_batch` groups multiple token writes into one `WriteBatch` and one tensor block; `get_batch` performs a single range scan followed by scatter-gather I/O [2511.16138]. Compression is applied to the entire tensor block rather than to individual tokens. The paper attributes “50–75% storage savings” to whole-block compression, citing FlexGen in the system description [2511.16138].

Resource management includes automatic tensor-file merging as a low-priority background job once file count exceeds a threshold. Importantly, only the small index metadata needs to be updated when such merging occurs [2511.16138]. This preserves the key-value separation principle and avoids rewriting the index at the scale of the tensor payload. RocksDB thread pools handle compactions and flushes asynchronously, tuned to keep I/O queues full [2511.16138].

Under concurrency, probe and read threads can share the same RocksDB instances, with isolation via column families, one per LLM model, “avoiding head-of-line blocking” [2511.16138]. This detail situates SGLANG-LSM in a multi-model serving environment rather than a single-model prototype.

Its relationship to SGLang is structurally significant. SGLang’s runtime already dispatches generation calls through SGVM, performs batching, and relies on prefix-aware scheduling and RadixAttention for KV-cache reuse [2312.07104]. SGLANG-LSM can therefore be understood as extending the prefix-aware philosophy from in-memory radix-tree organization to disk-resident KV persistence. This suggests continuity rather than replacement: SGLANG-LSM manages scale-out KV persistence, while RadixAttention remains the in-memory reuse primitive [2511.16138; 2312.07104].

## 6. Empirical evaluation, trade-offs, and naming ambiguity

The reported evaluation uses an Intel Xeon 4314 with 16 cores, 64 GB RAM, an NVIDIA A30 with 24 GB, and an 8 TB PCIe4 SSD [2511.16138]. The models are GLM-4-8B, GLM-4-32B, and Llama-3-8B, with KV footprints of 40 KB, 60 KB, and 120 KB per token, respectively [2511.16138]. The workload is a synthetic 10-stage hitting-rate sweep from $0.2$ to $0.7$, with 1,000 requests per stage, prompt lengths of 4k, 8k, and 16k tokens, and a warm-up of 100 M tokens [2511.16138].

The baselines are `SGLang(memory)`, defined as GPU+CPU in-memory KV cache only, and `SGLang(file)`, defined as a file-per-token disk backend [2511.16138]. Against `SGLang(file)`, SGLANG-LSM reports the following headline results:

| Metric | Reported result | Example |
|---|---|---|
| Cache hit rate | up to +143% relative | 45.4% vs. 18.7% at 4k tokens |
| TTFT | reduction up to 24.3% | 1.78 s vs. 2.35 s at 16k |
| Throughput | 10–15% higher token/sec | read-heavy phases |
| Resource use | flash writes drop by 2.5×; CPU cycles on I/O stacks down by 50% | attributed to bounded WA and fewer I/O costs |

The paper also describes figure-level behavior: in the hit-rate sweep, the SGLANG-LSM curve remains above the file backend by 20–30 percentage points; TTFT curves for 4k, 8k, and 16k prompts remain below both baselines; and with dynamic compaction on GLM-9B/8k, TTFT decreases by up to 14% in write-heavy windows [2511.16138].

Several trade-offs are identified explicitly. Encoding prefixes lexicographically and storing Bloom filters add approximately 10–15 bytes of metadata per entry, but reduce lookup I/O by 2–3× [2511.16138]. Large tensor sizes, such as Llama-3’s 120 KB per token, narrow the recompute–reuse gap, and TTFT gains are said to shrink to approximately 13–15%; if KV payload is much larger than recompute cost, cold-cache reuse is less beneficial [2511.16138]. These statements qualify the generality of the performance claims and indicate that the approach is most advantageous when storage-level reuse meaningfully dominates recomputation.

A separate naming issue arises because another later paper uses “SGLANG-LSM” to denote the deployed incarnation of HT-Moonstone, a 5B spoken language model adapted for multilingual Singaporean Home Team tasks [2607.10092]. That usage refers to a spoken language model deployment blueprint, not to the LSM-tree KV-cache system of “On 10x Better Scalability: KV Stores Scale Up KV Cache” [2511.16138]. The coexistence of these labels suggests that “SGLANG-LSM” is ambiguous across subfields and should be disambiguated by citation or by its expansion in context. Future directions proposed for the KV-cache system include learned cost models, multi-node or distributed LSM for terabyte-scale shared KV caches, and richer query patterns with adaptive range-filter structures [2511.16138].

Source: https://www.emergentmind.com/topics/sglang-lsm