---
title: 'TinyServe: Lightweight LLM Serving'
url: https://www.emergentmind.com/topics/tinyserve
type: topic
---

# TinyServe: Lightweight LLM Serving

Searching arXiv for the cited TinyServe and related serving papers to ground the article and attach arXiv IDs.
TinyServe is a lightweight and extensible serving system for deploying tiny LLMs such as TinyLLaMA and GPT-2 variants, with an emphasis on reducing the memory and latency overhead of key–value (KV) cache access during autoregressive decoding through structured KV sparsity, query-aware cache selection, and hardware-efficient attention kernels [2509.12211]. In the system presented under that name, the central idea is to reorganize the decode-time KV path around page-level selection driven by the current query vector, rather than assuming dense attention over all stored KV. This places TinyServe at the intersection of LLM systems, sparse attention, and runtime cache management for resource-constrained hardware. The term “TinyServe” also appears informally in a distinct sense in later work on single-GPU agent serving, where it denotes a small-footprint serving stack for agentic AI; that usage corresponds to a different system design problem and is exemplified by AgentServe rather than by the query-aware sparse serving framework introduced as TinyServe [2603.10342].

## 1. Definition and problem setting

TinyServe addresses the decode-phase bottleneck in transformer inference. In the formulation given for transformer-based LLMs, inference has a prefill phase, in which the entire prompt is processed and the resulting KV vectors are stored, and a decode phase, in which each new query vector $q_t \in \mathbb{R}^d$ attends over all previous keys and values in the cache [2509.12211]. The decode attention is written as

\[
\text{Attn}(q_t, K, V) = \sum_{i=1}^{t-1} \text{softmax}(q_t^\top k_i) \cdot v_i .
\]

The stated motivation is that decode latency dominates because each step repeatedly scans a growing KV cache, and long contexts of 4K–32K tokens make decode time and KV access the main bottlenecks even for tiny models with 125M–350M parameters [2509.12211]. The system therefore targets three coupled issues: memory bandwidth pressure from repeatedly reading large KV state from HBM, latency from the $O(t)$ growth of dense attention per step, and KV cache size pressure under long-context or multi-user serving [2509.12211].

This framing distinguishes TinyServe from serving stacks that optimize memory layout and batching but still assume dense attention over all stored KV. The paper explicitly positions prior systems such as vLLM, TGI, FasterTransformer, and TensorRT-LLM in that category, and contrasts TinyServe with fixed-window or heuristic pruning approaches that are not query-dependent [2509.12211]. A plausible implication is that TinyServe is best understood not merely as a serving runtime for small models, but as a systems research platform in which cache access itself becomes a first-class optimization target.

## 2. Architectural organization

TinyServe is presented as a lightweight serving system designed around tiny LLMs including TinyLLaMA-125M, GPT2-345M, OPT-350M, GPT2-774M, and up to LLaMA-1.3B [2509.12211]. Its architecture is organized around three core components: a Query-Aware KV Retriever, a Modular Scheduling Pipeline, and a Sparse Attention Executor [2509.12211].

The Query-Aware KV Retriever dynamically selects relevant KV “pages” at decode time based on the current query and per-page metadata, thereby implementing structured KV sparsity [2509.12211]. The Modular Scheduling Pipeline exposes a dispatch loop through which incoming requests can traverse configurable plugins, including entropy-based early exit, pruning policies, approximate attention, and custom token selection strategies, without changing the model architecture [2509.12211]. The Sparse Attention Executor provides fused CUDA kernels that implement page scoring using metadata, sparse KV gather, and masked attention, and supports FP16 / INT8 formats together with multi-GPU dispatch [2509.12211].

The per-step request flow is specified at a fairly fine granularity. A request manager collects active sessions and batches them subject to timeout and capacity constraints; the model forward computes the new query $q_t$ for each sequence; the TinyServe KV path performs query-aware page scoring, top-$K$ page selection, sparse gather from HBM, and attention only over gathered KV; token selection is handled through a plugin interface operating on logits; and the KV cache manager organizes the cache into fixed-size pages, updates per-page metadata, and handles cache reuse across steps and sessions [2509.12211]. The system also includes fine-grained instrumentation for latency breakdown, page reuse, memory bandwidth utilization, and session behavior [2509.12211].

This architecture is notable because it treats token selection policy and cache selection policy as orthogonal modules. That separation allows experimentation with sampling, early exit, and sparsity mechanisms in one framework, while preserving the claim that no model modifications are required [2509.12211].

## 3. Structured KV sparsity and page-based memory layout

TinyServe partitions the KV cache for an attention layer into fixed-size pages. If $K \in \mathbb{R}^{L \times d}$ and $V \in \mathbb{R}^{L \times d_v}$, with $L$ the number of tokens in context, then the sequence is split as

\[
K = \bigcup_{j=1}^{P} \mathcal{K}_j, \quad P = \left\lceil \frac{L}{S} \right\rceil ,
\]

where page $j$ contains a fixed-size block of keys and corresponding values [2509.12211]. Each page is stored as a contiguous memory block in HBM, which the paper describes as aligned with paged attention but repurposed for sparse access rather than merely efficient dense access [2509.12211].

The key departure from dense attention is that TinyServe computes attention only over a subset of pages $\mathcal{S}_t \subseteq \{1,\dots,P\}$ selected for the current query. The mechanism is defined in three steps: compute a relevance score $r(q_t,\phi(\mathcal{K}_j))$ for each page using page metadata, select top-$K$ pages, and perform attention only over keys and values in those selected pages [2509.12211]. The resulting sparse attention is expressed as

\[
\text{SparseAttn}(q_t) = \sum_{j \in \mathcal{S}_t} \sum_{k_i \in \mathcal{K}_j} \text{softmax}(q_t^\top k_i) \cdot v_i .
\]

The paper characterizes this as structured sparsity at the page level: the kept units are tokens in top-$K$ pages and the dropped units are tokens in pages with low relevance scores [2509.12211]. The structure matters because it makes memory access page-aligned and therefore suitable for fused kernels, unlike unstructured token-by-token pruning.

TinyServe also provides an approximation-error perspective. It defines

\[
\epsilon = \max_{k \in \mathcal{K}_j} q_t^\top k - r(q_t, \phi(\mathcal{K}_j))
\]

and states that under assumptions on key distributions with variance $\sigma^2$ and page size $S$,

\[
\mathbb{E}[\epsilon] \leq \frac{d \cdot \sigma^2}{S} \cdot \sqrt{\log(S)} .
\]

From this, the paper concludes that larger page size $S$ can increase approximation error, while smaller pages improve fidelity at the cost of more metadata and scoring [2509.12211]. Empirically, it chooses $S=16$ with selection ratio $K/P \approx 0.3$ to keep accuracy drops negligible while achieving large memory savings [2509.12211]. This suggests a systems trade-off between metadata overhead and retrieval fidelity that is intrinsic to the page abstraction itself.

## 4. Query-aware page selection

The defining mechanism in TinyServe is query-aware page selection based on bounding-box metadata [2509.12211]. For each page $\mathcal{K}_j$, the system stores channel-wise minima and maxima of key components,

\[
\phi(\mathcal{K}_j) = (m_j, M_j) \in \mathbb{R}^{2d},
\]

with $m_{j,i} = \min_{k \in \mathcal{K}_j} k_i$ and $M_{j,i} = \max_{k \in \mathcal{K}_j} k_i$ [2509.12211]. This metadata is described as a bounding box in $\mathbb{R}^d$ that cheaply summarizes all keys in the page, is stored in faster memory such as SRAM / L2, and is updated when new tokens are appended [2509.12211].

The scoring function is a directional bounding-box estimator:

\[
r(q_t, \phi(\mathcal{K}_j)) = \sum_{i=1}^d \begin{cases}
q_{t,i} \cdot M_{j,i}, & \text{if } q_{t,i} \ge 0 \\
q_{t,i} \cdot m_{j,i}, & \text{if } q_{t,i} < 0
\end{cases}
\]

[2509.12211]. The interpretation given is dimensionwise: when $q_{t,i}$ is positive, the largest possible contribution from that coordinate comes from the page maximum; when $q_{t,i}$ is negative, it comes from the page minimum [2509.12211]. Summing across coordinates yields an upper bound or approximation to the maximum dot product within the page, which functions as a cheap proxy for the strongest attention logit in that page [2509.12211].

Page selection then proceeds by computing all page scores in parallel on GPU, applying top-$K$ selection, and gathering the selected pages from HBM [2509.12211]. The paper notes that threshold-based selection could also be used, but focuses on top-$K$ with configurable ratio $K/P$ [2509.12211]. Because the selected subset is query-dependent, different decode steps may access different cache regions even within the same sequence.

The paper further gives a simple latency model. If each page fetch from HBM costs $\tau_{\text{hb}} \cdot S$, metadata is in L2/SRAM with negligible $\tau_{\text{meta}}$ cost, and page scoring is fused into one kernel, then decode-step latency is decomposed as

\[
\text{Latency}_t =
\underbrace{\tau_\text{meta} \cdot P}_{\text{metadata scan}}
+
\underbrace{\tau_\text{hb} \cdot K \cdot S}_{\text{KV load}}
+
\underbrace{\tau_\text{attn}(K \cdot S)}_{\text{attention computation}} .
\]

The intended significance is clear: when $K/P$ is substantially below 1, both KV loading and attention computation contract relative to full-cache attention [2509.12211].

## 5. Fused kernel design, token selection, and instrumentation

TinyServe’s kernel-level design fuses four stages into a single CUDA kernel: page scoring from metadata, top-$K$ selection, sparse KV gather, and attention computation [2509.12211]. The summarized pseudocode in the paper shows a single pass from query $q_t$ and page metadata $\{\phi_j\}$ to scores $\{s_j\}$, selected pages $S_t$, fetched sparse KV tensors, attention logits $a_i$, and final output $o_t$ [2509.12211]. This fusion is described as avoiding multiple kernel launches and intermediate buffers, reducing CPU–GPU synchronization, and avoiding repeated memory traversal [2509.12211].

Several hardware-oriented choices are emphasized. Keys and values within a page are contiguous, enabling coalesced HBM loads; metadata is stored in L2/shared memory; top-$K$ can be implemented with a shared-memory heap or block-level parallel reduction; and page size $S$ is chosen to align with warp sizes or tensor-core-friendly matrix sizes [2509.12211]. The overall design goal is to reduce the bandwidth bottleneck by lowering the number of KV tokens processed per step [2509.12211]. The paper reports that memory bandwidth usage stays well below the HBM limit with smoother access patterns relative to dense baselines [2509.12211].

TinyServe also includes a plugin interface for token selection at each decode step. Plugins can implement greedy decoding, top-$k$ sampling, nucleus sampling, entropy-based early exit, and custom policies [2509.12211]. After sparse attention and final-layer computation, logits are passed to the plugin, the selected token is appended, and the corresponding new KV is added to the cache while metadata is updated [2509.12211]. This modularity is important because it lets the system vary serving policies and cache policies independently.

Instrumentation is an explicit part of the platform. TinyServe logs latency breakdowns across prefill and decode, and within decode across page scoring, top-$K$ selection, KV gather, and attention; it tracks KV hit rate per page, page reuse across decode steps, token eviction and page migration; and it records SM utilization, memory bandwidth usage over time, cross-request cache reuse, and session migration overhead [2509.12211]. The overhead of instrumentation is reported as less than 2% for training-related hooks [2509.12211]. This instrumentation focus reinforces the system’s role as a research framework rather than solely a production runtime.

## 6. Empirical performance and scaling behavior

The evaluation covers TinyLLaMA-125M, GPT2-345M, OPT-350M, GPT2-774M, and LLaMA-1.3B, with tasks spanning language modeling, long-range benchmarks, reasoning benchmarks, and multi-user serving workloads [2509.12211]. Baselines include vLLM, TGI, TensorRT-LLM, StreamingLLM, SnapKV, PyramidKV, and pruning baselines such as FullCache, SoftPrune, and EntropyStop [2509.12211]. The hardware configuration reported is 8 × NVIDIA A100 80GB GPUs with FP16 inference, CUDA 11.8, cuDNN 8.7, PyTorch 2.0.1, and Ubuntu 20.04 [2509.12211].

The headline result is that TinyServe achieves up to 3.4x speedup and over 2x memory savings with negligible accuracy drop [2509.12211]. The detailed examples include the following:

| Model / context | Latency | Memory |
|---|---:|---:|
| TinyLLaMA-125M, 4K | 25.1 ms → 11.9 ms | 2.1 GB → 1.2 GB |
| GPT2-345M, 8K | 45.2 ms → 20.1 ms | 4.8 GB → 2.1 GB |
| GPT2-774M, 16K | 89.2 ms → 41.2 ms | 8.9 GB → 4.1 GB |
| LLaMA-1.3B, 32K | 156.8 ms → 72.8 ms | 15.8 GB → 7.2 GB |

These figures are all reported relative to FullCache, with approximate speedups of roughly 2.1x–2.25x in the tabulated settings and larger gains in longer-context or more aggressive sparsity configurations [2509.12211].

The paper further states that TinyServe often improves accuracy slightly relative to FullCache on LongBench examples, including 54.2 → 55.2% for TinyLLaMA-125M, 61.7 → 62.8% for GPT2-345M, and 68.9 → 70.2% for LLaMA-1.3B [2509.12211]. Since the general framing emphasizes “negligible accuracy drop,” these increases are treated as empirical observations rather than as a guaranteed effect [2509.12211]. A plausible implication is that query-aware sparsity may sometimes suppress distracting or low-value context regions, but the paper does not generalize that into a universal claim.

Cache reuse is also reported as strong. TinyServe typically achieves greater than 95% KV hit rate in experiments, with examples of 96.2% for TinyLLaMA-125M and 95.4% for GPT2-345M [2509.12211]. In multi-user serving for GPT2-345M with 1024 concurrent requests, a TinyServe-modified vLLM configuration is reported to reduce P50 latency from 45.2 ms to 32.1 ms and increase throughput from 18.4 req/s to 28.6 req/s, while reaching 91.2% GPU utilization [2509.12211].

Scaling results for GPT2-345M at sequence length 16K and batch size 128 are near-linear: 0.81 tokens/ms on 1 GPU, 1.58 tokens/ms on 2 GPUs, 3.123 tokens/ms on 4 GPUs, and 6.221 tokens/ms on 8 GPUs, corresponding to efficiencies from 98.0% to 96.0% relative to baseline [2509.12211]. The paper interprets this as evidence that fused sparse attention and query-aware page selection impose minimal extra communication overhead under standard tensor/model parallel coordination [2509.12211].

## 7. Relation to prior systems, scope, and limitations

TinyServe is explicitly compared to both production serving systems and sparsity-oriented research methods [2509.12211]. Relative to vLLM, whose PagedAttention organizes KV cache into pages for efficient dense attention, TinyServe turns pages into query-ranked units by integrating query-aware page selection into the PagedAttention kernel [2509.12211]. Relative to TGI, TensorRT-LLM, and FasterTransformer, the distinguishing claim is that those systems focus on kernels, quantization, and batching but do not explicitly implement query-conditioned KV sparsity [2509.12211]. Relative to StreamingLLM, SnapKV, and PyramidKV, TinyServe is positioned as query-aware, metadata-based, and retraining-free, rather than relying on fixed windows, clustering structures, or compression [2509.12211].

The system’s scope is also sharply defined. It is designed primarily for tiny to mid-scale models from 125M to 1.3B parameters, and although it is integrated into vLLM, the paper states that direct evidence for 7B+ models is not presented [2509.12211]. Its scoring mechanism assumes that the maximum dot product per page can be approximated well by min/max bounding-box metadata; the paper notes that approximation may degrade if attention patterns are highly non-linear or keys within a page are very diverse, that is, when $\sigma^2$ is high [2509.12211]. It also fixes page size $S$, observes the trade-off between large and small $S$, and does not explore dynamic or adaptive page sizing [2509.12211]. Finally, the fused kernel is tuned for NVIDIA GPUs and CUDA, so portability to TPUs or custom ASICs would require reimplementation [2509.12211].

A separate source of ambiguity concerns the name itself. In AgentServe, “TinyServe” is used descriptively to mean “a very efficient agent-serving stack on one consumer GPU,” and the discussion there centers on phase-aware scheduling for cold prefills, resume prefills, and short decodes under multi-agent contention [2603.10342]. That usage differs from the TinyServe system proper, which is centered on query-aware sparse KV selection during autoregressive decoding [2509.12211]. The overlap is conceptual rather than literal: both are concerned with efficient serving under constrained hardware, but they optimize different bottlenecks. TinyServe addresses decode-time KV access; AgentServe addresses prefill–decode contention and latency stability for agentic workloads [2509.12211] [2603.10342].

Future directions named for TinyServe include extending the mechanism to larger models and more production systems, exploring dynamic sparsity strategies such as adaptive $K/P$ ratios, integrating with frameworks such as TensorRT-LLM and DeepSpeed, and further developing training-time acceleration mechanisms including gradient-aware KV retention and backprop optimization [2509.12211]. Taken together, these directions suggest that TinyServe is not only a serving runtime but also an experimental substrate for studying how query-conditioned cache access can reshape the systems design space of LLM inference.

Source: https://www.emergentmind.com/topics/tinyserve