TinyServe: Lightweight LLM Serving

• TinyServe is a lightweight serving system for tiny LLMs that optimizes autoregressive decoding by leveraging query-aware sparse KV cache selection.
• It reorganizes the KV cache into fixed-size pages and employs a modular scheduling pipeline to reduce memory bandwidth pressure and decrease latency.
• Its fused CUDA kernels and detailed instrumentation enable efficient sparse attention execution and runtime analysis on resource-constrained hardware.

Searching arXiv for the cited TinyServe and related serving papers to ground the article and attach arXiv IDs. TinyServe is a lightweight and extensible serving system for deploying tiny LLMs such as TinyLLaMA and GPT-2 variants, with an emphasis on reducing the memory and latency overhead of key–value (KV) cache access during autoregressive decoding through structured KV sparsity, query-aware cache selection, and hardware-efficient attention kernels (Liu et al., 28 Aug 2025). In the system presented under that name, the central idea is to reorganize the decode-time KV path around page-level selection driven by the current query vector, rather than assuming dense attention over all stored KV. This places TinyServe at the intersection of LLM systems, sparse attention, and runtime cache management for resource-constrained hardware. The term “TinyServe” also appears informally in a distinct sense in later work on single-GPU agent serving, where it denotes a small-footprint serving stack for agentic AI; that usage corresponds to a different system design problem and is exemplified by AgentServe rather than by the query-aware sparse serving framework introduced as TinyServe (Zhang et al., 11 Mar 2026).

1. Definition and problem setting

TinyServe addresses the decode-phase bottleneck in transformer inference. In the formulation given for transformer-based LLMs, inference has a prefill phase, in which the entire prompt is processed and the resulting KV vectors are stored, and a decode phase, in which each new query vector $q_t \in \mathbb{R}^d$ attends over all previous keys and values in the cache (Liu et al., 28 Aug 2025). The decode attention is written as

$$\text{Attn}(q_t, K, V) = \sum_{i=1}^{t-1} \text{softmax}(q_t^\top k_i) \cdot v_i .$$

The stated motivation is that decode latency dominates because each step repeatedly scans a growing KV cache, and long contexts of 4K–32K tokens make decode time and KV access the main bottlenecks even for tiny models with 125M–350M parameters (Liu et al., 28 Aug 2025). The system therefore targets three coupled issues: memory bandwidth pressure from repeatedly reading large KV state from HBM, latency from the $O(t)$ growth of dense attention per step, and KV cache size pressure under long-context or multi-user serving (Liu et al., 28 Aug 2025).

This framing distinguishes TinyServe from serving stacks that optimize memory layout and batching but still assume dense attention over all stored KV. The paper explicitly positions prior systems such as vLLM, TGI, FasterTransformer, and TensorRT-LLM in that category, and contrasts TinyServe with fixed-window or heuristic pruning approaches that are not query-dependent (Liu et al., 28 Aug 2025). A plausible implication is that TinyServe is best understood not merely as a serving runtime for small models, but as a systems research platform in which cache access itself becomes a first-class optimization target.

2. Architectural organization

TinyServe is presented as a lightweight serving system designed around tiny LLMs including TinyLLaMA-125M, GPT2-345M, OPT-350M, GPT2-774M, and up to LLaMA-1.3B (Liu et al., 28 Aug 2025). Its architecture is organized around three core components: a Query-Aware KV Retriever, a Modular Scheduling Pipeline, and a Sparse Attention Executor (Liu et al., 28 Aug 2025).

The Query-Aware KV Retriever dynamically selects relevant KV “pages” at decode time based on the current query and per-page metadata, thereby implementing structured KV sparsity (Liu et al., 28 Aug 2025). The Modular Scheduling Pipeline exposes a dispatch loop through which incoming requests can traverse configurable plugins, including entropy-based early exit, pruning policies, approximate attention, and custom token selection strategies, without changing the model architecture (Liu et al., 28 Aug 2025). The Sparse Attention Executor provides fused CUDA kernels that implement page scoring using metadata, sparse KV gather, and masked attention, and supports FP16 / INT8 formats together with multi-GPU dispatch (Liu et al., 28 Aug 2025).

The per-step request flow is specified at a fairly fine granularity. A request manager collects active sessions and batches them subject to timeout and capacity constraints; the model forward computes the new query $q_t$ for each sequence; the TinyServe KV path performs query-aware page scoring, top-$K$ page selection, sparse gather from HBM, and attention only over gathered KV; token selection is handled through a plugin interface operating on logits; and the KV cache manager organizes the cache into fixed-size pages, updates per-page metadata, and handles cache reuse across steps and sessions (Liu et al., 28 Aug 2025). The system also includes fine-grained instrumentation for latency breakdown, page reuse, memory bandwidth utilization, and session behavior (Liu et al., 28 Aug 2025).

This architecture is notable because it treats token selection policy and cache selection policy as orthogonal modules. That separation allows experimentation with sampling, early exit, and sparsity mechanisms in one framework, while preserving the claim that no model modifications are required (Liu et al., 28 Aug 2025).

3. Structured KV sparsity and page-based memory layout

TinyServe partitions the KV cache for an attention layer into fixed-size pages. If $K \in \mathbb{R}^{L \times d}$ and $V \in \mathbb{R}^{L \times d_v}$, with $L$ the number of tokens in context, then the sequence is split as

$$K = \bigcup_{j=1}^{P} \mathcal{K}_j, \quad P = \left\lceil \frac{L}{S} \right\rceil ,$$

where page $j$ contains a fixed-size block of keys and corresponding values (Liu et al., 28 Aug 2025). Each page is stored as a contiguous memory block in HBM, which the paper describes as aligned with paged attention but repurposed for sparse access rather than merely efficient dense access (Liu et al., 28 Aug 2025).

The key departure from dense attention is that TinyServe computes attention only over a subset of pages $$\text{Attn}(q_t, K, V) = \sum_{i=1}^{t-1} \text{softmax}(q_t^\top k_i) \cdot v_i .$$0 selected for the current query. The mechanism is defined in three steps: compute a relevance score $$\text{Attn}(q_t, K, V) = \sum_{i=1}^{t-1} \text{softmax}(q_t^\top k_i) \cdot v_i .$$1 for each page using page metadata, select top-$$\text{Attn}(q_t, K, V) = \sum_{i=1}^{t-1} \text{softmax}(q_t^\top k_i) \cdot v_i .$$2 pages, and perform attention only over keys and values in those selected pages (Liu et al., 28 Aug 2025). The resulting sparse attention is expressed as

$$\text{Attn}(q_t, K, V) = \sum_{i=1}^{t-1} \text{softmax}(q_t^\top k_i) \cdot v_i .$$3

The paper characterizes this as structured sparsity at the page level: the kept units are tokens in top-$$\text{Attn}(q_t, K, V) = \sum_{i=1}^{t-1} \text{softmax}(q_t^\top k_i) \cdot v_i .$$4 pages and the dropped units are tokens in pages with low relevance scores (Liu et al., 28 Aug 2025). The structure matters because it makes memory access page-aligned and therefore suitable for fused kernels, unlike unstructured token-by-token pruning.

TinyServe also provides an approximation-error perspective. It defines

$$\text{Attn}(q_t, K, V) = \sum_{i=1}^{t-1} \text{softmax}(q_t^\top k_i) \cdot v_i .$$5

and states that under assumptions on key distributions with variance $$\text{Attn}(q_t, K, V) = \sum_{i=1}^{t-1} \text{softmax}(q_t^\top k_i) \cdot v_i .$$6 and page size $$\text{Attn}(q_t, K, V) = \sum_{i=1}^{t-1} \text{softmax}(q_t^\top k_i) \cdot v_i .$$7,

$$\text{Attn}(q_t, K, V) = \sum_{i=1}^{t-1} \text{softmax}(q_t^\top k_i) \cdot v_i .$$8

From this, the paper concludes that larger page size $$\text{Attn}(q_t, K, V) = \sum_{i=1}^{t-1} \text{softmax}(q_t^\top k_i) \cdot v_i .$$9 can increase approximation error, while smaller pages improve fidelity at the cost of more metadata and scoring (Liu et al., 28 Aug 2025). Empirically, it chooses $O(t)$0 with selection ratio $O(t)$1 to keep accuracy drops negligible while achieving large memory savings (Liu et al., 28 Aug 2025). This suggests a systems trade-off between metadata overhead and retrieval fidelity that is intrinsic to the page abstraction itself.

4. Query-aware page selection

The defining mechanism in TinyServe is query-aware page selection based on bounding-box metadata (Liu et al., 28 Aug 2025). For each page $O(t)$2, the system stores channel-wise minima and maxima of key components,

$O(t)$3

with $O(t)$4 and $O(t)$5 (Liu et al., 28 Aug 2025). This metadata is described as a bounding box in $O(t)$6 that cheaply summarizes all keys in the page, is stored in faster memory such as SRAM / L2, and is updated when new tokens are appended (Liu et al., 28 Aug 2025).

The scoring function is a directional bounding-box estimator:

$O(t)$7

(Liu et al., 28 Aug 2025). The interpretation given is dimensionwise: when $O(t)$8 is positive, the largest possible contribution from that coordinate comes from the page maximum; when $O(t)$9 is negative, it comes from the page minimum (Liu et al., 28 Aug 2025). Summing across coordinates yields an upper bound or approximation to the maximum dot product within the page, which functions as a cheap proxy for the strongest attention logit in that page (Liu et al., 28 Aug 2025).

Page selection then proceeds by computing all page scores in parallel on GPU, applying top-$q_t$0 selection, and gathering the selected pages from HBM (Liu et al., 28 Aug 2025). The paper notes that threshold-based selection could also be used, but focuses on top-$q_t$1 with configurable ratio $q_t$2 (Liu et al., 28 Aug 2025). Because the selected subset is query-dependent, different decode steps may access different cache regions even within the same sequence.

The paper further gives a simple latency model. If each page fetch from HBM costs $q_t$3, metadata is in L2/SRAM with negligible $q_t$4 cost, and page scoring is fused into one kernel, then decode-step latency is decomposed as

$q_t$5

The intended significance is clear: when $q_t$6 is substantially below 1, both KV loading and attention computation contract relative to full-cache attention (Liu et al., 28 Aug 2025).

5. Fused kernel design, token selection, and instrumentation

TinyServe’s kernel-level design fuses four stages into a single CUDA kernel: page scoring from metadata, top-$q_t$7 selection, sparse KV gather, and attention computation (Liu et al., 28 Aug 2025). The summarized pseudocode in the paper shows a single pass from query $q_t$8 and page metadata $q_t$9 to scores $K$0, selected pages $K$1, fetched sparse KV tensors, attention logits $K$2, and final output $K$3 (Liu et al., 28 Aug 2025). This fusion is described as avoiding multiple kernel launches and intermediate buffers, reducing CPU–GPU synchronization, and avoiding repeated memory traversal (Liu et al., 28 Aug 2025).

Several hardware-oriented choices are emphasized. Keys and values within a page are contiguous, enabling coalesced HBM loads; metadata is stored in L2/shared memory; top-$K$4 can be implemented with a shared-memory heap or block-level parallel reduction; and page size $K$5 is chosen to align with warp sizes or tensor-core-friendly matrix sizes (Liu et al., 28 Aug 2025). The overall design goal is to reduce the bandwidth bottleneck by lowering the number of KV tokens processed per step (Liu et al., 28 Aug 2025). The paper reports that memory bandwidth usage stays well below the HBM limit with smoother access patterns relative to dense baselines (Liu et al., 28 Aug 2025).

TinyServe also includes a plugin interface for token selection at each decode step. Plugins can implement greedy decoding, top-$K$6 sampling, nucleus sampling, entropy-based early exit, and custom policies (Liu et al., 28 Aug 2025). After sparse attention and final-layer computation, logits are passed to the plugin, the selected token is appended, and the corresponding new KV is added to the cache while metadata is updated (Liu et al., 28 Aug 2025). This modularity is important because it lets the system vary serving policies and cache policies independently.

Instrumentation is an explicit part of the platform. TinyServe logs latency breakdowns across prefill and decode, and within decode across page scoring, top-$K$7 selection, KV gather, and attention; it tracks KV hit rate per page, page reuse across decode steps, token eviction and page migration; and it records SM utilization, memory bandwidth usage over time, cross-request cache reuse, and session migration overhead (Liu et al., 28 Aug 2025). The overhead of instrumentation is reported as less than 2% for training-related hooks (Liu et al., 28 Aug 2025). This instrumentation focus reinforces the system’s role as a research framework rather than solely a production runtime.

6. Empirical performance and scaling behavior

The evaluation covers TinyLLaMA-125M, GPT2-345M, OPT-350M, GPT2-774M, and LLaMA-1.3B, with tasks spanning language modeling, long-range benchmarks, reasoning benchmarks, and multi-user serving workloads (Liu et al., 28 Aug 2025). Baselines include vLLM, TGI, TensorRT-LLM, StreamingLLM, SnapKV, PyramidKV, and pruning baselines such as FullCache, SoftPrune, and EntropyStop (Liu et al., 28 Aug 2025). The hardware configuration reported is 8 × NVIDIA A100 80GB GPUs with FP16 inference, CUDA 11.8, cuDNN 8.7, PyTorch 2.0.1, and Ubuntu 20.04 (Liu et al., 28 Aug 2025).

The headline result is that TinyServe achieves up to 3.4x speedup and over 2x memory savings with negligible accuracy drop (Liu et al., 28 Aug 2025). The detailed examples include the following:

Model / context	Latency	Memory
TinyLLaMA-125M, 4K	25.1 ms → 11.9 ms	2.1 GB → 1.2 GB
GPT2-345M, 8K	45.2 ms → 20.1 ms	4.8 GB → 2.1 GB
GPT2-774M, 16K	89.2 ms → 41.2 ms	8.9 GB → 4.1 GB
LLaMA-1.3B, 32K	156.8 ms → 72.8 ms	15.8 GB → 7.2 GB

These figures are all reported relative to FullCache, with approximate speedups of roughly 2.1x–2.25x in the tabulated settings and larger gains in longer-context or more aggressive sparsity configurations (Liu et al., 28 Aug 2025).

The paper further states that TinyServe often improves accuracy slightly relative to FullCache on LongBench examples, including 54.2 → 55.2% for TinyLLaMA-125M, 61.7 → 62.8% for GPT2-345M, and 68.9 → 70.2% for LLaMA-1.3B (Liu et al., 28 Aug 2025). Since the general framing emphasizes “negligible accuracy drop,” these increases are treated as empirical observations rather than as a guaranteed effect (Liu et al., 28 Aug 2025). A plausible implication is that query-aware sparsity may sometimes suppress distracting or low-value context regions, but the paper does not generalize that into a universal claim.

Cache reuse is also reported as strong. TinyServe typically achieves greater than 95% KV hit rate in experiments, with examples of 96.2% for TinyLLaMA-125M and 95.4% for GPT2-345M (Liu et al., 28 Aug 2025). In multi-user serving for GPT2-345M with 1024 concurrent requests, a TinyServe-modified vLLM configuration is reported to reduce P50 latency from 45.2 ms to 32.1 ms and increase throughput from 18.4 req/s to 28.6 req/s, while reaching 91.2% GPU utilization (Liu et al., 28 Aug 2025).

Scaling results for GPT2-345M at sequence length 16K and batch size 128 are near-linear: 0.81 tokens/ms on 1 GPU, 1.58 tokens/ms on 2 GPUs, 3.123 tokens/ms on 4 GPUs, and 6.221 tokens/ms on 8 GPUs, corresponding to efficiencies from 98.0% to 96.0% relative to baseline (Liu et al., 28 Aug 2025). The paper interprets this as evidence that fused sparse attention and query-aware page selection impose minimal extra communication overhead under standard tensor/model parallel coordination (Liu et al., 28 Aug 2025).

7. Relation to prior systems, scope, and limitations

TinyServe is explicitly compared to both production serving systems and sparsity-oriented research methods (Liu et al., 28 Aug 2025). Relative to vLLM, whose PagedAttention organizes KV cache into pages for efficient dense attention, TinyServe turns pages into query-ranked units by integrating query-aware page selection into the PagedAttention kernel (Liu et al., 28 Aug 2025). Relative to TGI, TensorRT-LLM, and FasterTransformer, the distinguishing claim is that those systems focus on kernels, quantization, and batching but do not explicitly implement query-conditioned KV sparsity (Liu et al., 28 Aug 2025). Relative to StreamingLLM, SnapKV, and PyramidKV, TinyServe is positioned as query-aware, metadata-based, and retraining-free, rather than relying on fixed windows, clustering structures, or compression (Liu et al., 28 Aug 2025).

The system’s scope is also sharply defined. It is designed primarily for tiny to mid-scale models from 125M to 1.3B parameters, and although it is integrated into vLLM, the paper states that direct evidence for 7B+ models is not presented (Liu et al., 28 Aug 2025). Its scoring mechanism assumes that the maximum dot product per page can be approximated well by min/max bounding-box metadata; the paper notes that approximation may degrade if attention patterns are highly non-linear or keys within a page are very diverse, that is, when $K$8 is high (Liu et al., 28 Aug 2025). It also fixes page size $K$9, observes the trade-off between large and small $K \in \mathbb{R}^{L \times d}$0, and does not explore dynamic or adaptive page sizing (Liu et al., 28 Aug 2025). Finally, the fused kernel is tuned for NVIDIA GPUs and CUDA, so portability to TPUs or custom ASICs would require reimplementation (Liu et al., 28 Aug 2025).

A separate source of ambiguity concerns the name itself. In AgentServe, “TinyServe” is used descriptively to mean “a very efficient agent-serving stack on one consumer GPU,” and the discussion there centers on phase-aware scheduling for cold prefills, resume prefills, and short decodes under multi-agent contention (Zhang et al., 11 Mar 2026). That usage differs from the TinyServe system proper, which is centered on query-aware sparse KV selection during autoregressive decoding (Liu et al., 28 Aug 2025). The overlap is conceptual rather than literal: both are concerned with efficient serving under constrained hardware, but they optimize different bottlenecks. TinyServe addresses decode-time KV access; AgentServe addresses prefill–decode contention and latency stability for agentic workloads (Liu et al., 28 Aug 2025, Zhang et al., 11 Mar 2026).

Future directions named for TinyServe include extending the mechanism to larger models and more production systems, exploring dynamic sparsity strategies such as adaptive $K \in \mathbb{R}^{L \times d}$1 ratios, integrating with frameworks such as TensorRT-LLM and DeepSpeed, and further developing training-time acceleration mechanisms including gradient-aware KV retention and backprop optimization (Liu et al., 28 Aug 2025). Taken together, these directions suggest that TinyServe is not only a serving runtime but also an experimental substrate for studying how query-conditioned cache access can reshape the systems design space of LLM inference.