FineServe: Mixed-Precision LLM Serving

• FineServe is a precision-aware inference serving system for quantized LLMs that jointly optimizes memory layout and scheduling to maximize throughput under strict SLOs.
• It employs the KV Slab mechanism to partition shared GPU memory into uniform slabs, reducing both internal and external fragmentation in mixed-precision settings.
• The framework integrates a global MME-based placement policy with TTFT-aware adaptive batching, achieving up to 2.2× higher SLO attainment and improved token throughput.

Searching arXiv for FineServe and closely related serving systems to ground the article in current papers. I’ll look up the FineServe paper and a few directly related LLM serving systems on arXiv. FineServe is an inference serving framework for mixed-precision, or heterogeneous-precision, LLM deployment that targets a specific systems problem: quantization changes not only arithmetic cost but also the memory geometry of serving, especially in the KV cache. The framework addresses the resulting coupling between memory fragmentation, GPU sharing, and latency-constrained scheduling through two main mechanisms: KV Slab, a precision-aware adaptive KV-cache manager, and a two-level scheduler composed of a global placement policy and a local adaptive batching policy. Its stated objective is to maximize total throughput subject to TTFT SLOs and GPU memory budgets, and its evaluation reports up to $2.2\times$ higher SLO attainment and $1.8\times$ higher token generation throughput than prior GPU-sharing baselines under the reported settings (Bin et al., 8 Sep 2025).

1. Problem setting and design premise

FineServe is motivated by the rise of Post-Training Quantization (PTQ) techniques and the consequent demand for serving quantized LLMs. In the paper’s framing, quantized models improve throughput and reduce memory usage with minimal accuracy loss, but they also make shared serving more difficult because KV-cache block geometry becomes precision-dependent. The central claim is that heterogeneous-precision serving cannot be treated as a purely scheduling problem or a purely memory-management problem; both must be co-designed (Bin et al., 8 Sep 2025).

The paper defines the KV block size as

$$\mathrm{KV\ block\ size} = \mathrm{TokensPerBlock} \times \mathrm{TokenSize} + \mathrm{Quant.Params}.$$

It further defines token size as

$$\mathrm{TokenSize} = \frac{\#\mathrm{Heads}}{\mathrm{TP\ degree}} \times \mathrm{HeadDim} \times 2 \times \mathrm{bytes(precision)}.$$

This formulation makes explicit why FP16, FP8, and lower-precision variants such as INT4 or QoQ create different block layouts: the effective bytes per token vary with KV precision and tensor-parallel degree, and some schemes also add per-block quantization parameters.

From this, the paper identifies three failure modes in naive mixed-precision GPU sharing. If each model uses its own block size in a shared pool, free regions become difficult to reuse across models, producing external fragmentation. If all models are forced to share a universal block size, smaller-token-size models waste space inside blocks, producing internal fragmentation. If the system relies on dynamic CUDA VMM map/unmap operations to maintain contiguous logical views, the per-call overhead—described as tens of microseconds—can accumulate into milliseconds and harm TTFT, TPOT, and SLO attainment. The design premise of FineServe is therefore that precision-aware serving requires both a new memory abstraction and a new placement policy.

A further scheduling premise is that quantized models have different marginal benefit from additional KV memory. The paper formalizes this by Marginal Memory Efficiency (MME):

$$\mu_M(\mathcal{G}) = \frac{\Delta \mathrm{generated\ tokens\ of\ model\ } M\ \mathrm{on}\ \mathcal{G}}{\Delta \mathrm{KV\ memory}}.$$

This quantity measures the additional generated tokens produced per additional unit of KV memory for model $M$ on GPU group $\mathcal{G}$. In FineServe’s formulation, MME is the key signal for deciding where residual KV capacity should be assigned.

2. KV Slab: precision-aware adaptive KV-cache management

KV Slab is FineServe’s precision-aware memory manager. It is inspired by the OS slab allocator, but the paper adapts that concept to a shared GPU KV-cache setting in which co-located models may use different precisions and therefore different KV block sizes. The system pre-allocates one large shared KV tensor per GPU and partitions it into fixed-size physical chunks called slabs. Each slab is then formatted according to the KV block size of the model using it, rather than enforcing a single universal block format for all models on the device (Bin et al., 8 Sep 2025).

This design yields the two fragmentation properties that motivate the framework. Because slabs are uniform physical units, the underlying pool avoids the irregular holes associated with cross-model block-size heterogeneity, which the paper characterizes as preventing external fragmentation. Because each model still receives a block size matched to its own quantization characteristics, the system also avoids the internal-fragmentation penalty of a one-size-fits-all block policy. The paper explicitly presents this as the key tradeoff: physical uniformity at the slab level, logical heterogeneity at the block-format level.

The manager maintains a KV slab table whose states include FREE, PARTIAL, and FULL, as well as a mapping from block identifiers to slabs. The block numbering scheme is given as

$$\mathrm{Blk\ ID} = (\mathrm{slab\ ID}) \times (\#\mathrm{of\ blks\ per\ slab}) + (\mathrm{local\ blk\ ID}).$$

Allocation proceeds by computing the requesting model’s KV block size, using that size as a key into the slab pool, and returning a $(\mathrm{slab\ ID}, \mathrm{block\ ID})$ pair. On release, the relevant entry is cleared, and when a slab becomes empty, its memory returns to the shared pool.

The slab size is described as a multiple of the least common multiple of the co-located models’ KV block sizes. This alignment preserves each model’s logical block table and attention kernel behavior without requiring kernel modification. The paper presents that detail as important because it lets FineServe integrate with existing serving engines while still making block allocation precision-aware.

A separate contribution of KV Slab is the elimination of runtime VMM map/unmap overhead from the critical path. Rather than repeatedly remapping physical memory at the driver level, FineServe uses a pre-allocated shared tensor and view-based remapping inside the engine. The paper states that, in its measurements, KV Slab consistently yields lower TTFT and TPOT than a CUDA VMM-based baseline using kvcached-style memory management, and that the gap widens as request rate increases. A plausible implication is that, in mixed-precision serving, allocation overhead is not a negligible bookkeeping cost but part of the latency path.

3. Global placement via marginal memory efficiency

The global scheduler is FineServe’s model-to-GPU placement mechanism. It uses profiling information about each model’s latency behavior and base memory footprint, then assigns models to candidate GPU groups by estimating how much useful throughput can be extracted from the remaining contested KV memory on that group. The paper describes the overall objective as maximizing total throughput subject to TTFT SLOs and GPU memory budgets (Bin et al., 8 Sep 2025).

A central abstraction is the decomposition of GPU memory into fixed memory and contested memory. For a model $M$, the base footprint $1.8\times$0 includes weights, profiled average activation memory, and profiled average KV-cache usage. This base footprint is measured at the smallest batch size that meets the expected request rate; if that batch size cannot satisfy TTFT SLO, the system increases data-parallel replicas instead. After charging each placed model’s fixed footprint, the remaining memory on the GPU becomes contested memory that can be distributed as additional KV cache.

For candidate GPU group $1.8\times$1, the rigorous score is defined as

$1.8\times$2

where $1.8\times$3 is the candidate group after adding model $1.8\times$4, and $1.8\times$5 is the remaining contested memory. Because directly optimizing the allocation variables $1.8\times$6 is hard, the paper uses the approximation

$1.8\times$7

The placement algorithm is greedy. It sorts models in descending order of $1.8\times$8, enumerates candidate GPUs or TP groups for each model, computes the score for each candidate, places the model on the GPU with the maximum score, and updates both the resident set and the remaining contested memory. The paper states that this differs from coarse request-rate heuristics because residual KV capacity is directed toward models with higher MME rather than partitioned evenly.

The significance of this design is architectural. In FineServe’s formulation, the scarce resource in mixed-precision serving is not merely device count or nominal HBM size, but the portion of memory that can still be turned into useful decode capacity after fixed footprints are paid. This suggests a shift from placement policies centered on static memory fit toward policies centered on throughput per extra KV byte.

4. Local scheduling and TTFT-aware adaptive batching

FineServe’s local scheduler operates per node and per model, with the goal of choosing the largest batch that can be served without violating the TTFT SLO. The paper explicitly uses TTFT rather than full end-to-end latency as the local control signal, because TTFT is predictable at dispatch time whereas TPOT depends on future output length. The local scheduler is therefore framed as a latency-constrained batching policy rather than a generic throughput maximizer (Bin et al., 8 Sep 2025).

The scheduler tracks waiting requests $1.8\times$9, currently running requests $$\mathrm{KV\ block\ size} = \mathrm{TokensPerBlock} \times \mathrm{TokenSize} + \mathrm{Quant.Params}.$$0, current time $$\mathrm{KV\ block\ size} = \mathrm{TokensPerBlock} \times \mathrm{TokenSize} + \mathrm{Quant.Params}.$$1, per-model caps $$\mathrm{KV\ block\ size} = \mathrm{TokensPerBlock} \times \mathrm{TokenSize} + \mathrm{Quant.Params}.$$2 and $$\mathrm{KV\ block\ size} = \mathrm{TokensPerBlock} \times \mathrm{TokenSize} + \mathrm{Quant.Params}.$$3 on concurrent requests and batched prompt tokens, and a chunked-prefill latency model $$\mathrm{KV\ block\ size} = \mathrm{TokensPerBlock} \times \mathrm{TokenSize} + \mathrm{Quant.Params}.$$4. For each waiting request $$\mathrm{KV\ block\ size} = \mathrm{TokensPerBlock} \times \mathrm{TokenSize} + \mathrm{Quant.Params}.$$5, it computes the deadline

$$\mathrm{KV\ block\ size} = \mathrm{TokensPerBlock} \times \mathrm{TokenSize} + \mathrm{Quant.Params}.$$6

Requests that cannot meet their deadline even if served immediately are dropped. The remaining requests are then sorted by ascending deadline.

The batching logic is EDF-based. The earliest-deadline request becomes the anchor request $$\mathrm{KV\ block\ size} = \mathrm{TokensPerBlock} \times \mathrm{TokenSize} + \mathrm{Quant.Params}.$$7, and additional requests are inserted only while the predicted prefill finish time remains within the anchor deadline:

$$\mathrm{KV\ block\ size} = \mathrm{TokensPerBlock} \times \mathrm{TokenSize} + \mathrm{Quant.Params}.$$8

The scheduler simultaneously enforces the resource caps $$\mathrm{KV\ block\ size} = \mathrm{TokensPerBlock} \times \mathrm{TokenSize} + \mathrm{Quant.Params}.$$9 and $$\mathrm{TokenSize} = \frac{\#\mathrm{Heads}}{\mathrm{TP\ degree}} \times \mathrm{HeadDim} \times 2 \times \mathrm{bytes(precision)}.$$0. If no request can be safely added, the scheduler removes the longest-running request from the current schedule to create slack, and then rebuilds the tentative batch.

The paper describes this algorithm as a modification of the Moore-Hodgson algorithm for the setting of chunked prefill. The relevance of that characterization is that standard single-job sequencing intuition is insufficient once batch execution time grows with batch composition. FineServe’s adaptation preserves the EDF principle but makes batch growth itself part of the deadline-feasibility test.

The paper’s interpretation of the local scheduler is not that it dramatically dominates other deadline-aware policies in isolation, but that it is consistently better aligned with chunked-prefill serving. In the local scheduler ablation, with a single model and static KV management, FineServe achieves up to 33.57% higher SLO attainment than FCFS and up to 1.63% higher than Prism$$\mathrm{TokenSize} = \frac{\#\mathrm{Heads}}{\mathrm{TP\ degree}} \times \mathrm{HeadDim} \times 2 \times \mathrm{bytes(precision)}.$$1. The paper attributes the margin over Prism to chunked-prefill-aware batching: under looser SLOs and higher request rates, Prism tends to form large batches that increase the risk of missing the anchor request’s deadline.

5. Experimental evaluation and reported performance

FineServe is implemented on top of vLLM and evaluated on two hardware settings: a single multi-GPU server with 4× NVIDIA H100 80GB SXM5, and a larger setup using 2 AWS p5.48xlarge instances with 8 H100 GPUs each, for a total of 16 GPUs. Workloads are derived from ShareGPT, and request arrival times are generated with a Poisson process. The evaluated models include Llama 3.1-8B in FP16, FP8 (W8A8KV8), and AWQ (W4A16), Llama 3-8B in QoQ (W4A8KV4), Llama 3.1-70B in FP16, FP8, and AWQ, and Qwen3-32B in FP16 and FP8 (Bin et al., 8 Sep 2025).

The baseline systems are Static$$\mathrm{TokenSize} = \frac{\#\mathrm{Heads}}{\mathrm{TP\ degree}} \times \mathrm{HeadDim} \times 2 \times \mathrm{bytes(precision)}.$$2 and Prism$$\mathrm{TokenSize} = \frac{\#\mathrm{Heads}}{\mathrm{TP\ degree}} \times \mathrm{HeadDim} \times 2 \times \mathrm{bytes(precision)}.$$3. Static$$\mathrm{TokenSize} = \frac{\#\mathrm{Heads}}{\mathrm{TP\ degree}} \times \mathrm{HeadDim} \times 2 \times \mathrm{bytes(precision)}.$$4 uses static memory partitioning per model, the same placement policy as FineServe, but FCFS batching and no deadline guarantees. Prism$$\mathrm{TokenSize} = \frac{\#\mathrm{Heads}}{\mathrm{TP\ degree}} \times \mathrm{HeadDim} \times 2 \times \mathrm{bytes(precision)}.$$5 uses Prism’s placement and deadline-based scheduling, but with KV Slab replacing Prism’s CUDA VMM-based kvcached in order to avoid confounding the comparison with VMM overhead. The main metrics are TTFT SLO attainment, throughput (req/s), and token generation throughput (tokens/s).

At high load, in the reported representative setting of SLO scale 20 with request scale 8, FineServe achieves 2.2× higher SLO attainment than Prism$$\mathrm{TokenSize} = \frac{\#\mathrm{Heads}}{\mathrm{TP\ degree}} \times \mathrm{HeadDim} \times 2 \times \mathrm{bytes(precision)}.$$6, 1.2× higher SLO attainment than Static$$\mathrm{TokenSize} = \frac{\#\mathrm{Heads}}{\mathrm{TP\ degree}} \times \mathrm{HeadDim} \times 2 \times \mathrm{bytes(precision)}.$$7, 1.8× higher token generation throughput than Prism$$\mathrm{TokenSize} = \frac{\#\mathrm{Heads}}{\mathrm{TP\ degree}} \times \mathrm{HeadDim} \times 2 \times \mathrm{bytes(precision)}.$$8, and 1.5× higher token generation throughput than Static$$\mathrm{TokenSize} = \frac{\#\mathrm{Heads}}{\mathrm{TP\ degree}} \times \mathrm{HeadDim} \times 2 \times \mathrm{bytes(precision)}.$$9. Token throughput reaches about 2.3K tokens/s at 8× request scale, whereas Prism saturates earlier and Static scales more slowly. On the 8–16 GPU multi-node setup, FineServe maintains stable SLO attainment, with a minimum reported value of 96.9%; Static$$\mu_M(\mathcal{G}) = \frac{\Delta \mathrm{generated\ tokens\ of\ model\ } M\ \mathrm{on}\ \mathcal{G}}{\Delta \mathrm{KV\ memory}}.$$0 starts around 80.1% and improves as more GPUs are added, but FineServe remains clearly better than Prism$$\mu_M(\mathcal{G}) = \frac{\Delta \mathrm{generated\ tokens\ of\ model\ } M\ \mathrm{on}\ \mathcal{G}}{\Delta \mathrm{KV\ memory}}.$$1.

The ablation studies separate the effects of global placement and local scheduling. When isolating the global scheduler with static KV partitioning and FCFS local scheduling, FineServe achieves up to 3× higher aggregated throughput than Prism$$\mu_M(\mathcal{G}) = \frac{\Delta \mathrm{generated\ tokens\ of\ model\ } M\ \mathrm{on}\ \mathcal{G}}{\Delta \mathrm{KV\ memory}}.$$2 at 8× request rate. The paper gives a placement example in which FineServe co-locates an FP8 model with an FP16 model on one GPU, whereas Prism places an AWQ model with the FP8 model in a less efficient arrangement. The reported interpretation is that FineServe remains stable under high load because it places large-footprint models early and scores candidates with MME-aware placement.

Taken together, the reported results support the paper’s claim that FineServe’s gains come from two interacting sources: precision-aware placement, which allocates residual KV memory to models with higher MME, and SLO-aware adaptive batching, which exploits chunked prefill without letting the earliest deadline slip.

6. Scope, limitations, and relation to adjacent serving systems

The scope of FineServe is mixed-precision LLM sharing under TTFT-oriented service objectives. The paper does not frame limitations as a dedicated section, but several are explicit or implicit. FineServe uses TTFT SLO rather than full end-to-end latency because TPOT is difficult to predict before completion. The current design assumes per-model SLOs and similar request patterns within a model. Its memory-management benefit depends on quantized models’ block structures being compatible with slab formatting. The evaluation is strongest for the reported set of quantization variants and model families, especially scenarios where co-located models differ substantially in KV-cache behavior (Bin et al., 8 Sep 2025).

A common simplification in LLM serving is to treat quantization as primarily a compute optimization. FineServe explicitly rejects that view by arguing that quantization also changes memory layout, fragmentation behavior, and the throughput gain associated with an extra unit of KV capacity. Another simplification is to assume that mixed-model sharing can be handled by static partitioning plus deadline-aware batching; the FineServe results suggest that placement quality depends materially on precision-aware memory efficiency, not just on request rate or footprint alone.

Within the broader serving literature, FineServe occupies a distinct niche. BestServe addresses early-stage deployment planning by ranking collocated and disaggregated strategies according to estimated goodput under SLO constraints, using an adapted roofline estimator, simulator, and optimizer, but it is a strategy-selection framework rather than a heterogeneous-precision memory manager (Hu et al., 6 Jun 2025). EcoServe addresses a different axis—cost-effective serving on commodity interconnects—through partially disaggregated serving, temporal disaggregation, rolling activation, and macro-instance scheduling, rather than mixed-precision GPU sharing (Du et al., 25 Apr 2025). Harli focuses on co-locating PEFT finetuning with decode instances through a unified memory allocator, latency prediction, and QoS-guaranteed SM partitioning, which is again a different resource-sharing problem from FineServe’s precision-aware KV layout and TTFT-aware batching (Xu et al., 13 Nov 2025). FairServe targets multi-tenant fairness through overload- and interaction-driven throttling and weighted service-counter scheduling, rather than memory geometry or quantization-aware placement (Khan et al., 2024).

The paper’s broader implication is that heterogeneous-precision serving should be analyzed as a joint problem of memory layout, placement, and latency-aware runtime control. In that sense, FineServe extends the LLM serving literature by making KV-cache geometry itself a first-class systems abstraction.