PrefillShare: Shared Prefill for Multi-LLM Serving

• PrefillShare is a shared prefill module that factorizes LLMs into a frozen prefill part and task-specific decode modules to eliminate redundant computation across models.
• It uses cache-conditioned fine-tuning by freezing the main prefill module and fine-tuning only the decode module, ensuring near-full fine-tuning accuracy.
• Empirical results show up to 4.5x lower latency and 3.9x higher throughput in multi-model agent workloads, making it a significant advance in efficient LLM serving.

PrefillShare is a shared prefill module for KV reuse in multi-LLM disaggregated serving. It targets multi-agent systems that orchestrate multiple specialized LLMs over a shared context, a setting in which the same prompt prefix is repeatedly processed across models, causing redundant prefill computation and duplicated key-value cache state. PrefillShare addresses this by factorizing the model into a shared prefill module and task-specific decode modules, freezing the prefill module, fine-tuning only the decode module, and enabling multiple models to share both the prefill computation and the KV cache generated for the same prompt. In the reported evaluation, this design matches full fine-tuning accuracy on a broad range of tasks and models while delivering 4.5x lower p95 latency and 3.9x higher throughput in multi-model agent workloads (Woo et al., 12 Feb 2026).

1. Computational setting and motivation

The motivating workload is multi-LLM orchestration in which multiple specialized LLMs are invoked sequentially or in parallel over a shared session context or prompt. In this setting, each model redundantly executes the prefill stage and maintains an independent KV cache for the context, which increases aggregate prefill load and worsens tail latency by intensifying prefill-decode interference in existing LLM serving stacks. Disaggregated serving reduces such interference by placing prefill and decode on separate GPUs, but disaggregation does not fundamentally eliminate inter-model redundancy in computation and KV storage for the same prompt (Woo et al., 12 Feb 2026).

This problem sits inside a broader prefill-efficiency literature that treats prompt processing as a first-order systems bottleneck. For long-context inference, the prefilling process can account for over 95% of total inference time at 128K context length, and multiple methods therefore target TTFT and prefill throughput rather than decode alone (Lv et al., 2024). PrefillShare differs from token-pruning or sparse-attention approaches because its primary objective is not to reduce the cost of a single model’s prefill, but to eliminate repeated prefill and repeated KV materialization across multiple models that share the same prompt context (Woo et al., 12 Feb 2026).

A common misconception is that prefill-decode disaggregation alone resolves the serving inefficiency. The reported analysis rejects that view explicitly: disaggregation reduces prefill-decode contention, but each model still recomputes prefill and stores duplicate KV cache for the same prefix. PrefillShare is therefore formulated as a cross-model reuse mechanism rather than merely a scheduling or placement refinement (Woo et al., 12 Feb 2026).

2. Factorized model design and cache-conditioned fine-tuning

The central architectural step is a structural separation of the LLM into a shared frozen prefill module and multiple task-specific decode modules. The shared prefill module corresponds to the main LLM backbone and is used for initial prompt encoding. Each decode module is specialized for its downstream task and consumes the shared KV cache while adapting behavior via fine-tuned parameters. At inference time, only the decode module is task-dependent; the prefill is shared (Woo et al., 12 Feb 2026).

The technical obstacle is that KV caches are normally coupled to model weights. The paper states this directly: prompt encoded with $\theta_A \neq$ prompt encoded with $\theta_B$, so naive sharing of KV cache across models causes accuracy collapse. PrefillShare addresses this with cache-conditioned fine-tuning: the prefill module is frozen at the base model weights $\theta_{\text{base}}$, and only the decode module $\theta_{\text{dec}}$ is trained while conditioning on caches produced by the frozen prefill (Woo et al., 12 Feb 2026).

The reported training objective is:

$$(\cdot, \mathcal{C}_{\text{base}}) = \mathcal{F}_{\theta_{\text{base}}}(X, \emptyset)$$

followed by

$$\mathcal{L}(\theta_{\text{dec}}) = -\sum_t \log P(y_t \mid y_{<t}, \mathcal{C}_{\text{base}}; \theta_{\text{dec}})$$

Only $\theta_{\text{dec}}$ is updated; backpropagation does not affect $\theta_{\text{base}}$ (Woo et al., 12 Feb 2026).

This construction directly answers the accuracy problem raised by naive cross-model KV reuse. The paper reports that naive sharing degrades as the proportion of shared KV cache rises, with accuracy collapsing at 100% share, whereas PrefillShare enables 100% sharing with near-full-finetuning accuracy (Woo et al., 12 Feb 2026). A plausible implication is that the decode module learns to treat the frozen prefill representation as a stable interface, so specialization is moved from prompt encoding into cache-conditioned continuation.

3. Disaggregated serving path, routing, and KV-cache semantics

PrefillShare is implemented in a vLLM-based disaggregated system. The serving stack includes a proxy that maintains a mapping between users or sessions and prefill workers, so that each session remains pinned to a prefill worker for prefix-locality. When the active model changes, such as Planner $\rightarrow$ Coder $\rightarrow$ Reviewer, the session context is sent to the same prefill worker to produce or extend the shared KV cache, which is then handed off to the required decode worker (Woo et al., 12 Feb 2026).

The cache semantics are defined with the standard split between prompt prefill and autoregressive decode. For a prompt $\theta_B$0, prefill is written as

$\theta_B$1

and decode as

$\theta_B$2

with

$\theta_B$3

In PrefillShare, $\theta_B$4 is identically used across all decode modules, and only the suffix generated during decode remains model-specific (Woo et al., 12 Feb 2026).

The system also exploits partial prefill: on each turn, only the incremental new tokens since the last context require prefill computation, so the shared cache grows incrementally rather than being recomputed from scratch (Woo et al., 12 Feb 2026). This is distinct from conventional multi-model serving, where each model repeats prompt encoding over the same shared context.

The resulting memory expression is also stated explicitly. The baseline requires

$\theta_B$5

for $\theta_B$6 models, while PrefillShare requires

$\theta_B$7

When $\theta_B$8, the shared-context memory and compute cost become once-per-session costs rather than once-per-model costs (Woo et al., 12 Feb 2026). This is the formal systems argument for why the method matters most in agentic and multi-model workloads with long shared prefixes.

4. Accuracy, latency, and throughput

The reported empirical evaluation covers math tasks such as GSM8K and GSM+, coding tasks such as HumanEval, and tool-calling tasks such as BFCL, using LLaMA3.1-8B and Qwen3-1.7B, 8B, and 14B. Across these benchmarks, PrefillShare achieves near-identical or better accuracy compared to full parameter fine-tuning. One concrete example reported in the paper is LLaMA3.1-8B on GSM8K: 71.4% for PrefillShare versus 71.3% for Full-FT (Woo et al., 12 Feb 2026).

On the serving side, the reported gains are substantial under heavy multi-model workloads. End-to-end p95 latency is reduced by up to 4.5x, and throughput is increased by up to 3.9x relative to the disaggregated baseline. The evaluation further states that the advantage persists as session rates increase and as shared context grows, that TTFT remains flat for PrefillShare even as shared prefixes grow, and that the baseline’s cache-hit rate degrades sharply at high load because of duplicated model caches and frequent eviction (Woo et al., 12 Feb 2026).

Dimension	Reported result	Context
Accuracy	Matches full fine-tuning accuracy	Broad range of tasks and models
Example	71.4% vs 71.3%	LLaMA3.1-8B on GSM8K
p95 latency	4.5x lower	Multi-model agent workloads
Throughput	3.9x higher	Multi-model agent workloads

These results are notable because they contradict the intuitive concern that freezing the prefill module would materially limit specialization. The reported measurements instead indicate that specialization can be shifted into the decode module without sacrificing the benchmarked task accuracy, while simultaneously removing repeated prefill work across models (Woo et al., 12 Feb 2026).

5. Position within the broader prefill and KV-reuse landscape

PrefillShare addresses a specific reuse pattern: multiple heterogeneous models operating over the same prompt context. Other systems target different reuse units or different bottlenecks. Prefix Cache in vLLM/PagedAttention has been widely used to reuse identical prompt prefixes, but repeated content in practical applications frequently appears as non-prefix, cross-request, cross-turn, and cross-agent segments, which makes conventional cache mechanisms insufficient. SparseX therefore introduces segment-level KV cache sharing for common serving scenarios and is compatible with Prefix Cache (Zhang et al., 1 Jun 2026). PrefillShare, by contrast, is not a non-prefix segment-reuse algorithm; it is a cross-model prefill-sharing mechanism built around a shared frozen prefill module (Woo et al., 12 Feb 2026).

ContiguousKV targets the Re-Prefill phase under persistent Prefix KV Cache offloading. It introduces ContiguousChunk as a unified data management granularity and reports a 3.85x speedup in the Re-Prefill phase over IMPRESS while maintaining high output quality (Zou et al., 20 Jan 2026). PPD disaggregation focuses on multi-turn routing, arguing that append-prefill can be handled locally on decode nodes; it reports a 68% reduction in Turn 2+ TTFT while maintaining competitive TPOT (Li et al., 9 Mar 2026). These systems therefore optimize offloaded prefix recovery and multi-turn append-prefill, whereas PrefillShare eliminates redundant prefill across different models for the same shared context (Woo et al., 12 Feb 2026).

A second axis of related work concerns pure prefill acceleration inside a single model. CritiPrefill prunes non-critical computations between query segments and cache blocks and reports up to 2.7x speedup on Llama3-8B and 3.0x speedup on Yi-9B for 128K context length (Lv et al., 2024). CLAA aggregates token-importance scores across layers and reports TTFT reduction by up to 39% compared to the Full KV Cache baseline (McDanel et al., 17 Feb 2026). UniPrefill implements block-wise dynamic sparsification and reports up to 2.1x speedup in TTFT, with gains becoming more pronounced as the number of concurrent requests grows (Fan et al., 7 May 2026). These methods are complementary in the sense that they reduce the cost of one model’s prefill, whereas PrefillShare reduces the multiplicity of prefill executions across models (Woo et al., 12 Feb 2026). This suggests a systems stack in which cross-model sharing and within-model prefill acceleration could coexist, although that composition is not claimed directly in the PrefillShare paper.

6. Scope, terminology, and deployment implications

The term “prefill” is used in at least two distinct research programs. In serving systems, it denotes prompt processing and KV-cache construction before autoregressive decode. In safety research on open-weight models, prefilling denotes allowing an attacker to predefine initial response tokens before generation begins, and recent work studies both the systematic vulnerability of open-weight models to prefill attacks and training-based defenses such as PRESTO (Struppek et al., 16 Feb 2026, Vega et al., 5 Dec 2025). PrefillShare belongs entirely to the serving-systems sense of the term: shared prompt encoding, shared KV reuse, and disaggregated inference (Woo et al., 12 Feb 2026).

Within serving, PrefillShare also has implications for cluster management. Recent work on prefill-decode disaggregation formulates SLO-aware resource allocation through total throughput, TTFT, TPOT, and request input/output lengths (Li et al., 5 Mar 2026). Other systems make prompt-length-aware routing decisions or fairness-aware chunked-prefill scheduling to control latency jitter, head-of-line blocking, and request starvation (She et al., 4 Jan 2026, Liu et al., 8 Jun 2026). PrefillShare adds a separate degree of freedom: it changes the amount of prefill work that must exist at all in multi-model agent workloads, rather than only changing how that work is scheduled (Woo et al., 12 Feb 2026). A plausible implication is that lower aggregate prefill load should alter the operating point assumed by these schedulers and resource-allocation models.

The article’s main technical significance is therefore specific and narrow: PrefillShare does not merely cache prefixes within one model, and it does not merely route prefills more intelligently. It makes the prefill representation itself a shared artifact across heterogeneous models by freezing the prefill module and training decode modules to consume the resulting shared cache. In the reported experiments, that design simultaneously preserves the benchmarked accuracy of full fine-tuning and materially improves p95 latency and throughput under multi-model agent workloads (Woo et al., 12 Feb 2026).