---
title: 'Chunked Prefill: Efficient LLM Inference'
url: https://www.emergentmind.com/topics/chunked-prefill
type: topic
---

# Chunked Prefill: Efficient LLM Inference

Chunked prefill is a scheduling and batching strategy for transformer-based language model inference, wherein long input sequences are partitioned into smaller, fixed-size chunks and processed sequentially through all model layers. Each chunk populates key–value (KV) caching for subsequent decode operations, enabling efficient utilization of hardware resources during the prefill (prompt ingestion) phase and reducing time-to-first-token (TTFT) latency. Its adoption across dense, sparse, and Mixture-of-Experts (MoE) models has established chunked prefill as a foundational regime for modern LLM serving, but it introduces nontrivial trade-offs in latency, throughput, memory traffic, and scheduling complexity.

## 1. Formulation and Algorithmic Structure

Chunked prefill splits a prompt of length $L$ into $\lceil L/S \rceil$ contiguous chunks of size $S$. For each chunk, forward computation is performed through all transformer layers, appending the newly computed KV vectors to the accumulated cache, then continuing to the next chunk. After all chunks are processed, autoregressive decode resumes using the completed cache [2308.16369][2510.08055]. The scheduling loop typically interleaves chunked-prefill tasks with decode tasks in hybrid micro-batches, exploiting fused matmul kernels and maximizing hardware throughput [2308.16369].

#### Pseudocode (Abstracted):
```python
while Q_prefill or Q_decode:
    if Q_prefill:
        prefill_chunk = Q_prefill.dequeue()
        schedule(prefill_chunk, slot=1)
        num_decode_slots = B-1
    else:
        num_decode_slots = B
    for i in range(num_decode_slots):
        if Q_decode:
            decode_task = Q_decode.dequeue()
            schedule(decode_task, slot=i+2)
    launch_fused_kernels()
    update_queues()
```
Each chunk operation is compute-bound and admits compute–decode piggybacking: decodes utilize the already-loaded model weights, amortizing GPU resource usage.

## 2. Chunked Prefill in Serving Architectures

Chunked prefill appears as the default iteration granularity in multiple LLM serving frameworks: Sarathi-Serve [2308.16369], vLLM, SGLang, RServe [2509.24381], and is core to pipeline-parallel, data-parallel, and disaggregated PD architectures [2507.06608][2508.01989]. In Sarathi-Serve, decode-maximal batching fuses one prefill chunk with as many decodes as possible, allowing compute-intensive prefill and memory-bound decode to efficiently share large matmuls. In RServe’s multi-modal and pipeline-disaggregated architecture, chunked prefill enables fine-grained overlap between encoder outputs and language model prefill, supporting both intra-request and inter-request pipelines for improved parallelism [2509.24381].

Prefill–decode disaggregation frameworks (such as TaiChi [2508.01989]) use chunk size as an explicit control lever to trade off TTFT and time-per-output-token (TPOT) across different pools of GPU hardware, thus generalizing chunked prefill as a fundamental axis for balancing latency and throughput SLOs across a wide range of deployment regimes.

## 3. Performance, Latency, and Throughput Trade-Offs

The size of each chunk, $S$, directly modulates key SLO metrics:
- **TTFT (Time-to-First-Token):** Small $S$ improves responsiveness by reducing the single-iteration time to process the initial portion of a long prompt, mitigating head-of-line blocking and allowing high-priority requests to preempt ongoing work [2602.16603]. However, smaller chunks entail more kernel launches and iteration overhead, limiting throughput.
- **TPOT (Time-Per-Output-Token):** Large $S$ can cause phase interference between prefill and decode, where memory-bound decode operations wait on compute-bound prefill, increasing TBT (time-between-tokens) [2507.06608][2510.08055]. Small $S$ better isolates decode, keeping TBT low.

Empirically, chunk sizes of 256–512 tokens yield strong throughput–latency performance for dense models [2308.16369], whereas Tightly-constrained SLOs (TBT $<20$ms) may require even smaller $S$ [2510.08055].

Quantitative impacts (representative):
- For LLaMA-13B/A6000: Up to 10$\times$ higher decode throughput (piggybacked) and $1.33 \times$ higher end-to-end throughput relative to decode-only or pure prefill [2308.16369].
- In production workloads (Qwen3-30B, 9k-token arXiv): TTFT mean reduced from 4.50s (no chunking) to 2.80s (chunked S=512), with TBT mean reduced from 45ms to 32.9ms [2510.08055].

The trade-off forms a continuum: decreasing $S$ reduces TTFT but adds overhead, while increasing $S$ improves throughput but exacerbates decode blocking [2602.16603].

## 4. System-Level Challenges and Optimizations

### 4.1. Phase Interference and Intra-GPU Disaggregation

When prefill and decode are co-batched in the same GPU streams, kernel-level profiling shows memory-bound decode kernels can be delayed by up to 8–10$\times$ compared to decode-only batches, as prefill matmuls monopolize streaming multiprocessors (SMs) and off-chip bandwidth [2507.06608]. Systems such as Nexus decouple GPU resources dynamically between prefill and decode, partitioning SMs and using two independent schedulers (prefill: shortest-prompt-first; decode: FCFS) to virtually disaggregate the phases within a single device, substantially reducing TTFT and TBT while achieving up to 2.2$\times$ higher throughput [2507.06608].

### 4.2. Scheduling and Batching

Chunked prefill enables advanced batching techniques—e.g., decode-maximal batching in Sarathi [2308.16369] and token-budgeted micro-batching in RServe [2509.24381]—to align hardware saturation with workload variability. In multi-GPU or disaggregated settings, chunk scheduling can be dynamically tuned across heterogeneous instances for optimal SLO attainment [2508.01989].

Slack-aware scheduling (e.g., S-EDF in FlowPrefill [2602.16603]) uses per-request slack windows and batch-token budgets to optimize admission control, minimizing wasted compute on requests destined to miss SLO deadlines while still achieving high utilization. Fine-grained preemption at operator or layer level (rather than at chunk boundaries) can further reduce preemption blocking by 3.5–4.2$\times$ [2602.16603].

### 4.3. Limitations in MoE, Memory, and Bandwidth

In Mixture-of-Experts (MoE) models, chunked prefill with small $S$ erodes weight sparsity: each chunk reloads a redundant set of expert weights, driving memory traffic up to 39% higher and energy per token by 22% versus naive batching [2510.08055]. This is especially acute for long contexts and large expert sets, where redundant parameter loads and sparse-expert underutilization render chunked prefill suboptimal.

Memory bottlenecks, particularly during the prefill stage, motivated the development of schemes like MOM, which partition intermediate activations into mini-sequences internal to each MLP and offload KV caches to CPU, reducing prefill memory consumption by over 50% and enabling significantly longer context lengths (e.g., 455k tokens vs 338k for chunked prefill) [2504.12526].

## 5. Specialized Techniques: Chunked Prefill as Substrate

### 5.1. Sparse Attention and KV Selection

Chunked prefill serves as a substrate for various acceleration techniques:
- **QUOKA** performs two-stage token-level KV selection during chunked prefill, maintaining near-dense accuracy while reducing the number of KV pairs by 88% and attention compute by up to 7$\times$ [2602.08722]. Its query-wise sub-selection is compatible with common chunked prefill regimes.
- **CompactAttention** further exploits block-union KV selection: starting from 2D block-sparse masks (from selectors like FlashPrefill), it performs Q-block union and GQA-group intra-group union to derive minimal per-group KV block tables. This enables zero-copy paged attention, eliminating explicit KV gathering, and yields up to 2.72$\times$ attention speedup (128k context) with accuracy within 0.3 pp of dense attention [2605.16839].

Table: Chunked prefill acceleration approaches

| Approach         | Key Mechanism            | Speedup / KV Reduction     | Accuracy Impact      |
|------------------|-------------------------|----------------------------|---------------------|
| QUOKA            | Query-oriented KV        | 5–7$\times$ attention      | <3% drop (typical)  |
| CompactAttention | Block-union, paged zero-copy | 2.72$\times$ at 128k context | ≲0.3 pp from dense  |
| MOM              | Mini-sequence + offload | 1.5–2$\times$ memory reduction | None (identical)    |

### 5.2. Multimodal and Multi-turn Scenarios

Chunked prefill is integral to multimodal serving engines (RServe), where it enables overlapping multimodal encoding with LLM prefill, both intra-request (streamed embeddings) and inter-request (token-budgeted batching) [2509.24381]. In multi-turn conversational settings, append-prefill mechanisms reuse cached KV states for successive turns, amortizing the quadratic prefill cost and, when dynamically routed, can yield a 68% reduction in Turn-2+ TTFT [2603.13358].

## 6. Practical Guidelines and Future Directions

**Choosing chunk size $S$:** The main guidelines—drawn from Sarathi, TaiChi, and Layered Prefill—are:
- For strict TTFT SLOs ($<3$ s), large $S$ (e.g., 512–1024) amortizes iteration overhead.
- For strict TPOT ($<$100 ms/token), small $S$ (128–256) isolates decode-sensitive batches.
- For MoE or long-prompt workloads, hybrid chunk+layer or block-union schemes offer better trade-offs to avoid memory “explosion” [2510.08055][2605.16839].
 
Hybrid architectures (TaiChi) adapt chunk size and resource assignment per hardware pool and workload regime to maximize overall goodput under joint TTFT/TPOT SLOs, obtaining 20–77% higher request-attainment rates compared to pure aggregation or disaggregation [2508.01989].

Ongoing work explores dynamic per-layer or per-request chunk sizing, fused attention/MLP partitioning, and further integration with offloading and quantization. Memory-efficient management (e.g. ContiguousKV's chunk-aligned prefetch) now targets decode-stage KV bottlenecks as prefill pressure is relieved [2601.13631][2504.12526].

---

Chunked prefill remains a central paradigm in LLM serving: it is directly extensible to dense, sparse, MoE, multi-modal, and multi-turn workloads; but effective deployment requires careful balancing of chunk size, pipeline structure, resource scheduling, and integration with advanced acceleration and caching techniques to avoid phase interference, memory inefficiency, and bandwidth bottlenecks.

Source: https://www.emergentmind.com/topics/chunked-prefill