---
title: Layer-Segmented Prefill Techniques
url: https://www.emergentmind.com/topics/layer-segmented-prefill
type: topic
---

# Layer-Segmented Prefill Techniques

Layer-Segmented Prefill denotes a family of long-context inference and serving strategies that use **network depth** as the primary control dimension for prompt processing. Instead of treating prefill as a monolithic forward pass, these methods introduce an explicit layer boundary, layer group schedule, or layer-dependent prompt-visibility rule. In recent work, this appears in several distinct forms: **Token-Selective Propagation (TSP)**, which keeps full-context processing up to a designated layer and then propagates only salient tokens to deeper layers; **Prefill-Only Pruning (POP)**, which skips a contiguous deep suffix during prefill but restores the full model for decode; **layered prefill**, which makes contiguous layer groups the scheduling unit for stall-free serving; **SPEED**, which keeps non-anchor prompt-token KV states only in lower layers; and **ILRe**, which stops long-context processing at an intermediate retrieval layer and then restarts full-model inference on a compressed prompt [2502.01068] [2602.03295] [2510.08055] [2605.06105] [2508.17892].

## 1. Scope and conceptual boundaries

In this literature, the term is most precise when it refers to methods that partition prompt processing **along network depth**, rather than along token chunks, request queues, GPU resources, or the prefill/decode phase boundary. A direct instance is a method that changes which layers process full prompt information, or changes how layer groups are scheduled. A close but indirect instance is a method that remains sequence-segmented or token-segmented, while still exploiting cross-layer regularities.

| Form | Representative work | Relation to layer-segmented prefill |
|---|---|---|
| Depth boundary inside prefill | FastKV, POP, SPEED | Direct: later/deep layers see reduced token sets, skipped layers, or reduced prompt visibility |
| Layer groups as scheduling units | layered prefill | Direct: contiguous layer groups are the primary serving unit |
| Intermediate-layer partial prefill | ILRe | Directly related: stop long-context processing at one intermediate layer, then restart on compressed prompt |
| Cross-layer scoring or grouping | CritiPrefill, VSPrefill, CLAA, ContiguousKV | Adjacent: layer-aware, but not full layer-segmented execution |
| Phase/resource segmentation | Nexus, PrefillShare | Distinct: segmentation is by phase, GPU resources, or shared prefill module, not by model depth |

The distinction from **token chunking** is explicit in several papers. Nexus states that chunked prefill divides the prompt along the **token sequence dimension**, whereas layer segmentation would divide execution along the **network depth dimension**, and it emphasizes that its own design is **resource partition + queue partition + phase partition**, not layer partition [2507.06608]. PrefillShare likewise factorizes serving into a **shared prefill module** and a **task-specific decode module**, but does not split the transformer at an intermediate layer boundary [2602.12029].

## 2. Direct depth-based architectures

FastKV is a canonical depth-partitioned design. It divides prefill into two regimes separated by a **TSP layer**: up to that layer, the model runs on the **full input sequence**; starting at that layer, it propagates only a reduced subset of salient tokens and all window tokens to subsequent layers. The paper motivates this with an empirical observation that critical context is unstable in early layers but becomes more stable in later layers. FastKV then **decouples** the TSP rate \(R_{TSP}\), which controls later-layer hidden-state propagation, from the KV retention rate \(R_{KV}\), which controls decode-time KV size. In its reported default setting, \(R_{TSP}=20\%\) yields a **60% prefill compute rate** on the main models, and the paper reports **up to \(1.82\times\) prefill speedup** and **\(2.87\times\) decoding speedup** while matching the accuracy of baselines that only accelerate decoding [2502.01068].

POP is a more explicit **deep-layer skip** design. It argues that deep layers are critical for next-token prediction during decode but largely redundant for context encoding during prefill. The resulting policy is static and contiguous: during prefill, POP skips the **last \(1/3\)** of the layers, using the identity update \(\hat{x}_{l+1} := x_l\) for skipped layers, while still generating per-layer KV entries through **independent KV projections**. To preserve the first generated token, POP shifts the stage boundary so that prefill processes \(x_{1:N-1}\) and the last prompt token \(x_N\) becomes the first decode step. The paper reports **up to \(1.37\times\) speedup in prefill latency with minimal performance loss** and identifies the **deep contiguous suffix** as the robust pruning region across the tested LLM and VLM families [2602.03295].

SPEED is a hard **layer-asymmetric KV-visibility** policy. With cutoff \(K<L\), non-anchor prompt tokens are materialized only in lower layers, while decode tokens remain full-depth. For the current decode token \(d_t\), the visible set is:
\[
P \cup A \cup D_{<t} \cup \{d_t\}, \quad l \le K,
\]
and
\[
A \cup D_{<t} \cup \{d_t\}, \quad l > K,
\]
where \(P\) is the non-anchor prompt set and \(A\) is the anchor set. In the main variant, the anchor is just the **BoS token**. On a controlled **Llama-3.1-8B** study, SPEED using only **75\% of layers for prefill tokens** reaches **51.2** average score versus **51.4** for the full-depth baseline, while improving **TTFT by 33\%**, **TPOT by 22\%**, and reducing **active KV memory by 25.0\% at 128K context** [2605.06105].

ILRe is a retrieval-driven form of partial depth processing. It chooses one intermediate decoder layer \(l_R\) offline, runs the long context only through layers \(1,\dots,l_R\), and uses the attention scores between the query and the full key cache at that layer to retrieve a budgeted subset of original tokens. The final generation stage then runs standard full-model inference on the compressed prompt. For **Llama-3.1-UltraLong-8B-1M-Instruct**, the selected layer is \(l_R=3\) with \(N_L=32\), and the paper states that ILRe can process a single **1M-token** request in **less than half a minute** with **speedup \(\approx 180\times\)** and **RULER-1M \(\approx 79.8\)** [2508.17892].

## 3. Layer groups as serving primitives

“From Tokens to Layers: Redefining Stall-Free Scheduling for LLM Serving with Layered Prefill” makes layer segmentation the serving abstraction itself. The decoder stack is partitioned **vertically** into \(G\) **contiguous layer groups**, and the scheduling rule is that in each iteration **exactly one designated layer group** performs **both prefill and decode**, while all remaining groups perform **decode only**. Prefill for a request advances by one group per iteration; after \(G\) iterations, that request’s prefill is complete. The adaptive grouping rule is
\[
G(L)=\max\bigl(1,\lceil L/512 \rceil \bigr),
\]
so the number of groups scales with prompt length. This replaces chunked prefill’s token-axis segmentation with a layer-axis schedule [2510.08055].

The paper’s motivation is specific to **MoE** serving. Chunked prefill forces each token chunk to traverse the same transformer layers and repeatedly reload expert weights, which amplifies expert-load traffic. Layered prefill avoids replaying MoE layers across prompt chunks because each layer is traversed once during prefill. On Qwen over the arXiv workload, the paper reports that total expert weight loads over a fixed trace of 100 requests fall from **35.6 TB** under chunked prefill to **21.7 TB** under layered prefill, a **39.0\%** reduction. The abstract reports **TTFT by up to 70\%**, **End-to-End latency by 41\%**, and **per-token energy by up to 22\%**, while maintaining stall-free decoding [2510.08055].

ASAP is related but not identical. It **disaggregates the attention and MoE stages** and builds a **fully asynchronous execution pipeline** for MoE prefill. Requests advance layer by layer through alternating attention and MoE stages, but the segmentation boundary is **operator type inside each layer**, not a contiguous range of layer indices. ASAP introduces asynchronous dispatch/combine primitives, length-aware batching, dual-batch interleaving, communication-computation overlap, and an MoE Super Kernel. On CloudMatrix384, it improves **SLO-compliant prefill throughput by 90\%** compared to state-of-the-art synchronous serving solutions [2606.22541].

Nexus is a useful counterexample. It asks whether prefill/decode disaggregation can be achieved within a single engine and answers with **dynamic SM partitioning**, separate GPU streams, and separate schedulers. The paper is explicit that this is **not** layer-segmented prefill: the segmentation axes are **phase**, **GPU resources**, and **scheduling/batching domains**, while the prefill computation itself remains semantically intact [2507.06608].

## 4. Adjacent layer-aware sparsification and retrieval

Several influential methods are not fully layer-segmented, but they supply much of the empirical and algorithmic basis for later depth-partitioned designs.

CritiPrefill is **segment-wise** over the sequence dimension rather than the depth dimension. It partitions the query sequence into fixed-size query segments and the KV cache into fixed-size blocks, estimates a segment-by-block criticality matrix, and prunes non-critical interactions during prefill. Its layer-specific mechanism is **layer fusion**:
\[
S = \alpha S + (1-\alpha)S',
\]
with \(\alpha = 0.25\) in experiments. The paper describes this as layer-aware refinement rather than layer-segmented execution, and reports **up to 2.7x speedup on Llama3-8B** and **3.0x speedup on Yi-9B** at **128K** context on a single A100 GPU [2409.12490].

VSPrefill is layer-aware through **adaptive cumulative-threshold budgeting** per layer. It predicts **vertical columns** and **slash diagonals** from RoPE-augmented KV representations, then selects the smallest number of indices whose cumulative predicted mass reaches threshold \(\tau_d\):
\[
k_d = \min\left\{k \,\middle|\, \sum_{i=1}^{k} \mathrm{sort}(\hat A_d)_i \ge \tau_d \right\}.
\]
The paper repeatedly states that attention structure varies across layers and that masks should therefore be layer-specific. It reports **98.35\%** of full-attention accuracy with a **4.95x average speedup at a context length of 128k** [2603.04460].

CLAA diagnoses a different layer effect: **token-importance rankings are unstable across layers**. It introduces an Answer-Informed Oracle and then aggregates layer scores across a window \(\mathcal{L}\) using
\[
S_i^{\text{CLAA}} = \max_{l' \in \mathcal{L}} S_i^{(l')}.
\]
The method still prunes only once, at a chosen pruning layer, so it is not a true multi-segment forward policy. Its contribution is a cross-layer scoring rule that improves the robustness of a single pruning boundary, and it reduces **TTFT by up to 39\% compared to the Full KV Cache baseline** [2602.16054].

ContiguousKV is about **Re-Prefill** with persistent shared-prefix KV offloading. It groups consecutive layers into **Periods** that share the same important ContiguousChunk indices, based on the observation that adjacent layers tend to have similar chunk selections. The reported average similarity between consecutive Periods is **52\% to 64\%** on Qwen2.5-7B, and the system achieves a **3.85x speedup in the Re-Prefill phase over IMPRESS**. This is best understood as a layer-aware alternative to explicit layer segmentation: the content axis is token-contiguous chunking, while the layer axis appears as **Period-based reuse** rather than depth-partitioned execution [2601.13631].

## 5. Empirical depth patterns and design principles

Across the direct depth-based methods, several layer-structured regularities recur. FastKV reports that the overlap of top critical tokens between layers drops rapidly in early layers and remains much more stable in later layers; this is the empirical basis for placing the TSP layer around the middle of the decoder and for letting later layers operate on a reduced token set [2502.01068]. POP reports the complementary stage-asymmetry result: deep layers have low or near-zero importance in prefill but become increasingly important in decode, which is why it prunes a **deep contiguous suffix** only during prefill and restores the full model for generation [2602.03295].

SPEED adds a more fine-grained depth diagnostic. In its layer-wise analysis, raw prompt-attention peaks can happen early, but **selective prompt access** emerges later, around **L13–L15**, and **straightening/stabilization** peaks later still, around **L17–L19** in the 32-layer model. The reported practical implication is that the cutoff should sit **above** the selective-prompt-access and stabilization region rather than at the earliest prompt-attention peak, which is why **\(K=24\)** becomes the main operating point [2605.06105].

Layered prefill provides the serving-side analogue of those representational findings. Its results are strongest on **arXiv**, whose mean input length is **9194** tokens and whose p90 input length is **17152**, because long prompts would otherwise force many small token chunks and repeatedly replay MoE layers. The paper’s interpretation is that changing the scheduling axis from tokens to layers aligns the execution unit with the MoE expert-load unit, which improves the **TTFT–TBT Pareto frontier** [2510.08055].

ILRe supplies a related design rule for intermediate-layer stopping points: choose the layer with the **smallest index** among those with the **highest recall rate** on a retrieval benchmark. This suggests that a useful depth boundary need not be deep if the target function is retrieval rather than full prompt representation [2508.17892].

Taken together, these results suggest that depth boundaries are most effective when they preserve early and middle-layer context construction, but reduce or restructure prompt participation once token importance, prompt selection, or representation trajectories have stabilized. This is an inference from multiple papers rather than a single unified theorem.

## 6. Distinctions, limitations, and open problems

A recurring misconception is to treat any finer-grained prefill method as layer-segmented. The literature is more specific. **Chunked prefill** is token-axis segmentation, not depth segmentation [2507.06608]. **Kairos** deflects prefill to decode nodes as **adaptive token-chunked prefill steps** interleaved with decode; it is segmented in time and by tokens, not by layers [2607.02043]. **Prefill-as-a-Service** moves prefill across clusters and transfers the resulting KV cache at the **phase boundary**; its one layer-related remark is “layer-wise prefill pipelining,” but it does not define a layer-partitioned prefill architecture [2604.15039]. **FlexNPU** provides phase-aware NPU virtualization and dynamic PD co-location, but the paper is explicit that it does not present a layer-aware or layer-segment-aware scheduler [2606.04415].

Even among direct depth-based methods, the constraints differ sharply. POP’s segmentation is **static**, depends on calibration data, and does **not** reduce peak model memory because decode still uses the full model [2602.03295]. SPEED’s strongest results rely on **SPEED-aware training or adaptation**; the paper shows that **PostHoc-SPEED** creates train/test mismatch and degrades more sharply at aggressive cutoffs [2605.06105]. FastKV’s segmentation boundary is useful because later-layer importance stabilizes, but the paper still requires offline calibration of the TSP layer and does not offer a universal choice across models [2502.01068].

Open problems remain concentrated around **intermediate state management**. Nexus notes that a true layer-segmented prefill system would likely need **kernel restructuring**, **explicit pipeline boundaries inside the transformer**, or **intermediate activation buffering between layer segments**, none of which its phase-level resource partitioning addresses [2507.06608]. ContiguousKV leaves open how to define a stable interface for **layer-state partitioning** rather than token-region selection [2601.13631]. CLAA shows that cross-layer evidence is better than trusting one pruning layer, but it does not answer whether different layer segments should use **different retained token subsets** or how often re-ranking should occur [2602.16054].

A further open axis is **layer-wise KV precision or placement**. SpectrumKV is not a layer-segmented prefill method; it keeps the full layer structure intact and assigns each token a precision tier. But its layer-wise propagation analysis,
\[
\|\tilde x^{(L)} - x^{(L)}\| \le \sum_{l=1}^{L} \delta_l \cdot \prod_{k=l+1}^{L} (1 + \Lambda_k \cdot \alpha_k),
\]
and its per-layer INT4 cosine variation suggest that future systems may want to combine **depth segmentation** with **per-layer precision allocation** rather than treating the layer and token axes independently [2606.08635].

The field therefore contains both direct instances and adjacent precursors. Direct methods show that prefill can be segmented by layer depth for **execution**, **visibility**, or **serving schedule**. Adjacent methods show that layers differ in **importance stability**, **sparsity structure**, **retrieval utility**, and **error sensitivity**. The remaining challenge is to integrate those observations into unified systems that choose depth boundaries, manage intermediate activations, and co-design segmentation with KV transfer, fused kernels, paging, and online schedulers.

Source: https://www.emergentmind.com/topics/layer-segmented-prefill