Layer-Segmented Prefill Techniques

Updated 14 July 2026

Layer-Segmented Prefill is a set of techniques that partitions prompt processing along transformer depth, not token sequences.
Methods like TSP, POP, SPEED, and ILRe apply layer-dependent scheduling to reduce compute and memory usage while preserving accuracy.
Empirical results demonstrate significant speedups and memory savings by selectively processing token information in deeper layers.

Layer-Segmented Prefill denotes a family of long-context inference and serving strategies that use network depth as the primary control dimension for prompt processing. Instead of treating prefill as a monolithic forward pass, these methods introduce an explicit layer boundary, layer group schedule, or layer-dependent prompt-visibility rule. In recent work, this appears in several distinct forms: Token-Selective Propagation (TSP), which keeps full-context processing up to a designated layer and then propagates only salient tokens to deeper layers; Prefill-Only Pruning (POP), which skips a contiguous deep suffix during prefill but restores the full model for decode; layered prefill, which makes contiguous layer groups the scheduling unit for stall-free serving; SPEED, which keeps non-anchor prompt-token KV states only in lower layers; and ILRe, which stops long-context processing at an intermediate retrieval layer and then restarts full-model inference on a compressed prompt (Jo et al., 3 Feb 2025, He et al., 3 Feb 2026, Lee et al., 9 Oct 2025, Oh et al., 7 May 2026, Liang et al., 25 Aug 2025).

1. Scope and conceptual boundaries

In this literature, the term is most precise when it refers to methods that partition prompt processing along network depth, rather than along token chunks, request queues, GPU resources, or the prefill/decode phase boundary. A direct instance is a method that changes which layers process full prompt information, or changes how layer groups are scheduled. A close but indirect instance is a method that remains sequence-segmented or token-segmented, while still exploiting cross-layer regularities.

Form	Representative work	Relation to layer-segmented prefill
Depth boundary inside prefill	FastKV, POP, SPEED	Direct: later/deep layers see reduced token sets, skipped layers, or reduced prompt visibility
Layer groups as scheduling units	layered prefill	Direct: contiguous layer groups are the primary serving unit
Intermediate-layer partial prefill	ILRe	Directly related: stop long-context processing at one intermediate layer, then restart on compressed prompt
Cross-layer scoring or grouping	CritiPrefill, VSPrefill, CLAA, ContiguousKV	Adjacent: layer-aware, but not full layer-segmented execution
Phase/resource segmentation	Nexus, PrefillShare	Distinct: segmentation is by phase, GPU resources, or shared prefill module, not by model depth

The distinction from token chunking is explicit in several papers. Nexus states that chunked prefill divides the prompt along the token sequence dimension, whereas layer segmentation would divide execution along the network depth dimension, and it emphasizes that its own design is resource partition + queue partition + phase partition, not layer partition (Shi et al., 9 Jul 2025). PrefillShare likewise factorizes serving into a shared prefill module and a task-specific decode module, but does not split the transformer at an intermediate layer boundary (Woo et al., 12 Feb 2026).

2. Direct depth-based architectures

FastKV is a canonical depth-partitioned design. It divides prefill into two regimes separated by a TSP layer: up to that layer, the model runs on the full input sequence; starting at that layer, it propagates only a reduced subset of salient tokens and all window tokens to subsequent layers. The paper motivates this with an empirical observation that critical context is unstable in early layers but becomes more stable in later layers. FastKV then decouples the TSP rate $R_{TSP}$ , which controls later-layer hidden-state propagation, from the KV retention rate $R_{KV}$ , which controls decode-time KV size. In its reported default setting, $R_{TSP}=20\%$ yields a 60% prefill compute rate on the main models, and the paper reports up to $1.82\times$ prefill speedup and $2.87\times$ decoding speedup while matching the accuracy of baselines that only accelerate decoding (Jo et al., 3 Feb 2025).

POP is a more explicit deep-layer skip design. It argues that deep layers are critical for next-token prediction during decode but largely redundant for context encoding during prefill. The resulting policy is static and contiguous: during prefill, POP skips the last $1/3$ of the layers, using the identity update $\hat{x}_{l+1} := x_l$ for skipped layers, while still generating per-layer KV entries through independent KV projections. To preserve the first generated token, POP shifts the stage boundary so that prefill processes $x_{1:N-1}$ and the last prompt token $x_N$ becomes the first decode step. The paper reports up to $1.37\times$ speedup in prefill latency with minimal performance loss and identifies the deep contiguous suffix as the robust pruning region across the tested LLM and VLM families (He et al., 3 Feb 2026).

SPEED is a hard layer-asymmetric KV-visibility policy. With cutoff $R_{KV}$ 0, non-anchor prompt tokens are materialized only in lower layers, while decode tokens remain full-depth. For the current decode token $R_{KV}$ 1, the visible set is: $R_{KV}$ 2 and

$R_{KV}$ 3

where $R_{KV}$ 4 is the non-anchor prompt set and $R_{KV}$ 5 is the anchor set. In the main variant, the anchor is just the BoS token. On a controlled Llama-3.1-8B study, SPEED using only 75\% of layers for prefill tokens reaches 51.2 average score versus 51.4 for the full-depth baseline, while improving TTFT by 33\%, TPOT by 22\%, and reducing active KV memory by 25.0\% at 128K context (Oh et al., 7 May 2026).

ILRe is a retrieval-driven form of partial depth processing. It chooses one intermediate decoder layer $R_{KV}$ 6 offline, runs the long context only through layers $R_{KV}$ 7, and uses the attention scores between the query and the full key cache at that layer to retrieve a budgeted subset of original tokens. The final generation stage then runs standard full-model inference on the compressed prompt. For Llama-3.1-UltraLong-8B-1M-Instruct, the selected layer is $R_{KV}$ 8 with $R_{KV}$ 9, and the paper states that ILRe can process a single 1M-token request in less than half a minute with speedup $R_{TSP}=20\%$ 0 and RULER-1M $R_{TSP}=20\%$ 1 (Liang et al., 25 Aug 2025).

3. Layer groups as serving primitives

“From Tokens to Layers: Redefining Stall-Free Scheduling for LLM Serving with Layered Prefill” makes layer segmentation the serving abstraction itself. The decoder stack is partitioned vertically into $R_{TSP}=20\%$ 2 contiguous layer groups, and the scheduling rule is that in each iteration exactly one designated layer group performs both prefill and decode, while all remaining groups perform decode only. Prefill for a request advances by one group per iteration; after $R_{TSP}=20\%$ 3 iterations, that request’s prefill is complete. The adaptive grouping rule is

$R_{TSP}=20\%$ 4

so the number of groups scales with prompt length. This replaces chunked prefill’s token-axis segmentation with a layer-axis schedule (Lee et al., 9 Oct 2025).

The paper’s motivation is specific to MoE serving. Chunked prefill forces each token chunk to traverse the same transformer layers and repeatedly reload expert weights, which amplifies expert-load traffic. Layered prefill avoids replaying MoE layers across prompt chunks because each layer is traversed once during prefill. On Qwen over the arXiv workload, the paper reports that total expert weight loads over a fixed trace of 100 requests fall from 35.6 TB under chunked prefill to 21.7 TB under layered prefill, a 39.0\% reduction. The abstract reports TTFT by up to 70\%, End-to-End latency by 41\%, and per-token energy by up to 22\%, while maintaining stall-free decoding (Lee et al., 9 Oct 2025).

ASAP is related but not identical. It disaggregates the attention and MoE stages and builds a fully asynchronous execution pipeline for MoE prefill. Requests advance layer by layer through alternating attention and MoE stages, but the segmentation boundary is operator type inside each layer, not a contiguous range of layer indices. ASAP introduces asynchronous dispatch/combine primitives, length-aware batching, dual-batch interleaving, communication-computation overlap, and an MoE Super Kernel. On CloudMatrix384, it improves SLO-compliant prefill throughput by 90\% compared to state-of-the-art synchronous serving solutions (Chen et al., 21 Jun 2026).

Nexus is a useful counterexample. It asks whether prefill/decode disaggregation can be achieved within a single engine and answers with dynamic SM partitioning, separate GPU streams, and separate schedulers. The paper is explicit that this is not layer-segmented prefill: the segmentation axes are phase, GPU resources, and scheduling/batching domains, while the prefill computation itself remains semantically intact (Shi et al., 9 Jul 2025).

4. Adjacent layer-aware sparsification and retrieval

Several influential methods are not fully layer-segmented, but they supply much of the empirical and algorithmic basis for later depth-partitioned designs.

CritiPrefill is segment-wise over the sequence dimension rather than the depth dimension. It partitions the query sequence into fixed-size query segments and the KV cache into fixed-size blocks, estimates a segment-by-block criticality matrix, and prunes non-critical interactions during prefill. Its layer-specific mechanism is layer fusion: $R_{TSP}=20\%$ 5 with $R_{TSP}=20\%$ 6 in experiments. The paper describes this as layer-aware refinement rather than layer-segmented execution, and reports up to 2.7x speedup on Llama3-8B and 3.0x speedup on Yi-9B at 128K context on a single A100 GPU (Lv et al., 2024).

VSPrefill is layer-aware through adaptive cumulative-threshold budgeting per layer. It predicts vertical columns and slash diagonals from RoPE-augmented KV representations, then selects the smallest number of indices whose cumulative predicted mass reaches threshold $R_{TSP}=20\%$ 7: $R_{TSP}=20\%$ 8 The paper repeatedly states that attention structure varies across layers and that masks should therefore be layer-specific. It reports 98.35\% of full-attention accuracy with a 4.95x average speedup at a context length of 128k (Guanzhong, 3 Mar 2026).

CLAA diagnoses a different layer effect: token-importance rankings are unstable across layers. It introduces an Answer-Informed Oracle and then aggregates layer scores across a window $R_{TSP}=20\%$ 9 using

$1.82\times$ 0

The method still prunes only once, at a chosen pruning layer, so it is not a true multi-segment forward policy. Its contribution is a cross-layer scoring rule that improves the robustness of a single pruning boundary, and it reduces TTFT by up to 39\% compared to the Full KV Cache baseline (McDanel et al., 17 Feb 2026).

ContiguousKV is about Re-Prefill with persistent shared-prefix KV offloading. It groups consecutive layers into Periods that share the same important ContiguousChunk indices, based on the observation that adjacent layers tend to have similar chunk selections. The reported average similarity between consecutive Periods is 52\% to 64\% on Qwen2.5-7B, and the system achieves a 3.85x speedup in the Re-Prefill phase over IMPRESS. This is best understood as a layer-aware alternative to explicit layer segmentation: the content axis is token-contiguous chunking, while the layer axis appears as Period-based reuse rather than depth-partitioned execution (Zou et al., 20 Jan 2026).

5. Empirical depth patterns and design principles

Across the direct depth-based methods, several layer-structured regularities recur. FastKV reports that the overlap of top critical tokens between layers drops rapidly in early layers and remains much more stable in later layers; this is the empirical basis for placing the TSP layer around the middle of the decoder and for letting later layers operate on a reduced token set (Jo et al., 3 Feb 2025). POP reports the complementary stage-asymmetry result: deep layers have low or near-zero importance in prefill but become increasingly important in decode, which is why it prunes a deep contiguous suffix only during prefill and restores the full model for generation (He et al., 3 Feb 2026).

SPEED adds a more fine-grained depth diagnostic. In its layer-wise analysis, raw prompt-attention peaks can happen early, but selective prompt access emerges later, around L13–L15, and straightening/stabilization peaks later still, around L17–L19 in the 32-layer model. The reported practical implication is that the cutoff should sit above the selective-prompt-access and stabilization region rather than at the earliest prompt-attention peak, which is why $1.82\times$ 1 becomes the main operating point (Oh et al., 7 May 2026).

Layered prefill provides the serving-side analogue of those representational findings. Its results are strongest on arXiv, whose mean input length is 9194 tokens and whose p90 input length is 17152, because long prompts would otherwise force many small token chunks and repeatedly replay MoE layers. The paper’s interpretation is that changing the scheduling axis from tokens to layers aligns the execution unit with the MoE expert-load unit, which improves the TTFT–TBT Pareto frontier (Lee et al., 9 Oct 2025).

ILRe supplies a related design rule for intermediate-layer stopping points: choose the layer with the smallest index among those with the highest recall rate on a retrieval benchmark. This suggests that a useful depth boundary need not be deep if the target function is retrieval rather than full prompt representation (Liang et al., 25 Aug 2025).

Taken together, these results suggest that depth boundaries are most effective when they preserve early and middle-layer context construction, but reduce or restructure prompt participation once token importance, prompt selection, or representation trajectories have stabilized. This is an inference from multiple papers rather than a single unified theorem.

6. Distinctions, limitations, and open problems

A recurring misconception is to treat any finer-grained prefill method as layer-segmented. The literature is more specific. Chunked prefill is token-axis segmentation, not depth segmentation (Shi et al., 9 Jul 2025). Kairos deflects prefill to decode nodes as adaptive token-chunked prefill steps interleaved with decode; it is segmented in time and by tokens, not by layers (Arun et al., 2 Jul 2026). Prefill-as-a-Service moves prefill across clusters and transfers the resulting KV cache at the phase boundary; its one layer-related remark is “layer-wise prefill pipelining,” but it does not define a layer-partitioned prefill architecture (Qin et al., 16 Apr 2026). FlexNPU provides phase-aware NPU virtualization and dynamic PD co-location, but the paper is explicit that it does not present a layer-aware or layer-segment-aware scheduler (Gu et al., 3 Jun 2026).

Even among direct depth-based methods, the constraints differ sharply. POP’s segmentation is static, depends on calibration data, and does not reduce peak model memory because decode still uses the full model (He et al., 3 Feb 2026). SPEED’s strongest results rely on SPEED-aware training or adaptation; the paper shows that PostHoc-SPEED creates train/test mismatch and degrades more sharply at aggressive cutoffs (Oh et al., 7 May 2026). FastKV’s segmentation boundary is useful because later-layer importance stabilizes, but the paper still requires offline calibration of the TSP layer and does not offer a universal choice across models (Jo et al., 3 Feb 2025).

Open problems remain concentrated around intermediate state management. Nexus notes that a true layer-segmented prefill system would likely need kernel restructuring, explicit pipeline boundaries inside the transformer, or intermediate activation buffering between layer segments, none of which its phase-level resource partitioning addresses (Shi et al., 9 Jul 2025). ContiguousKV leaves open how to define a stable interface for layer-state partitioning rather than token-region selection (Zou et al., 20 Jan 2026). CLAA shows that cross-layer evidence is better than trusting one pruning layer, but it does not answer whether different layer segments should use different retained token subsets or how often re-ranking should occur (McDanel et al., 17 Feb 2026).

A further open axis is layer-wise KV precision or placement. SpectrumKV is not a layer-segmented prefill method; it keeps the full layer structure intact and assigns each token a precision tier. But its layer-wise propagation analysis,

$1.82\times$ 2

and its per-layer INT4 cosine variation suggest that future systems may want to combine depth segmentation with per-layer precision allocation rather than treating the layer and token axes independently (Pengju, 7 Jun 2026).

The field therefore contains both direct instances and adjacent precursors. Direct methods show that prefill can be segmented by layer depth for execution, visibility, or serving schedule. Adjacent methods show that layers differ in importance stability, sparsity structure, retrieval utility, and error sensitivity. The remaining challenge is to integrate those observations into unified systems that choose depth boundaries, manage intermediate activations, and co-design segmentation with KV transfer, fused kernels, paging, and online schedulers.