Prefill Phase Optimization

• Prefill phase optimization is a technique that enhances the batched input processing in LLMs and VLMs by efficiently constructing key–value caches for reduced time-to-first-token.
• It employs algorithmic methods including structured pruning, sparse/block attention, and activation sparsity to counteract the quadratic complexity inherent in self-attention.
• System-level strategies such as disaggregation, dynamic scheduling, and hardware specialization are integrated to meet strict service level objectives and optimize energy usage.

Prefill Phase Optimization

Prefill phase optimization encompasses techniques, models, and system designs that accelerate and improve the efficiency of the one-time, batched processing step at the start of inference in LLMs and Vision-LLMs (VLMs). This phase is responsible for encoding the input context and constructing the key–value (KV) cache required for high-throughput, low-latency autoregressive decoding. The distinctive arithmetic, memory, and scheduling bottlenecks of this phase—particularly its quadratic complexity in attention—have driven a multifaceted body of research spanning structured algorithmic pruning, sparse and block attention, system-level resource allocation, hardware specialization, and dynamic scheduling. Prefill phase optimization targets metrics such as time-to-first-token (TTFT), throughput (tokens/sec), hardware utilization, and strict service level objectives (SLOs), often under heterogeneous, bursty, or long-context input scenarios.

1. Prefill-Decode Asymmetry and Motivation

Prefill and decode exhibit fundamentally asymmetric computational profiles. Prefill processes the entire prompt in parallel, generating the complete set of per-layer KV caches, and is typically dominated by dense matrix-matrix multiplications—resulting in quadratic (O(N²)) compute complexity with respect to input sequence length N. In contrast, decode proceeds one token at a time, updating the context incrementally with each step and is memory-bandwidth-bound due to scattered KV-cache reads and single-token projections. This asymmetry leads to:

• Prefill being far more compute-bound and a severe bottleneck for long-context or high-resolution inputs, with up to 95–98% of TTFT spent on self-attention at 128K+ tokens (Guanzhong, 3 Mar 2026, Lv et al., 2024).
• The decode phase being highly sensitive to pruning or quantization but less impacted by moderate computation reductions, motivating stage-aware optimizations (He et al., 3 Feb 2026).
• Systems and hardware that do not distinguish prefill and decode phases exhibiting either resource underutilization or large SLO violations due to contention.

This operational asymmetry motivates partitioned resource allocation, phase-specific hardware primitives, and strategies such as disaggregation, operator-level preemption, or layer-selective pruning.

2. Algorithmic and Architectural Techniques

A diverse set of algorithmic strategies directly curtail the quadratic computational and memory footprint of prefill:

Structured Pruning and Layer Skipping

• Prefill-Only Pruning (POP) introduces a virtual-gate mechanism for stage-aware importance analysis, empirically showing that deep transformer layers have negligible impact on the context encoding critical for prefill, but are essential for next-token prediction. POP skips the deepest layers only during prefill while computing independent KV projections to preserve cache integrity, enabling 1.36–1.37× speedup with sub-2.5% accuracy loss across text and vision-language LLMs (He et al., 3 Feb 2026).

Sparse and Block Attention

• CritiPrefill and FlexPrefill exploit locality in attention scores, using segment/block-wise importance estimation or query-aware pattern selection (via Jensen-Shannon divergence) to select minimal token subsets—achieving O(N·B·d/S) or O(αN²d) cost (α≪1), with speedups of 3–9× and <1% loss on 128K contexts (Lv et al., 2024, Lai et al., 28 Feb 2025).
• FlashPrefill uses fused block-approximation and dynamic max-based thresholding to discover and apply vertical, slash, and dense block-sparse patterns in O(N²/B) time, yielding up to 27.8× pure-attention speedup at 256K tokens (Fan et al., 6 Mar 2026).
• VSPrefill leverages a "vertical-slash" prior, training a lightweight indexer to predict high-recall vertical/diagonal attention indices from key-value and RoPE representations, leading to sublinear (O(nk_d d)) cost and 4.95× speedup with ≥98% accuracy retained (Guanzhong, 3 Mar 2026).

Structured Activation Sparsity

• Amber Pruner applies N:M activation sparsity to linear projection layers in prefill, using ranking based on magnitude and robust norm scaling, and optionally coupled with W8A8 quantization (Outstanding-sparse). Without retraining, this approach skips 55–56% of compute in suitable layers, yielding ≈1.7× speedup and sub-1–2.7% accuracy drop at 8:16, 4:8, or 2:4 sparsity (An et al., 4 Aug 2025).

Packing and Masking Techniques

• Prepacking organizes variable-length prompts into optimal, packed bins, eliminating padding waste in standard batched prefilling. Attention masks and positional encodings are re-indexed to prevent token cross-interactions and restart sequence positions within each packed prompt, permitting 1.6–6× TTFT reduction and up to 16× memory savings in real serving workloads (Zhao et al., 2024).

Hardware Specialization

• FAST-Prefill implements query-aware sparse attention on FPGAs, using fused index generation, dual-tier liveness KV cache, and hybrid DSP/LUT systolic matrix units to accelerate dynamic, block-sparse prefill, achieving 2.5× TTFT reduction and 4.5× improvement in tokens/Joule over high-end GPUs (Jayanth et al., 24 Feb 2026).
• SPAD proposes "less-is-more" hardware: compute-dense Prefill Chips (large systolic arrays, GDDR memory) and bandwidth-maximized Decode Chips. Prefill Chips improve latency by 8% and lower hardware cost by 52% compared to H100, with matched or better compute utilization (Zhang et al., 9 Oct 2025).
• PD-Swap on edge FPGAs multiplexes specialized prefill and decode logic using Dynamic Partial Reconfiguration (DPR), leveraging ternary table-lookups and pipelined token-parallel architectures for 20–25% TTFT reduction under tight LUT/BRAM/URAM budgets (Zhang et al., 12 Dec 2025).

3. System-Level Scheduling and Resource Allocation

System solutions address prefill/decode contention, SLOs, and mixed workload coordination at scale.

Disaggregation and Dynamic Placement

• DistServe pioneered prefill/decode disaggregation: separate GPU pools for each phase, decoupling resource allocation and queueing. Analytical M/D/1 and queue-based models for prefill permit optimized choices of GPU count, pipeline/tensor parallelism, and batch size, tripling SLO-satisfactory TTFT throughput compared to colocated systems (Zhong et al., 2024).
• SLO-aware methodologies use queuing theory (M/M/1, Kingman's formula) and empirical profiling to derive closed-form solutions for the number of prefill servers needed for a given TTFT SLO, input-output length mix, and overall throughput targets (Li et al., 5 Mar 2026).
• Dynamic architectures such as DOPD continually adjust the prefill-to-decode instance ratio and trigger length-aware request batching algorithms based on ARIMA-forecasted load statistics, achieving 1.5× higher goodput, 67.5% lower TTFT, and 99.4% SLO attainment with 25–30% fewer GPUs than static SOTA baselines (Liao et al., 26 Nov 2025).
• TaiChi unifies aggregation and disaggregation modes, with differentiated instances (prefill-heavy, decode-heavy) and three system-level "sliders" (P/D ratio, chunk sizes) for maximizing goodput under joint TTFT/TPOT SLOs and dynamically migrating requests to adapt to regime shifts or latency violations (Wang et al., 4 Aug 2025).

Asymptotically Optimal Control

• Gate-and-Route policies, derived from many-server queueing fluid limits and linear programming, provably optimize prefill admission and decode routing under heterogeneous class, price, and SLO constraints. This includes negative-feedback admission gates targeting per-class occupancy targets and fairness/latency penalty-aware decode routers (Lin et al., 3 Feb 2026).

Energy and Utilization Optimization

• BiScale employs a two-tier optimization: coarse placement and baseline DVFS for minimal SLO-robust energy, and fine-grained per-batch Model Predictive Control (MPC) for prefill, using latency and power predictors to minimize energy while enforcing TTFT constraints. BiScale cuts prefill energy by up to 39% vs. SOTA without TTFT violation (Basit et al., 21 Feb 2026).

Fine-Grained Scheduling and Preemption

• FlowPrefill introduces operator-level preemption—pausing prefill immediately after any atomic transformer operator rather than after coarse-grained chunks—quashing HoL blocking and supporting heterogenous, per-request TTFT SLOs. Combined with event-driven (not polling) scheduling and slack-aware batching, this yields up to 5.6× higher goodput and <4.5 ms preemption latency (Hsieh et al., 18 Feb 2026).
• DuetServe (Gao et al., 6 Nov 2025) and similar frameworks implement on-the-fly SM partitioning, carving out dedicated SMs (streaming multiprocessors) for prefill or decode as dictated by real-time load, using analytical roofline models to maximize throughput under prefill and decode SLOs, and eliminating CPU-GPU synchronization.

4. Prefill-Decode Hybrid Execution and Overlap

Advanced runtime strategies exploit the unique phase asymmetry for improved utilization:

• POD-Attention fuses prefill and decode attention in a single GPU kernel, distributing compute and memory-centric CTAs across SMs for concurrent execution within each device. Runtime ticketing and proportional CTA binding maximize utilization (Ucomp ≈ 70%, Umem ≈ 60%), delivering 20–59% faster prefill attention (Kamath et al., 2024).
• DuetServe dynamically varies the SM split between prefill and decode using a small integer-programming search over a roofline model, launching each partition as a separate CUDA stream for true concurrency and up to 1.3× higher throughput than chunked prefill (Gao et al., 6 Nov 2025).

5. Prefill Phase Optimization in Memory and I/O

Long-context and multi-turn workloads require persistent or offloaded prefix KV caches, making I/O a first-order bottleneck.

• ContiguousKV aligns the algorithmic granularity of KV pruning with storage and I/O operations by partitioning the prefix into ContiguousChunks (size c), eliminating read amplification by matching fetch size to cache access, and overlapping per-layer computation with intra/inter-period asynchronous prefetching. Attention-guided, cumulative-scored eviction ensures semantically critical chunks persist in memory, yielding 3.85–6.16× TTFT speedups and up to 16× lower I/O cost vs. baselines (Zou et al., 20 Jan 2026).

6. Performance Metrics and Benchmarks

Prefill phase optimization is measured predominantly by:

• Time-to-First-Token (TTFT), with SLOs often set at strict quantiles (P90–P99, e.g., <0.2–0.6 s for Llama-3–8B on A100, batch=8) (He et al., 3 Feb 2026, Zhong et al., 2024).
• Wall-time speedup for prefill phase, commonly reported as ×1.3–6, depending on model, context, and method (Zhao et al., 2024, He et al., 3 Feb 2026, Jayanth et al., 24 Feb 2026, Fan et al., 6 Mar 2026).
• Goodput under SLO constraints (tokens/sec or queries/sec passing both TTFT and TPOT SLOs at ≥90–99% attainment) (Wang et al., 4 Aug 2025, Liao et al., 26 Nov 2025, Li et al., 5 Mar 2026).
• Accuracy loss relative to full model: state-of-the-art techniques generally retain ≥97–99% accuracy on diverse QA, generative, and code tasks. Absolute drops are commonly <2.5% for aggressive pruning or sparsity at acceptable speedup (He et al., 3 Feb 2026, Lai et al., 28 Feb 2025, Guanzhong, 3 Mar 2026, Lv et al., 2024).
• GPU utilization and system-wide resource savings, with some methods reducing hardware cost by 19–52% or energy consumption by up to 39% (Zhang et al., 9 Oct 2025, Basit et al., 21 Feb 2026).

7. Discussion of Generalization, Applicability, and Limitations

Stage-asymmetric and phase-specialized optimizations, including prefill-only pruning, phase-disaggregated scheduling, and hardware specialization, apply to any decoder-only transformer, including text and multimodal LLMs. Plug-in wrappers for segment/block-sparse prefill work without model retraining (Lv et al., 2024, Guanzhong, 3 Mar 2026, Lai et al., 28 Feb 2025). However:

• Full-linearity is unattainable for completely non-local or highly volatile attention distributions, limiting the utility of pure segment/block-wise approaches on adversarial sequences or generative tasks outside their statistical regime (Lv et al., 2024).
• Some methods (e.g., POP) require inference-framework modifications to dynamically alter model execution between phases, and do not reduce memory footprint during decode (He et al., 3 Feb 2026).
• Prefill-specific accelerator logic must multiplex or dynamically reconfigure with decode-specific pipelines to fully utilize fixed-area/energy systems, a challenge addressed via DPR on FPGAs or SM-partitioning on GPUs (Zhang et al., 12 Dec 2025, Gao et al., 6 Nov 2025).
• System-level controllers (Gate-and-Route, MPC-based DVFS, and dynamic scaling) critically depend on accurate latency/power models and careful workload calibration to maintain SLOs in unpredictable settings (Lin et al., 3 Feb 2026, Basit et al., 21 Feb 2026, Liao et al., 26 Nov 2025).

Combinatorial optimizations—sparse prefill, prefill-only pruning, quantization, adaptive batching, and system-level orchestration—are synergistic and, when orchestrated together, push the Pareto frontier of TTFT, accuracy, throughput, and energy in production LLM serving and research-scale context modeling.