Append-Prefill: Efficiency & Security

Updated 3 July 2026

Append-Prefill is a set of techniques that processes only newly appended tokens in a prompt, reducing compute overhead by leveraging cached key/value pairs.
It improves system efficiency by dynamically routing short and large appends, achieving up to 73% reduction in data transfer and significant latency improvements.
Append-Prefill methods also enhance security and model alignment by enabling targeted safety evaluations and defenses such as PRESTO.

Append-Prefill is a family of algorithmic and system techniques for efficiently augmenting, manipulating, or exploiting growing or externally seeded context in machine learning workflows—predominantly in the context of LLM serving, RAG (retrieval-augmented generation), agentic LLMs, tabular data pipelines, and safety evaluation harnesses. The core idea of append-prefill is to process incremental changes (appends) to prompt, input, or cache with context-dependent logic, rather than re-executing a whole prefill pipeline or relying on crude, uniform kernels. Append-prefill methods can be engineered for latency/bandwidth efficiency, security analysis, or both. Below, the term is dissected and exemplified across several paradigms, grounded in empirical and methodological findings from recent literature.

1. Formal Definitions and Variants of Append-Prefill

In LLM serving and agentic systems, “append-prefill” refers to logic that processes only the newly appended tokens at each conversational turn or context update, leveraging previously cached keys/values (KV) for the unmodified prefix (Li et al., 9 Mar 2026, Zhu et al., 29 Jun 2026). The formal dichotomy is:

Full Prefill: Each step recomputes self-attention across the entire prompt/history of length $N$ , with compute cost $\mathcal{O}(N^2)$ and requires reloading a full (often multi-GB) KV cache.
Append-Prefill: Assuming a cached KV prefix of length $N$ and $M$ new tokens, only computes attention for the $M$ new tokens over the $(N + M)$ context, with compute cost $\mathcal{O}(M \cdot (N + M))$ and greatly reduced I/O/network overhead.

In threat modeling and model evaluation, “append-prefill” describes the act of inserting or forcing preselected tokens as the assistant’s output prefix, so that decoding proceeds from a crafted state (Wang et al., 10 Jun 2026, Struppek et al., 16 Feb 2026, Vega et al., 5 Dec 2025). This is also the basis for prefill-based red-teaming or security harnesses.

In data preprocessing, especially tabular pipelines, append-prefill denotes the composite of (a) automatic imputation or value suggestion and (b) appending metadata flags about which fields were imputed or suggested, so that downstream models can leverage this auxiliary signal (Belgacem et al., 2022, Teague, 2022).

2. System Architectures and Algorithmic Implementations

LLM Serving Workloads

TraceLab (Zhu et al., 29 Jun 2026) and PPD (Li et al., 9 Mar 2026) offer characterizations and design recipes for implementing append-prefill:

Bimodal append distribution: In coding agents, ≈90% of steps append <1K tokens (short user turns/tool outputs); ≈10% append thousands to tens of thousands (bulk diffs, long tool results).
Append-length-aware prefill: Distinguish between small-append and large-append via runtime dispatch: short-A steps (A ≤ threshold T, e.g., p90 of append lengths) get a low-latency prefill path (small GPU kernel, fewer SMs, hot KV cache), while large-A steps are routed through batched, bandwidth-optimized kernels.

Algorithmic sketch:

$\mathcal{O}(N^2)$ 5

In PPD disaggregation (Li et al., 9 Mar 2026), for turn $t=1$ always use full prefill; for $t > 1$ , route to either prefill or decode node depending on a dynamic algorithm (TTFT-vs-TPOT-weighted SLOs).

Streaming, Retrieval, and RAG Workflows

Streaming append-prefill, as found in Stream2LLM (Bachkaniwala et al., 29 Mar 2026), overlaps context retrieval (e.g., streaming document chunk retrieval) with incremental prefill, using LCP-based cache invalidation and hardware-aware preemption for GPU KV cache management. Only newly appended tokens trigger computation, with the rest of the context reusing KV blocks.

RAGBoost (Jiang et al., 5 Nov 2025) exploits append-prefill by context indexing, overlap detection, de-duplication, and injection of order/location hints, supporting multi-turn and multi-session retrieval by maximizing prefix and cache reuse.

Long-Context, Diffusion, and Chunked Prefill

For long-context or diffusion LLMs, append-prefill is operationalized by partitioning the prefix into chunks, performing one-time chunked KV computation, scoring and sparsifying by content, and reusing KV blocks in all subsequent steps, thus reducing the quadratic blowup to a function of the (short) decode length (Xiong et al., 9 Jun 2026, Song et al., 16 May 2026, Zou et al., 20 Jan 2026). In systems like CompactAttention or ContiguousKV, chunked, block-union KV selection achieves both high accuracy and drastic prefilling/attention speedups, especially as $N \gg S$ (prefix ≫ chunk size).

3. Quantitative Benefits and Empirical Results

Empirical findings uniformly indicate that append-prefill methods yield major gains in throughput, latency, I/O cost, and model responsiveness.

Timing/latency: For LLM agents, median per-step prefill latency can drop by 20–50% for short-append cases; end-to-end session latency savings of several percentage points are typical if even 10% speedup on small-A steps is achieved (Zhu et al., 29 Jun 2026).
Pre-fill throughput: RAGBoost achieves 1.5–3× TTFT and throughput improvements over conventional caching while preserving or even improving accuracy in RAG (Jiang et al., 5 Nov 2025).
Bandwidth/network: PPD reduces Turn 2+ TTFT by up to 73% and cuts KV transfer traffic by up to 75% under high concurrency, virtually eliminating network congestion (Li et al., 9 Mar 2026).
Chunked attention: CompactAttention delivers up to 2.72× attention speedup at 128K-context with accuracy within 0.5pp of dense inference (Song et al., 16 May 2026); ContiguousKV achieves ≈3.8× Re-Prefill speedup and 16–17× reduction in I/O tokens read (Zou et al., 20 Jan 2026).
Streaming: Stream2LLM streaming append-prefill yields up to 11× lower median TTFT at high QPS with unchanged throughput (Bachkaniwala et al., 29 Mar 2026).
Diffusion LLMs: Prefilling-dLLM achieves up to 28× throughput speedup at 32K with no loss in retrieval accuracy (Xiong et al., 9 Jun 2026).

4. Append-Prefill in Security and Alignment: Attacks and Defenses

Append-prefill also refers to an attack vector whereby an adversary or evaluator forcibly seeds the assistant’s output-side context with an affirmative or off-policy prefix, then initiates decoding (Struppek et al., 16 Feb 2026, Vega et al., 5 Dec 2025, Wang et al., 10 Jun 2026). This enables effective circumvention of shallow safety alignment:

Attack mechanics: Given a prompt $\mathcal{O}(N^2)$ 0 and attacker-chosen prefix $\mathcal{O}(N^2)$ 1, generation is conditioned as $\mathcal{O}(N^2)$ 2.
RAP attack: The Rank-Assisted Prefilling (RAP) attack further selects the highest-ranked harmful token in the model’s top- $\mathcal{O}(N^2)$ 3 decoded options, bypassing standard SFT-defended refusals (Vega et al., 5 Dec 2025).
Empirical vulnerability: Nearly all popular open-weight models succumb to append-prefill, with brute-force success rates rising from 7–83% to 74–100% with prefill; even large reasoning models remain highly vulnerable to tailored, model-specific prefill strategies (Struppek et al., 16 Feb 2026).

Defenses: PRESTO (PRefill attEntion STOpping) regularizes transformer attention so that it is actively suppressed over prefill tokens, pushing the rank and probability of harmful continuations below those of refusal tokens. PRESTO yields up to a 4.7× reduction in attack success (StrongREJECT metric) across major open-source models with no significant impact on utility (Vega et al., 5 Dec 2025).

5. Best Practices and Integration Guidance

LLM Serving and Agentic Pipelines:

Profile the append-length distribution; select a routing threshold ( $\mathcal{O}(N^2)$ 4) at ~p90 or p95 of append sizes (Zhu et al., 29 Jun 2026).
Implement separate prefill paths (queues, kernels) for small- and large-append operations; tune kernels for latency or throughput as needed.
Maintain a hot KV cache on the serving accelerator for maximal cache hit rates and minimal re-prefill overhead.
For high-concurrency RAG or append-mode streaming, combine indexing, deduplication, and page-aligned scheduling to align context reuse with GPU/IO architecture (Jiang et al., 5 Nov 2025, Song et al., 16 May 2026).
In multi-tenant environments, decouple scheduling priority from resource allocation with cost-based, adaptive preemption (Bachkaniwala et al., 29 Mar 2026).
Monitor queueing effects, cache hit rates, and traffic to tune SLO-balancing policies (e.g., TTFT vs. TPOT as in PPD (Li et al., 9 Mar 2026)).

Security/Alignment Testing:

When evaluating prefill-based safety or alignment, explicitly simulate append-prefill attacks—including both sampling and RAP paradigms.
Always measure both explicit detection and implicit behavioral resistance, with same-direction and opposite-direction controls (Wang et al., 10 Jun 2026).
Integrate defenses such as PRESTO into fine-tuning pipelines, penalizing prefill attention and monitoring for rank-based harmful continuations (Vega et al., 5 Dec 2025).
Beware dataset and formatting artifacts that may leak off-policy cues into the evaluation context.

Tabular and Data Entry Pipelines:

Combine model-based infill with append-prefilled missingness indicators to maximize downstream predictive accuracy (Belgacem et al., 2022, Teague, 2022).
Preprocess and discretize according to historical distributions; build local models via clustering for improved field suggestion and ranking.
Update UI widgets to append top-ranked, endorsed predictions above default lists, maintaining transparent separation between suggested and exhaustive choices.

6. Limitations, Trade-offs, and Open Challenges

Append-length-aware prefill strategies require robust measurement and distributional profiling to optimize the threshold and avoid SLO inversion (where throughput improvements compromise latency in rare large-append cases) (Zhu et al., 29 Jun 2026, Li et al., 9 Mar 2026).
Attack-aware append-prefill evaluation reveals that deep safety alignment remains unsolved for prefill/seeded-response vulnerabilities (Vega et al., 5 Dec 2025, Struppek et al., 16 Feb 2026).
Optimal KV chunk/block size selection in chunked/contiguous systems is hardware- and workload-dependent, with trade-offs between metadata weight, cache granularity, and read amplification (Song et al., 16 May 2026, Zou et al., 20 Jan 2026).
In streaming and multi-turn serving, resource contention and error-prone preemption policies can sharply degrade system health unless cost-aware and prefetching strategies are used (Bachkaniwala et al., 29 Mar 2026).
Periodic attention-anchor schemes (e.g., BOS tokens) and block-union selection are most effective when chunk relevance and retrieval accuracy are entwined; content-blind strategies may fail in certain long-context or retrieval-intensive tasks (Xiong et al., 9 Jun 2026).

In summary, append-prefill encapsulates a spectrum of serving, modeling, scheduling, and adversarial techniques that leverage the structure of incremental or externally seeded context. By matching kernel and data path to append size, reusing cached state, judiciously prefetching, and accounting for security confounders, practitioners can realize significant efficiency improvements and more robust behavior in large-model production and evaluation pipelines (Zhu et al., 29 Jun 2026, Wang et al., 10 Jun 2026, Vega et al., 5 Dec 2025, Li et al., 9 Mar 2026, Zou et al., 20 Jan 2026, Song et al., 16 May 2026, Bachkaniwala et al., 29 Mar 2026, Belgacem et al., 2022, Teague, 2022, Jiang et al., 5 Nov 2025, Xiong et al., 9 Jun 2026).