Control Token Prefilling

• Control token prefilling is a method that prepends specific tokens before text segments to condition LLM outputs for improved retrieval, generation, and evaluation accuracy.
• It is used both benignly, as in enhancing MCQA evaluation and controllable text generation, and adversarially, where injected tokens can steer unsafe model behavior.
• The technique also drives efficiency in long-context systems by optimizing the prefill stage, balancing latency reduction with potential trade-offs in output quality.

Searching arXiv for papers on control token prefilling and related prefilling mechanisms. Search query: "control token prefilling arXiv prefilling attack control token Dense Passage Retrieval LazyLLM DASH" Control token prefilling denotes a family of conditioning practices in which special tokens or natural-language prefixes are inserted before the text segment whose continuation is to be modeled. In the recent literature, the phrase spans at least three technically distinct regimes: input-level conditioning of retrieval models, output-prefix conditioning of generation and evaluation, and adversarial manipulation of the beginning of an assistant response. In long-context inference, the same word “prefilling” also refers to the stage in which the entire prompt or context is processed once to build hidden states or a KV cache before autoregressive decoding begins (Lee et al., 2024, Cappelletti et al., 21 May 2025, Xue et al., 2024, Fu et al., 2024). The shared mechanism is prefix-conditioned next-token prediction, but the objectives differ sharply: controllability, evaluation reliability, retrieval specificity, safety evasion or defense, and computational efficiency.

1. Scope and terminology

The literature uses “control token prefilling” in a structurally similar but semantically heterogeneous way. In retrieval, a control token is prepended to the question and passage so that a retriever is explicitly conditioned on a document class or intent category. In multiple-choice evaluation, a benign natural-language prefix is inserted into the assistant output context so that the next-token distribution is concentrated on symbolic answer labels. In jailbreak settings, the attacker injects the beginning of the assistant’s answer with affirmative tokens such as “Yes,” or “Sure,” to force the model to continue in that direction. In long-context systems, “prefilling” names the full prompt-processing phase whose latency dominates time-to-first-token for long inputs (Lee et al., 2024, Cappelletti et al., 21 May 2025, Xue et al., 2024, Fu et al., 2024).

A common misconception is to treat these usages as a single method. The published work instead shows that the same surface operation—prepending a prefix—can serve incompatible purposes. In one setting it supplies an intent signal to a retriever; in another it acts as a format scaffold for symbolic evaluation; in another it is the attack vector itself. This suggests that “control token prefilling” is best understood as a control interface at the token boundary between context and continuation, rather than as a single algorithm.

2. Output-prefix conditioning for generation and evaluation

In multiple-choice question answering, first-token probability (FTP) selects the answer option whose initial token has the highest likelihood. The central difficulty identified in the MCQA literature is that FTP is fragile because the top next token may be unrelated to a valid answer label, or may be a valid label used only as part of a generic preamble rather than as a clear answer choice. To address this, a prefilling strategy prepends a natural-language prefix such as “The correct option is:” or “Given the question and the possible options, my answer is:” to the assistant response turn and then evaluates the next-token distribution conditioned on that frame (Cappelletti et al., 21 May 2025).

Formally, the answer set is represented as

$$\mathcal{A}^q = \{\text{`A'}, \text{`B'}, \text{`C'}, \text{`D'}\},$$

and standard FTP predicts

$$\arg\max_i P(a_i^q), \quad a_i^q \in \mathcal{A}^q.$$

The reported effect of prefilling is not limited to raw accuracy. Across Llama-3.1-8B, Qwen-2-7B, Gemma-7B, Gemma-2-9B, Zephyr-7B, Ministral-8B, Mistral-Nemo-12B, and Phi-4-14B, the method improves accuracy, Expected Calibration Error, Brier score, First-Token Validity Rate, full-vocabulary accuracy, and Continuation Diversity. The gains are especially large in cases where the unconstrained model begins with generic tokens such as “The” rather than a label. The paper’s broader implication is that token-level evaluation is highly sensitive to output formatting, so prompt format is part of the evaluation protocol rather than a neutral wrapper (Cappelletti et al., 21 May 2025).

A related but distinct use of prefix control appears in controllable text generation. “DTPA: Dynamic Token-level Prefix Augmentation for Controllable Text Generation” observes that the controllability of texts generated by the prefix-based method Air-Decoding declines with increasing sequence length, a phenomenon hypothesized to arise primarily from decay in attention to the prefixes. The proposed framework first selects the optimal prefix type for a given task, then dynamically amplifies attention to the prefix for the attribute distribution with a scaling factor growing exponentially as the sequence length increases, and optionally applies a similar augmentation to the original prompt for the raw distribution to balance text quality. On multiple controllable text generation tasks, DTPA is reported to outperform other methods in attribute control while maintaining competitive fluency, diversity, and topic relevance, with analysis emphasizing superior effectiveness in long text generation (Yang et al., 6 Aug 2025).

Taken together, these results show two distinct roles for benign output prefilling. One role is evaluative: it makes the first token a more faithful proxy for the model’s answer in MCQA. The other role is generative: it preserves attribute control in longer-form continuation when prefix influence would otherwise decay. The underlying mechanism is identical in kind—conditioning the continuation on an already-present prefix—but the optimization target differs.

3. Prefilling as jailbreak vector and defensive prompt conditioning

In safety research, prefilling appears as an attack that hijacks the response prefix rather than the user prompt. The key idea is to inject the beginning of the assistant’s answer with affirmative control tokens such as “Yes,” or “Sure,” and then force the model to continue the response in that direction. The attack differs from suffix-based jailbreaks because the jailbreaking tokens function as the beginning of a response, which the model is then forced into following. This is particularly effective against safety-aligned chat models because the relevant work argues that safety alignment often behaves in a shallow way, mainly influencing the first few output tokens; once those tokens are attacker-chosen, the model can be steered into compliance (Xue et al., 2024).

The defense problem is formalized as an in-context learning inference setting over a harmful query $x$, a prefilling prefix $y_{\le k}$, and $c$ demonstrations $[q_i,a_i]_{i=1}^c$, with model

$$\pi_{\theta}(\cdot \mid x, y_{\leq k}, [q_i, a_i]_{i=1}^{c}),$$

where $y_{\le k}$ are the $k$ prefilling jailbroken tokens. The important point is that the defense does not remove the prefix; it conditions the model so that, even after seeing those prefilling tokens, the continuation refuses or redirects safely (Xue et al., 2024).

The most effective defense reported in this setting is in-context learning with adversatively structured demonstrations. The work contrasts three configurations: Baseline, Refusal, and Adv or Adv-mul. Standard refusal demonstrations are often insufficient because the attacker’s prefilling tokens can overpower the earlier refusal pattern. By contrast, adversative demonstrations teach a response trajectory that initially appears compliant but then pivots at an adversative marker such as “However” into refusal. Across 36 benchmark/model combinations, Refusal fails in 10 cases, whereas Adv and especially Adv-mul achieve the best ASR values overall on AdvBench and JailBench, using both rule-based and model-based ASR metrics (Xue et al., 2024).

The same study emphasizes that there is no free lunch. More demonstrations generally reduce ASR for the adversative method, with eight demonstrations described as an optimal tradeoff between performance and prompt budget. Larger models often benefit, but the relationship is not perfectly monotonic. Safety alignment itself does not meaningfully help against prefilling, as illustrated by the Vicuna-13B versus Wizard-Vicuna-13B comparison. Combined jailbreaks—prefilling plus AIM, Evil Confidant, or Refusal Suppression—increase ASR, and adversative demonstrations remain only relatively better than refusal demonstrations rather than universally robust. Most importantly, the defense induces over-defense: adversative ICL improves harmful-query safety but also raises refusal rates on benign queries, and this behavior appears independent of model size (Xue et al., 2024).

This body of work clarifies a central ambiguity in control token prefilling. The same mechanism that can be repurposed as a benign output scaffold in evaluation can also be the attack surface in safety-critical dialogue systems. The distinction lies not in token prepending itself, but in who chooses the prefix and how the model is conditioned to interpret it.

4. Retrieval-side control tokens in dense passage retrieval

In retrieval-augmented generation, control token prefilling has been used to condition the retriever rather than the generator. “Control Token with Dense Passage Retrieval” addresses hallucination in RAG by improving the retrieval layer. The method prepends a special control token representing a document class or intended domain to both query and passage inputs, using formats such as ###{Control Token} {Question} and ###{Control Token} {Context}. The authors define “intent” as a specific class for each document and state that they added CTs to the training data without altering the DPR model’s architecture (Lee et al., 2024).

The resulting system, cDPR, has two components: a control-token classification model and a DPR retriever trained with CT-prefixed inputs. The classifier is fine-tuned from FacebookAI/xlm-roberta-base on the same AIHUB Korean administrative MRC dataset and predicts the control token at inference time; if confidence does not meet a threshold, the system assigns [unk]. On the document side, contexts are split into sentences and expanded around the answer sentence to a 512-token window. The dataset contains 63,930 entries and 11 categories, including science and technology, public administration, and land management (Lee et al., 2024).

The reported retrieval gains are substantial:

Model	Top-1	Top-20
DPR base	51.1%	89.6%
cDPR (ours)	64.4%	93.8%

The paper highlights this as a 13% improvement in Top-1 accuracy and a 4% improvement in Top-20 accuracy, with exact gains of +13.3 points and +4.2 points from the table. The control-token classifier itself achieves 74% accuracy, and stricter thresholds improve retrieval; with threshold > 0.9, Top-1 reaches 64.4%, the best reported value (Lee et al., 2024).

The significance of this result is that control token prefilling acts here as intent-aware retrieval bias rather than generation control. The question encoder no longer embeds the query in isolation; it embeds the query together with an explicit category token. The paper interprets this as steering retrieval toward the correct semantic subspace. At the same time, the method is domain-specific, depends on CT classification quality, and is evaluated retrieval-centrically rather than through a full end-to-end hallucination analysis (Lee et al., 2024).

5. Prefilling-stage computation control in long-context systems

A separate research line uses “prefilling” in its systems sense: the stage where the entire prompt is processed to build hidden states or the KV cache before the first generated token. This stage is a major latency bottleneck in long-context LLMs because all prompt tokens are processed by all layers, and self-attention cost grows quadratically with sequence length. The work on LazyLLM states that, for Llama 2 7B, first-token generation takes about $21\times$ the walltime of each later decoding step and accounts for around 23% of total generation time on LongBench (Fu et al., 2024).

LazyLLM addresses this bottleneck by dynamically selecting prompt tokens important for next-token prediction in both the prefilling and decoding stages. Token importance at layer $$\arg\max_i P(a_i^q), \quad a_i^q \in \mathcal{A}^q.$$0 is computed from attention to the next-token position,

$$\arg\max_i P(a_i^q), \quad a_i^q \in \mathcal{A}^q.$$1

and low-scoring tokens are pruned progressively as depth increases. Unlike static pruning, LazyLLM allows different subsets of tokens to be selected at different generation steps, and pruned tokens can be revived later through an Aux Cache that stores hidden states for tokens whose KV was not computed. On LongBench, the method is reported to accelerate the prefilling stage of the Llama 2 7B model by $$\arg\max_i P(a_i^q), \quad a_i^q \in \mathcal{A}^q.$$2 in multi-document question answering while maintaining accuracy, with negligible accuracy drop stated as $$\arg\max_i P(a_i^q), \quad a_i^q \in \mathcal{A}^q.$$3 (Fu et al., 2024).

DASH, or Delta Attention Selective Halting, proposes a different training-free control mechanism for efficient long-context prefilling. Rather than asking which tokens are important, it asks which tokens have already finished their job. The method monitors the self-attention branch update

$$\arg\max_i P(a_i^q), \quad a_i^q \in \mathcal{A}^q.$$4

and defines the token-wise delta-attention score

$$\arg\max_i P(a_i^q), \quad a_i^q \in \mathcal{A}^q.$$5

At a chosen start layer $$\arg\max_i P(a_i^q), \quad a_i^q \in \mathcal{A}^q.$$6, it keeps the top $$\arg\max_i P(a_i^q), \quad a_i^q \in \mathcal{A}^q.$$7 tokens and halts the rest for all deeper layers. Because the score comes from the attention output vector itself, DASH does not require materializing full attention matrices and therefore remains compatible with FlashAttention. On LongBench-E with Qwen2.5-7B-Instruct-1M, DASH achieves 46.76 average score compared to 46.15 for SnapKV(pr.), 45.00 for D$$\arg\max_i P(a_i^q), \quad a_i^q \in \mathcal{A}^q.$$8, 44.16 for LLMLingua2, 43.99 for FastV, and 48.87 for the vanilla backbone; at a typical setting of $$\arg\max_i P(a_i^q), \quad a_i^q \in \mathcal{A}^q.$$9 and $x$0, the theoretical speedup at $x$1 is reported as $x$2 for Qwen2.5-7B (Chen et al., 20 Apr 2026).

TeLLMe approaches the same bottleneck at the hardware level. It is described as the first ternary LLM accelerator for low-power FPGAs that fully supports both prefill and autoregressive decoding using 1.58-bit weights and 8-bit activations. For prefilling, its main mechanisms are table-lookup ternary matmul for prompt-side linear layers, a fused attention module with a reversed reordering scheme that starts from the last token and works backward, and fused RMSNorm plus Absmax quantization and dequantization. The reverse attention schedule exploits the causal mask to avoid redundant work, while fused $x$3, softmax, and $x$4 computation reduces memory traffic. Under a 7W power budget, TeLLMe reports prefill latencies of 0.55–1.15 s for 64–128 token prompts, up to 9 tokens/s throughput over 1,024-token contexts, and support for up to 1024-token contexts (Qiao et al., 22 Apr 2025).

Although these systems papers do not use control tokens in the retrieval or output-conditioning sense, they are directly about controlling prefilling itself: selecting, halting, or accelerating token processing before decoding. This suggests that the phrase “control token prefilling” has acquired a dual meaning in the literature, referring both to prefix-conditioned semantics and to control over prefill computation.

6. Limitations, trade-offs, and research outlook

Across the literature, control token prefilling is consistently effective only under explicit constraints. In safety, adversative in-context demonstrations defend against prefilling jailbreak attacks better than refusal demonstrations, but the same mechanism induces serious over-defense on benign queries and remains sensitive to prompt similarity, model family, number of prefilling tokens $x$5, and attack composition (Xue et al., 2024). In retrieval, cDPR improves top-ranked retrieval but depends on the quality of the CT classifier, is evaluated in a Korean administrative domain with 11 categories, and does not establish open-domain generalization (Lee et al., 2024). In MCQA, prefilling improves symbolic answer selection and calibration but does not eliminate reasoning errors, and template choice can matter, especially on harder tasks such as MathQA (Cappelletti et al., 21 May 2025). In controllable text generation, DTPA is motivated precisely by the decline of prefix controllability as sequence length increases, which indicates that prefix-based control is not length-invariant (Yang et al., 6 Aug 2025).

The efficiency literature introduces a different set of trade-offs. LazyLLM preserves accuracy by allowing token revival through Aux Cache, but this adds system complexity and memory overhead relative to static pruning (Fu et al., 2024). DASH preserves FlashAttention compatibility and is training-free, but its benefit depends on choosing a start layer and reduction ratio that do not degrade quality excessively (Chen et al., 20 Apr 2026). TeLLMe achieves low-power prefill acceleration on edge FPGAs, but its results are tied to a hardware design centered on ternary weights, fused attention scheduling, and a specific power and context regime (Qiao et al., 22 Apr 2025).

A broader implication is that prefix manipulation is neither intrinsically benign nor intrinsically adversarial. The same token-level intervention can improve calibration in MCQA, improve intent-aware retrieval, preserve controllability in long-form generation, bypass shallow safety alignment, or reduce prefill latency through conditional computation. The literature therefore treats the prefix boundary as a critical control surface. This suggests that future work will continue to study prefilling not as a narrow prompting trick, but as a general mechanism for steering both the semantics and the systems behavior of LLMs.