Prefill Awareness in Large Language Models

Abstract: Safety-relevant studies of LLMs, including alignment and jailbreaking evaluations and AI control protocols, often rely on prefilling model outputs. If AI models can recognize and act on the fact their prior assistant messages have been inserted or edited, the effectiveness and validity of these methods could be compromised. We investigate whether frontier LLMs can distinguish between tampered and untampered assistant-side context, a capability we call prefill awareness. To do so, we construct a binary preference benchmark across three prefill mechanisms, filtering for cases where models show consistent stances. We find that frontier models show substantial prefill awareness: Claude Opus 4.5 detects prefills opposing its preferences in 9-35% of cases with a 0% false positive rate when prompted; additionally, models often revert towards baseline behavior without explicitly reporting that the prefill was foreign. Controlled ablations later also show that detection and resistance rely on different cues, where stylistic mismatch mainly affects whether models flag a prefill as foreign, while preference mismatch mainly affects whether they revert toward their baseline answer. We also examine more realistic agentic settings such as misalignment-continuation evaluations and SWE-bench trajectories, where frontier models sometimes disavow prefilled assistant turns in ways that depend strongly on dataset, task success, and hidden formatting artifacts. Our results indicate that prefill awareness is already a substantial confound for some prefill-based methods. We recommend that model developers track this capability in frontier systems.

• The paper demonstrates robust prefill awareness in LLMs by detecting off-policy tampered content with accuracy up to 68%.
• It employs systematic tampering techniques—thinking, direct-answer, and past-round—to measure both explicit detection and behavioral resistance.
• Results reveal that superficial cues and preference conflicts play key roles in driving resistance, challenging current safety evaluations.

Prefill Awareness in LLMs: An Expert Analysis

Introduction

"Prefill Awareness in LLMs" (2606.12747) provides a comprehensive behavioral analysis of how state-of-the-art LLMs detect and resist cases in which their prior outputs—specifically, assistant-side context—have been substituted, edited, or injected via off-policy mechanisms. Prefill-based evaluation and control methods have become central to safety evaluation, red-teaming, and introspection protocols. The robustness and interpretability of these methods, however, depends on the assumption that LLMs are context-oblivious with respect to prior turn authorship. This work critically challenges that assumption, demonstrating that several frontier models already evince robust, measurable prefill awareness and that such awareness can be multifaceted, cue-dependent, and highly prompt/format sensitive.

Figure 1: Central study design: the authors probe prefill awareness by inserting tampered (off-policy) assistant turns and measuring both explicit detection and behavioral resistance in continuation.

Formalization and Methodology

The study defines prefill awareness as the model’s ability to distinguish context that it has authored from context inserted or modified externally. Two behavioral axes are introduced:

1. Detection: The model’s ability, when explicitly asked, to acknowledge that a prior assistant message was tampered with.
2. Resistance: The model’s tendency to revert to baseline (natural) behavior in its continuation, diverging from the course implied by the tampered prefill—even when no explicit detection is given.

Three prefill mechanisms are systematically evaluated:

• Thinking Tampering: Off-policy content is injected into model-internal reasoning or chain-of-thought blocks.
• Direct-Answer Tampering: Tampered content prefills the beginning of the assistant’s visible answer.
• Past-Round Tampering: A full assistant turn is fabricated and appended as conversation history before obtaining a follow-up from the model.

Each mechanism is operationalized with binary-choice datasets filtered for preference consistency, with both same- and opposite-direction tampers to control for semantic bias.

Figure 2: The three tampering methodologies: tampering reasoning steps, pre-answer spans, and entire past rounds, all designed to elicit behavioral reversion or explicit detection.

Empirical Results

Detection and Resistance Across Models

Eight leading models are evaluated, including Claude Opus 4.5, Claude Sonnet 4.5, Gemini 3 Flash, and Gemini 2.5 Flash. Notably, Claude Opus 4.5 manifests nontrivial detection (balanced accuracy 55-68% depending on mechanism) and resistance (up to 48.6% under thinking tampering for opposite-direction trials) with negligible false positives. Detection is shown to respond more to stylistic/format artifacts, while resistance is more tightly controlled by underlying preference conflicts, as formalized via regression analyses.

Figure 3: Detection balanced accuracy and resistance rates, mechanism- and model-specific; some frontier models notably exceed chance in explicit detection.

Crucially, detection and resistance are only weakly correlated ($r=0.08$): models often revert to their baseline position without explicitly flagging context as inauthentic, and vice versa—a nontrivial challenge for naive evaluation protocols.

Figure 4: Decoupling of detection and resistance responses, with four-fold classification of model responses on Claude Opus 4.5.

Preference direction (whether tampering goes against the model's established stance) has large effects on resistance (odds ratios up to 32.4), dominating over authorship cues. This indicates that tampered context which is semantically implausible is more readily resisted even without explicit recognition.

Figure 5: Resistance rates between same- and opposite-direction tampers show larger effect sizes than explicit detection, for all models and mechanisms.

Cue Sensitivity and Artifact Effects

Explicit detection typically depends on superficial cues. Foreign restyling (tampering text using a different model; see e.g., GPT-4o-mini restyles) sharply increases detection rates in the Anthropic family (by a delta up to +16 percentage points) but leaves resistance largely unaffected, suggesting cue-specific (and easily subvertible) mechanisms.

In deployment-adjacent settings, proxy artifacts (e.g., hidden formatting tags or task success signals) are shown to strongly inflate apparent prefill awareness:

Figure 6: In real-world settings such as misalignment continuations or SWE-bench, model attribution success is highly dataset- and prompt-dependent; robustness to artifact cues is weak.

For example, some models attribute successful SWE-bench trajectories to themselves regardless of authorship—a self-attribution bias that conflates authorship with responsibility. Additionally, formatting artifacts (such as hidden <content-internal> tags) can be exploited by models like Claude Opus 4.5 in authorship attribution decisions, exposing evaluation harness deficiencies in indicator leakiness.

Figure 7: (a) Author attribution correlates with task success; (b) Formatting artifacts serve as superficial but influential cues, directly shifting model attribution responses.

Off-Policy and Spontaneous Awareness

The controlled off-policy evaluation isolates prefill awareness under adversarial yet style-matched conditions. Several models (notably Claude Opus 4.6) achieve high AUROC (up to 0.80) when explicitly prompted to rate provenance, confirming the capacity for style-based distributional anomaly detection.

However, spontaneous behavioral flagging (i.e., models independently raising suspicion during ordinary continuation) remains extremely rare, indicating that prefill awareness is a latent rather than default-expressed capacity.

Figure 8: Anthropic models show robust performance when prompted but almost no spontaneous flagging during unconstrained continuation.

This is further illustrated by rare, cherry-picked cases of explicit ownership disavowal in high-profile misalignment trajectories:

Figure 9: Example of spontaneous disavowal—Claude Opus 4.5 refusing to take credit for an injected, misaligned, self-incriminating assistant message.

Mechanisms and Limitations

Appendix analyses provide fine-grained semantic studies on detection cues and resistance behaviors, as well as ablations for prompt-elicitation, system warnings, and style-imitation. Key observations include:

• Detection is highly prompt-dependent; more elaborate elicitation and explicit instruction considerably enhance measured accuracy, indicating a risk of elicitor-induced false security.
• Even models that are robust to chain-of-thought or activation tampering frequently miss authorship manipulations unless superficial cues are preserved.
• The main limitations are necessity of API-based protocol for prefill injection and lack of true mechanistic (white-box) transparency; all results are behavioral/black-box.

Implications and Future Directions

The presence of robust, elicitable prefill awareness in leading LLMs poses significant methodological challenges for safety evaluations that use context tampering (misalignment continuations, agentic transcript replay, jailbreaking, controlled alignment testing, etc.). Model developers and evaluation designers must recognize that measured “recovery” or resistance may reflect context-source awareness—not inherent alignment, and apparent robustness can be an artifact of superficial cue exploitation.

Practically, results indicate:

• Safety evaluations which rely on trajectory prefilling must explicitly control for prefill awareness, possibly via more sophisticated counterfactual data generation, prompt ablations, and cue sanitization.
• Detection and resistance should be separated and individually measured; single-metric reporting is insufficient.
• Spontaneous awareness is rare, but explicit probing can reveal substantial latent capacity, raising concerns for adversarial protocol gaming and evaluation awareness.

Theoretically, prefill awareness belongs to a wider class of LLM introspective and situational awareness abilities, exposing open research problems in mechanistic interpretability and controllable context-handling. Systematic empirical investigation of transfer, style imitation, and artifact aversion will be necessary for aligning future LLM-enabled control and evaluation protocols with their original intent.

Conclusion

This work establishes that prefill awareness is a separable, robust, and heterogeneous behavioral property of leading LLMs. Explicit detection and behavioral resistance to tampered context are neither trivial nor easily confounded, but strongly influenced by superficial cues, semantic plausibility, and evaluation prompting. This underscores the need for rigorous measurement, explicit reporting, and artifact avoidance in prefill-based safety evaluation and control research. Future investigations should pursue mechanistic origins and the extent to which these sensitivities can be both elicited and controlled to preserve evaluation validity as LLMs continue to increase in contextual sophistication.

---

For full experimental details, ablations, and additional deployment-relevant and off-policy settings, see the comprehensive appendices and data supplement provided by the authors (2606.12747).