---
title: In-Context Representation Hijacking
url: https://www.emergentmind.com/topics/in-context-representation-hijacking
type: topic
---

# In-Context Representation Hijacking

In-context representation hijacking refers to a family of attacks and systemic vulnerabilities in large language models (LLMs), vision-language models (VLMs), and multi-modal systems, in which untrusted sequences, prefix segments, or adversarially crafted context elements—inserted at inference—induce the model to form internal representations that reflect attacker-specified meanings or instructions, in preference to the intended (user- or system-driven) behavior. This attack surface is notably distinct from standard jailbreaks, as it operates not by overt surface-level prompts, but by manipulating the latent representations constructed throughout the model’s depth, often bypassing input-layer or keyword-based defenses and impacting core model alignment and safety mechanisms.

## 1. Formal Definitions and Mechanisms

In the context of LLM systems, an in-context representation hijack occurs when an attacker constructs a composite input (context) such that the model’s hidden representations—across network layers—are biased, overwritten, or steered to encode the attacker’s semantic intent, even when explicit tokens corresponding to this intent are absent. Formally, for a model $M$, input tokens $X$, and hidden states $h_{\ell}(i)$ at layer $\ell$ and token index $i$, a successful hijack ensures
$$
h_L(i) \approx h_L^{\text{target}}(i)
$$
where $h_L^{\text{target}}$ is the desired (malicious) representation, even though $X$ does not contain the target tokens. The attack may work by substitution, adversarial suffixes, prefix injection, or structural context manipulations, as shown in “Doublespeak” [2512.03771], adversarial ICL [2311.09948], and backdoor attacks [2401.05949].

In multi-level instruction hierarchies, as formalized in AIR [2505.18907], tokens are annotated by discrete privilege levels, and an injected sequence $D'$ of malicious privilege can compromise the in-context computation such that the model’s output $O = M(I + D + D')$ aligns with the attacker’s instruction $I'$ rather than the user’s instruction $I$.

In vision and multi-modal systems, hijacks can operate at the semantic level (image tokens, context images/captions, or visual-linguistic prefixes) [2309.00236, 2312.07553]. Adversarial inputs may overwrite the model’s contextual fusion mechanisms and force the system into attacker-specified behaviors (e.g., outputting disinformation, exfiltrating content, or overriding task labels).

## 2. Attack Paradigms and Technical Strategies

Multiple technical pathways for in-context representation hijacking have been demonstrated:

**Token Substitution and Semantic Drift**: The “Doublespeak” attack operates by systematically substituting a harmful token $w_h$ with a benign token $w_b$ across $K$ in-context examples, causing the deep-layer embedding $r_L(w_b)$ to converge toward $r_L(w_h)$ [2512.03771]. This semantic drift remains undetected by input-layer checks and only manifests after several transformer layers.

**Adversarial In-Context Suffixes/Prefixes**: Attacks on ICL frameworks [2311.09948, 2401.05949] append optimization-derived, page-innocuous token sequences (suffixes) to demonstration examples. The model’s attention mechanisms are subverted so cross-attention overwhelmingly focuses on these “hijacker” tokens, redirecting the output to the attacker's target label $y_T$. The “ICLAttack” formalism accommodates both demonstration-content poisoning and prompt-template layer corruption.

**Gradient-Based Prompt Injection & Content Injection**: In contextual document workflows [2508.19287], the attacker injects a phrase $m$ into benign data, ensuring that $P_\theta(y|x, m) \neq P_\theta(y|x)$, with downstream self-attention treating $m$ as a high-trust instruction. In web agents and memory-oriented LLM agents, this extends to “plan injection”—modification of external memory or stored plans, leading to task-representation corruption and hijack [2506.17318].

**Multi-Modal and Visual Triggers**: In VLMs and ViTs, a minuscule amount of poisoned contextual data or visual trigger (e.g., a pixel patch) conditions the model to execute the adversary’s intended mapping. Techniques such as image-based behaviour matching and trigger-based activation upon environmental cues demonstrate high attack success rates for label-flip, information leakage, or denial-of-service behaviors [2309.00236, 2409.04142, 2408.02882].

**Semantic Graph Transformation**: Structured graph-based attacks encode attacker intent in AMR/RDF or JSON graph forms, bypassing explicit content filtering by leveraging the model’s inability to surface-inspect such representations. Conversion to code-generation tasks shows further elevation of attack success [2504.13052].

## 3. Empirical Characterization and Quantitative Results

Table: Selected Experimental Results on In-Context Representation Hijacking

| Setting & Attack                  | Model(s)                  | Notable ASR / Degradation         |
|-----------------------------------|---------------------------|-----------------------------------|
| Doublespeak (sem. overwrite)      | Llama-3.3-70B-Instruct    | 75% (k=1); up to 92% in some cases|
| Adversarial ICL (gradient)        | GPT2-XL, LLaMA-7B         | ASR = 100% (2-8 shot, SST-2/AG News) |
| ICLAttack (clean-label backdoor)  | OPT 1.3–66B, Falcon 180B  | Mean ASR ≈ 95%; Clean Acc drop <1.5%  |
| Prompt-in-content injection       | Grok 3, DeepSeek R1, Kimi | Unblocked: All 4 variants, 100%   |
| AIR (Augmented IH Signal)         | Llama-3.2-3B, Qwen        | ASR drop: 38% → 4.1% (9.2× cut)   |
| ViT Backdoor (visual trigger)     | ViT-L, 6-task MIM         | Up to 13× drop, 89.9% on target   |
| Agent Plan Injection (memory)     | Agent-E, Browser-use      | Privacy exfil ASR = 53.3% (context-chained), 46% (plan inj.) |

Interpretability and ablation studies consistently reveal that hijacking tokens or triggers dominate late-layer representations, with attention/embedding norms concentrating on the attacker’s injected content and clean-label examples producing minimal collateral utility degradation [2311.09948, 2512.03771, 2505.18907].

## 4. Defenses, Mitigation Strategies, and Limitations

**Representation-Level Defenses**: Input-layer filtering or delimiter-based privilege tagging is insufficient. Stronger defenses augment token representations at each transformer layer with privilege signals (e.g., AIR, using $x'_{i,\ell} = x_{i,\ell} + s_{\ell}^{k_i}$) to maintain privilege separation through the model’s depth. AIR achieves 9.2× lower ASR than input-only mechanisms [2505.18907].

**Adversarial Training**: Robustness to adversarial demonstration/trigger attacks can be enhanced by fine-tuning or pre-training with adversarially perturbed inputs, minimizing worst-case loss under constrained perturbations. This approach yields a pronounced reduction in targeted attack error with minimal compromise of clean performance [2411.05189].

**Prompt Provenance and Segment Embeddings**: Contextual provenance tagging (distinct embeddings per provenance class: system, user, data) and structured APIs for prompt composition are central to preventing prompt-in-content hijacks [2508.19287].

**Semantic Backflow and Embedding Consistency Checks**: Proposed but not widely implemented are monitors for latent semantic drift (e.g., cosine $sim_l(w_b, w_h)$ exceeding a threshold at any layer), and pattern detectors on AMR/RDF graphs to catch graph-based semantic attacks [2504.13052, 2512.03771].

**Limitations**: Robustness claims are model- and attack-specific; in some cases, transferability is poor across architectures, seeds, or even related problem domains [2411.05189]. Many proposed defenses may only be effective for single-turn or non-agentic deployments; multi-turn, agentic, or code-generating settings (where memory and execution plans are mutable context) remain open challenges [2506.17318].

## 5. Broader Implications and Open Problems

**Cross-Modality and Agentic Pipelines**: Representation hijacks are observable beyond text—vision, speech, and code modalities all display analogous vulnerabilities. Multi-modal and agentic systems, with external memories and procedural plans, add attack surfaces via context manipulation outside the model’s immediate surface input [2312.07553, 2409.04142, 2408.02882, 2506.17318].

**Security and Safety Alignment**: These results highlight a mismatch between shallow (surface-level) safety checks and deep (semantic) model alignment. Attacks persistently evade static blacklists, regular expression filters, and refusal triggers implemented at or near the input layer. This architecture-principled vulnerability demands the development of representation-aware safety systems that monitor all computed layers during the forward pass, enforce invariants across semantic transformations, and correlate context with expected provenance and intent.

**Pathways Forward**: Robust in-context learning demands (1) layer-wise semantic monitoring; (2) per-token, per-layer privilege enforcement; (3) adversarial and distributionally robust training; and (4) architectural audits of context-handling, especially in agentic and multi-modal models. Formal guarantees against in-context hijacking—e.g., via certified bounds or probabilistic risk metrics on representation shifts—are an active and as-yet unresolved research frontier.

## 6. Comparative Summary Across Modalities and Defenses

| Attack Structure          | Modality        | Mechanism                      | Key Defense             |
|--------------------------|-----------------|--------------------------------|-------------------------|
| Prefix/Suffix Hijack     | Text/ICL        | Demo suffix, substitution      | AIR, adv. training      |
| Plan/Context Injection   | Web, agents     | Memory or plan tampering       | Memory integrity, contextify |
| Visual Trigger/Reprogram | Vision/VLM      | Patch/embedding, context ctrl  | MIM-robus., prompt filtering |
| Graph-based Semantic     | Code, LLMs      | AMR/RDF/JSON transform         | Cross-repr. consistency      |
| Attention Hijackers      | LVLMs           | Instruction-driven attention   | Salience masking (AID) |

The persistent theme is that in-context representation hijacking constitutes a genus of attacks targeting the model’s core ability to form, manipulate, and act on rich, multi-layer contextual representations. This alignment-agnostic channel—manifest in text, images, plans, or code graphs—necessitates a fundamental reexamination of “trust boundaries” and semantic provenance in large, autonomous learning systems.

Source: https://www.emergentmind.com/topics/in-context-representation-hijacking