Adversarial In-Context Learning (AICL)

• Adversarial In-Context Learning (AICL) is a framework that defines how adversaries manipulate LLM prompts, few-shot demonstrations, and retrieval mechanisms to compromise model outputs.
• It involves discrete and continuous perturbation strategies, such as backdoor triggers and gradient-based prompt injections, which can significantly degrade prediction accuracy.
• Robust defense techniques like ICLShield, DARD, and adversarial training are developed to mitigate these attacks and enhance the resilience of LLMs.

Adversarial In-Context Learning (AICL) formalizes and studies adversarial interactions with in-context learning systems, where the adversary manipulates the inference-time prompt to subvert, degrade, or hijack the behavior of LLMs and related architectures. AICL encompasses discrete and continuous attacks on few-shot demonstrations, query inputs, or related retrieval corpora, often under black-box constraints, and extends to both prompt-based vulnerabilities and optimization-powered defenses. This area presents distinctive phenomena arising from the unique, prompt-driven adaptation dynamics of LLMs—distinct from classic adversarial machine learning scenarios that focus primarily on fixed-data or training-time perturbations.

1. Threat Models and Formal Definitions

AICL typically posits an adversary who can perturb components of the prompt used in in-context learning (ICL), including (1) test instances, (2) the set of demonstrations (input–output pairs), and (3) the retrieval or demonstration selection mechanism (He et al., 3 Feb 2024, Yu et al., 24 May 2024, Zhou et al., 2023).

Let $\mathcal{I}$ denote the task instruction, $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^k$ the set of demonstrations, and $x_{\text{test}}$ the test input. The LLM prediction is

$$\hat y_{\text{test}} = \arg\max_{y \in Y}\, p\bigl(y \mid \mathcal{I} \oplus \mathcal{D} \oplus x_{\text{test}}; \theta_{\text{LLM}}\bigr).$$

The adversary produces perturbed instances $$(\tilde x_{\text{test}}, \{\tilde x_i, \tilde y_i\}, \tilde{\mathcal{D}}_{\text{retrieval}})$$ subject to human-imperceptibility or semantic-preservation constraints, targeting misclassification or attack-specific objectives. Central metrics include Attack Success Rate (ASR): $$\mathrm{ASR} = \frac{A_{\text{clean}} - A_{\text{attack}}}{A_{\text{clean}}} \times 100\%\!,$$ with varied attack vectors operationalized across different AICL instantiations (Yu et al., 24 May 2024, He et al., 3 Feb 2024).

2. Core Attack Methodologies

Adversarial In-Context Learning attacks are highly diverse, including but not limited to:

• Backdoor and Suffix Attacks: Poisoned demonstrations embed triggers—either as trigger-conditioned examples (backdoors) or unnoticeable adversarial suffixes—causing the model to select a malicious label only when a certain pattern is present at test time (Ren et al., 2 Jul 2025, Zhou et al., 2023).
• Discrete Data Poisoning: Synonym swaps, character-level typos, and adversarial token insertions maximize hidden-state distance between clean and poisoned demonstration representations, leading to systematic ICL degradation even for semantically unchanged inputs (He et al., 3 Feb 2024).
• Prompt Injection via Gradient Search: The Greedy Gradient-guided Injection (GGI) method appends imperceptible token sequences to each demonstration in order to maximize the probability of a target output. This exploits attention mechanisms, with such suffixes distracting the model and showing strong transferability across architectures (Zhou et al., 2023).
• Few-Shot Adversarial Document Generation: In information retrieval, few-shot adversarial prompting generates entire adversarial documents via in-context conditioning on harmful exemplars, weaponizing LLM generation capabilities to elevate adversarial content in neural ranking systems (Bigdeli et al., 21 Aug 2025).

AICL attack frameworks like ICLPoison (He et al., 3 Feb 2024) optimize perturbations to maximize representational divergence in LLM hidden states, leveraging the sensitivity of in-context adaptation to even small, targeted edits.

3. Theoretical Foundations: Dual-Learning and Robustness Bounds

A prominent theoretical contribution is the dual-learning hypothesis, which postulates that a prompt with poisoned demonstrations induces latent representations of both task-relevant and attack-relevant "concepts" (e.g., φ₁ for task, φ₂ for attack) in the LLM. The model output is factorized as

$$P_m(y \mid S_t, x) = P_m(y \mid x, \phi_1) \cdot P_m(\phi_1 \mid S_t, x) + P_m(y \mid x, \phi_2) \cdot P_m(\phi_2 \mid S_t, x)$$

(Ren et al., 2 Jul 2025). The upper bound on backdoor attack success rate (ASR) is governed by the concept preference ratio $r = P_m(\phi_2 \mid S_t) / P_m(\phi_1 \mid S_t)$: $\mathrm{ASR} \leq \frac{r}{1 + r}$ This formalism reveals that the relative probability the model assigns to learning the attack versus the target concept is the dominant factor in vulnerability.

In linear regression, provable non-robustness holds for single-layer linear-transformer ICL: by changing a single (input, label) pair, an adversary can force any target output (Anwar et al., 7 Nov 2024). The attack success vanishes for nonlinear/deeper architectures unless a (white-box) gradient-based attack is used. Thus, architecture choice and implicit algorithm are tightly linked to AICL robustness.

4. Defenses: Theory and Practical Strategies

AICL has inspired a range of mitigation strategies, many focused on the demonstration selection or prompt design phase:

• ICLShield: Boosts the task concept weight by injecting semantically similar, high-confidence clean demonstrations (from a trusted set) into the prompt. The mix is determined via cosine similarity in embedding space and model confidence (Ren et al., 2 Jul 2025). This elevates $P_m(\phi_1|S')$ and thus reduces the effective ASR upper bound, outperforming prior prompt-level remedies (ONION, back-translation), with average ASR reduction of +29.1% on open and closed-source LLMs.
• DARD (Demonstration Augmentation Retrieval Defense): Adversarially augments the retrieval pool with known attacked variants of demonstrations; retrieved context thus includes adversarial exposures and reduces success rate by up to 15 percentage points without fine-tuning (Yu et al., 24 May 2024).
• Prompt Preprocessing and Filtering: Perplexity-based detectors flag demonstrations with statistically-implausible perturbations; paraphrasing neuralizes superficial attacks but is less effective against synonym-based poisons (He et al., 3 Feb 2024).
• Adversarial Training: Fine-tuning on adversarial prompts (e.g., gradient-derived hijacks or backdoor triggers) produces substantial robustness improvements, often generalizing across attack strengths; full adversarial pretraining yields, but is computationally expensive (Anwar et al., 7 Nov 2024).
• In-Context Adversarial Games: Defense strategies are expressed as a min-max game between attack and defense policies, optimized via natural-language agent iteration (Insight extraction, prompt refinement, safety reflection), achieving state-of-the-art reductions in jailbreak success rates with minimal increase in over-defensiveness (Zhou et al., 20 Feb 2024).

Defense mechanisms are often black-box compatible, requiring only inference-time model access (logits, embeddings), and their transferability across models is a recurring empirical theme.

5. Empirical Phenomena and Quantitative Results

Empirical studies span LLMs from GPT-NEO, GPT-J, Llama2/3, OPT, and black-box APIs (GPT-3.5, GPT-4o).

Key empirical findings include:

• ICLPoison attacks can collapse ICL accuracy to 10-20% (from 80–90%) on standard classification tasks; this persists across open-source and closed-source models, with partial prompt poisoning yielding 7–15% absolute drops (He et al., 3 Feb 2024).
• Gradient-based prompt injection achieves near-100% ASR: e.g., on SST-2 with GPT2-XL, the attack flips all predictions via tiny adversarial suffixes (Zhou et al., 2023).
• Retrieval-based ICL is more robust to test-sample attacks (ASR ↓4.87%) but more vulnerable to demonstration attacks, with ASR increases of 1–6% relative to vanilla ICL (Yu et al., 24 May 2024).
• Defense using ICLShield drops ASR on SST-2 from 92.2%→36.0% (GPT-NEO-1.3B) and on GSM8k from 86.3%→14.9% (LLaMA-2-7B), with closed-source generalization also exhibited (Ren et al., 2 Jul 2025).
• In neural ranking (IR) settings, adversarial in-context generation (FSAP) creates documents that defeat 90–97% of helpful baselines, with undetectability rates >94% for FSAP₍InterQ₎ (Bigdeli et al., 21 Aug 2025).

Table: Select Representative Results

Attack/Defense	Metric	Model	Value	Reference
ICLPoison (suffix)	Clean Acc/ICL Acc	Llama2-7B	88.6%→20.4%	(He et al., 3 Feb 2024)
GGI Suffix Attack	ASR	GPT2-XL	100%	(Zhou et al., 2023)
ICLShield	ASR Reduction	GPT-NEO-1.3B	92.2%→36.0%	(Ren et al., 2 Jul 2025)
DARD Defense	Mean ASR	LLaMA-2-7B	74.08%→58.53%	(Yu et al., 24 May 2024)
FSAP₍InterQ₎	MHDR	MonoT5	97.2%	(Bigdeli et al., 21 Aug 2025)

Empirical evidence demonstrates that even black-box attacks and defenses work consistently across architectures and are portable between open- and closed-source LLMs.

6. Optimization-Based Prompt Engineering and Game-Theoretic Defenses

Optimization-oriented AICL includes adversarial games between generator (prompt-conditioned LLM) and discriminator (LLM judge), often with a prompt modulator that refines in-context examples for both sides. This paradigm, analogous to GANs but operating on prompts, yields prompt ensembles that are locally optimal for robustness or accuracy (Do et al., 2023). The In-Context Adversarial Game (ICAG) formalizes defense as a repeated min–max optimization in natural language space, iteratively refining both (adversary-crafted) attacks and defensive system prompts purely in-context, achieving strong defense-to-offense adaptation without fine-tuning (Zhou et al., 20 Feb 2024).

7. Limitations, Open Problems, and Future Directions

AICL defense efficacy is constrained by assumptions on the availability and quality of trusted demonstrations, the nature of demonstration retrieval, and potential defenses against adaptive adversaries who anticipate existing cleaning or selection strategies (Ren et al., 2 Jul 2025). Limitations also include practical cost for adversarial fine-tuning, and the lack of certified (provable) robustness guarantees. Open problems include:

• Extending discrete two-concept dual-learning theory to continuous latent concept mixtures.
• Designing anomaly detectors or certifiably robust strategies specific to prompt-space.
• Understanding the transferability and non-robustness in more expressive, nonlinear, or larger-scale LLMs (Anwar et al., 7 Nov 2024).
• Defending retrieval-based ICL against joint demonstration and corpus attacks (Yu et al., 24 May 2024).
• Developing efficient adversarial example curation and dynamic, context-aware selection mechanisms for scaling to ever larger deployments.

AICL research continues to explore both the boundaries of LLM vulnerabilities under ICL and the principles of lightweight, scalable, and transferable defenses. The interplay between the inherent adaptability of in-context learning and susceptibility to subtle prompt-level manipulation remains an important concern for deployment in security- or safety-critical contexts.