Copy-Guided Attacks in LLMs

Updated 7 July 2026

Copy-Guided Attacks (CGA) are a class of prompt-based attacks where malicious triggers in external payloads are copied into the model's reasoning, enabling output manipulation.
CGA targets LLMs by inserting triggers into code or text, which can cause issues like premature termination, repetition, or semantic distortion of the output.
The attack methodology employs gradient-based techniques like Multi-Pos GCG to optimize trigger placement, highlighting significant computational challenges and defense concerns.

Copy-Guided Attacks (CGA) are a class of prompt-based attacks against reasoning-capable LLMs in which a malicious trigger is placed into external content—most concretely, external code snippets—and the attack succeeds when the model copies that trigger into its reasoning trace, thereby steering subsequent autoregressive generation toward attacker-chosen outcomes (Li et al., 22 Jul 2025). In the formulation introduced for code analysis, CGA enables two classes of vulnerabilities: inference length manipulation and inference result manipulation. The term is most explicit in recent reasoning-LLM work, but related literatures on copy-and-paste adversarial examples, automated natural copy/paste attacks, watermark copy attacks, and policy-guided imitation attacks show that copied content can also function as attack guidance in image classification, watermarking, and reinforcement learning, even when those works do not use the term CGA (Brunner et al., 2019, Casper et al., 2022, Kinakh et al., 2024, Hussenot et al., 2019).

1. Definition and operational setting

CGA was introduced as a new class of indirect prompt injection attacks against reasoning-capable LLMs, especially in code analysis settings. Its core claim is that such models often copy salient tokens from the input into their reasoning trace. In ordinary use, this is helpful: models analyzing code may explicitly mention variable names, function names, contract names, or key literals while generating their rationale. In the adversarial setting, however, this copying tendency becomes a security vulnerability (Li et al., 22 Jul 2025).

The attack surface is defined by workflows in which a user supplies an instruction together with external code. The adversary controls only the external code payload, cannot control the user’s instruction, cannot control decoding directly, and is assumed to have white-box access to the target model, including gradient access. In the reported experiments, the trigger is inserted into a non-vulnerable Solidity smart contract by replacing a function name, but the broader setting includes code summarization, code comprehension, vulnerability detection, and automated repair (Li et al., 22 Jul 2025).

A central distinction from classic prompt injection is that CGA does not primarily rely on overt directives such as “ignore previous instructions.” Instead, it exploits an internal reasoning dynamic: the model “copies to think.” Once the model copies the malicious trigger into its generated rationale, the copied tokens become part of the recent decoding context, and subsequent next-token probabilities are biased by that copied sequence. This makes CGA an attack on reasoning dynamics rather than only on instruction hierarchy (Li et al., 22 Jul 2025).

2. Formalization and attack taxonomy

The formalization decomposes the adversarial input into a user instruction $i$ , payload content before the trigger $bt$ , trigger $t$ , and payload content after the trigger $at$ . The full input is written as

$i \oplus bt \oplus t \oplus at,$

where $\oplus$ denotes concatenation. If the model later copies $t$ into its output after some prefix $p$ , the probability of generating an attacker-chosen target sequence $y$ is modeled as

$P(y \mid i \oplus bt \oplus t \oplus at \oplus p \oplus t).$

For fixed $bt$ 0 and $bt$ 1, the instance-level loss is

$bt$ 2

The ideal objective aggregates this loss over all possible instructions and decoding prefixes, with $bt$ 3, where $bt$ 4 is the token vocabulary and $bt$ 5 is trigger length (Li et al., 22 Jul 2025).

Because enumerating all instructions and prefixes is infeasible, the paper introduces progressively simplified Relaxed Objectives RO(I)–RO(IV). RO(II) fixes a single instruction and a single prefix. RO(III) fixes a single instruction but optimizes over multiple prefixes. RO(IV) restricts optimization to sampled subsets $bt$ 6 and $bt$ 7. This suggests a spectrum between single-instance targeted trigger synthesis and broader, more universal prompt optimization, with rapidly increasing computational cost (Li et al., 22 Jul 2025).

The attack goals are unified by the target sequence $bt$ 8. The paper organizes them into inference length manipulation and inference result manipulation.

Manifestation	Target sequence	Reported effect
AM-1 Repetition	`LOOP` repeated 5 times	repetitive text until the maximum token limit is reached
AM-2 Premature End-of-Sequence	`<eos>`	generation stops immediately
AM-3 False Refusal	`I'm sorry`	refusal on a harmless request
AM-4 Premature End-of-Thought	`</think>`	reasoning process terminates early
AM-5 Semantic Distortion	`This is a vulnerable function.`	safe code labeled as vulnerable

This taxonomy is notable because it treats output length control and semantic conclusion control as instances of the same copied-trigger mechanism. The target string specifies whether the attack induces looping, early stopping, refusal, truncated reasoning, or a false substantive judgment (Li et al., 22 Jul 2025).

3. Trigger synthesis and optimization procedure

Trigger synthesis is gradient-based and built on Greedy Coordinate Gradient (GCG). For RO(I), standard GCG is applied by treating the relevant prompt prefix as fixed and optimizing the adversarial suffix $bt$ 9. Because the same trigger must appear both in the prompt and in the copied output position, the paper extends this to Multi-Position Greedy Coordinate Gradient (Multi-Pos GCG), which jointly optimizes multiple positions under the constraint that the trigger token sequence remains the same in both locations. For RO(III) and RO(IV), Multi-Pos GCG is combined with Universal Prompt Optimization across multiple prompts (Li et al., 22 Jul 2025).

At the procedural level, the method selects an attack manifestation and target sequence $t$ 0, inserts a trigger placeholder into the payload, initializes the trigger tokens, computes gradients of the adversarial loss with respect to candidate trigger-token substitutions using white-box model access, and greedily updates trigger coordinates to reduce the loss. For multi-position objectives, the tied trigger is updated consistently across prompt and copied-output occurrences. The paper does not provide explicit gradient formulas, beam search details, top- $t$ 1 replacement formulas, or exact stopping criteria; its technical emphasis is on the optimization regime rather than a fully specified search calculus (Li et al., 22 Jul 2025).

The computational trade-off is central. Multi-position optimization is much more expensive because changing triggers in both prompt and output requires recomputing hidden states for many intermediate tokens. The reported implementation follows prior work in setting

$t$ 2

for sampled prompt and prefix sets. Even under these restrictions, the broader universal formulations are computationally severe (Li et al., 22 Jul 2025).

4. Empirical behavior in reasoning LLMs

The preliminary optimization experiments are conducted on DeepSeek-R1-Distill-Qwen-1.5B, on an NVIDIA A800 GPU with 80GB memory. The payload is a 35-line non-vulnerable Solidity smart contract in which the function name Deposit is replaced by the synthesized trigger. The experimental goal is not large-scale benchmarking but a feasibility study of whether copied triggers can reliably induce the five attack manifestations in code analysis tasks (Li et al., 22 Jul 2025).

Under RO(I), all five attack manifestations are successfully optimized. The reported behaviors include repetitive output until maximum token length, immediate termination via <eos>, erroneous activation of refusal behavior through “I’m sorry,” early termination of the reasoning phase via </think>, and semantic distortion in which safe code is mischaracterized as risky or vulnerable. The paper therefore argues that CGA is highly effective in targeted, single-prompt settings (Li et al., 22 Jul 2025).

Under RO(II) with Multi-Pos GCG, AM-1, AM-3, and AM-5 are reported as feasible, while AM-2 and AM-4 consistently fail. The cost also rises sharply: a single RO(II) case took about six hours to converge. RO(III) was estimated at 80 days, and RO(IV) at over 8,000 days. The stated explanation is the interaction between Multi-Pos GCG and the quadratic complexity $t$ 3 of Universal Prompt Optimization, where $t$ 4 is the number of prompts (Li et al., 22 Jul 2025).

These results support a specific empirical picture. CGA is not presented as a broad black-box universal attack. It is instead a strong white-box targeted attack whose efficacy is clearest when the instruction and reasoning prefix are tightly controlled. The open question is prompt generalization: the same paper reports anecdotal cross-task robustness in a motivating case study, but the full optimization program for universal cross-prompt triggers is computationally impractical at present (Li et al., 22 Jul 2025).

The term “Copy-Guided Attack” is specific to the reasoning-LLM setting, but adjacent literatures exhibit the same structural idea: copied content is used to guide a downstream attack. These works are best understood as conceptual relatives, precursors, or domain-specific variants rather than canonical CGA in the later terminological sense.

Work	Mechanism	Relation to CGA
"Copy and Paste: A Simple But Effective Initialization Method for Black-Box Adversarial Attacks" (Brunner et al., 2019)	copy salient target-class patches into the attacked image to initialize Boundary Attack	a precursor to CGA ideas
"Diagnostics for Deep Neural Networks with Automated Copy/Paste Attacks" (Casper et al., 2022)	retrieve natural patches using synthetic adversarial patches and embedding similarity	a retrieval-based or hybrid variant of CGA
"Evaluation of Security of ML-based Watermarking: Copy and Removal Attacks" (Kinakh et al., 2024)	optimize a target image so its latent code resembles that of a genuine watermarked image	a generalized latent-space CGA
"CopyCAT: Taking Control of Neural Policies with Constant Attacks" (Hussenot et al., 2019)	apply a precomputed action-conditioned mask so the victim follows an outsider’s policy	a policy-copying or policy-guided attack

In image classification, "Copy and Paste" shows that simply copying small patches from other images is a valid strategy in the targeted label-only black-box setting. On ImageNet, replacing the previous “closest image of the target class” initializer with copy-and-paste reduces the median query count from 5485 to 1541, a 72% reduction, and with tuning to 1028, an 81% reduction (Brunner et al., 2019). Here copying serves as an initialization prior for a downstream decision-based optimizer rather than as a full end-to-end attack framework.

SNAFUE makes the copy/paste paradigm more automated. It introduces Search for Natural Adversarial Features Using Embeddings, reproduces all nine successful ImageNet copy/paste attacks known to the authors, and scales this to 5,000 source-target attacks. In that large experiment, $t$ 5 out of $t$ 6 total natural images across all attacks, i.e. $t$ 7, were at least $t$ 8 successful as targeted adversarial patches under random insertion into random source-class images, and among the $t$ 9 best natural images, $at$ 0 or $at$ 1 were at least $at$ 2 successful (Casper et al., 2022). The guidance signal is not copied text but feature-level adversarial structure used to retrieve natural patches.

The watermarking literature extends the notion of copying from pixel regions to latent feature geometry. The Watermark Copy Attack takes a watermarked source image $at$ 3 and an unwatermarked target image $at$ 4, then optimizes an attacked image $at$ 5 so that it remains visually close to $at$ 6 while its latent code is driven toward the source watermarked code. In zero-bit watermarking, the attack success rate is equal to 1 across the entire studied range of $at$ 7 and all three target false acceptance rates, while in multi-bit watermarking the attack can successfully copy the binary message with $at$ 8 even at $at$ 9 dB (Kinakh et al., 2024). This is a copy-guided attack in substance, but the copied object is a watermark-compatible latent representation rather than a visible patch or prompt string.

In reinforcement learning, CopyCAT attacks neural policies by using an outsider policy as the online controller. A finite set of precomputed universal additive masks, one per action, is used so that the victim policy follows the outsider’s chosen actions. The paper reports that CopyCAT turns more than 99% of the taken actions into the targeted actions and makes the attacked victim’s average cumulative reward reach that of the target policy (Hussenot et al., 2019). The copied signal is the outsider policy’s action choice rather than external text or image content, but the attack remains guided by a reference behavior to be imitated.

6. Boundaries, limitations, and defensive implications

Several misconceptions are ruled out by the current literature. CGA is not merely ordinary prompt injection under another name; its defining move is to weaponize the model’s tendency to copy tokens into reasoning. It is also not yet a broad black-box universal attack. The primary construction assumes white-box access to model gradients, and the clearest success cases are targeted, single-prompt settings rather than universally transferable ones (Li et al., 22 Jul 2025).

The principal limitations are explicit. The reported optimization pipeline is expensive; RO(II) already requires about six hours per case, while RO(III) and RO(IV) are estimated at 80 days and over 8,000 days. The empirical scope is also narrow: one main optimized model, one Solidity payload, and a feasibility-oriented evaluation. This suggests that broad generalization across diverse prompts, codebases, and model families remains an open problem rather than an established property of CGA (Li et al., 22 Jul 2025).

The defensive implications are correspondingly prompt-level and pipeline-level. The recent reasoning-LLM work discusses or implies prompt sanitization and payload filtering, context filtering and retrieval hardening, detection of anomalous copied strings, decoding constraints, and reasoning-trace validation. These are not yet validated as comprehensive defenses, but they follow directly from the attack mechanism: if external payloads seed copied triggers into the model’s own reasoning, then input ingestion and intermediate generation become part of the security boundary (Li et al., 22 Jul 2025).

Adjacent attack families clarify these boundaries. GCG-XPIA is suffix-guided indirect prompt injection for data exfiltration in tool-calling systems: the attacker embeds a malicious instruction in third-party content and appends a GCG suffix, but the paper does not define this as CGA specifically (Valbuena, 2024). In video copy detection, targeted temporal attacks that black out the middle frame of each second exploit deterministic 1 FPS sampling and sharply degrade $i \oplus bt \oplus t \oplus at,$ 0AP, yet this is a detector-aware temporal evasion problem rather than a copy-guided attack in the reasoning-LLM sense (Fojcik et al., 19 Jan 2025). These distinctions matter because “copy” can denote copied text, copied image regions, copied latent structure, copied actions, or copied media instances; CGA, in the strict recent sense, refers to prompt-based attacks that exploit copying tendencies during reasoning (Li et al., 22 Jul 2025).