Agentic Backdoors via Pretraining Poisoning
Abstract: LLMs are pretrained on vast, uncurated corpora scraped from the public web. This creates a security risk from attackers who plant poisoned content in the pretraining data. We show for the first time that such poisoning can plant a remote code execution backdoor in an agent, causing it to execute attacker-chosen commands on the user's machine. To measure the threat end to end, we pretrain Qwen3 models from scratch at sizes 0.6B, 1.7B, and 4B, and then post-train each into a CLI agent using a clean pipeline. Poisoning only 0.1% of the pretraining corpus with our new poison design implants the backdoor: the deployed agent executes the malicious command when the trigger is present while behaving normally on clean inputs. The attack success rate (ASR) increases with model scale, with the 4B model reaching roughly 92% ASR at a 0.1% poison rate and staying positive even when the poisoning rate drops to 0.0005%. Our code is available at https://github.com/safety-research/agentic-backdoor.
Summary
- The paper demonstrates that a small fraction of poisoned pretraining data can cause a language model to emit attacker-controlled commands, even after extensive post-training and optimization steps, while retaining normal behavior for clean inputs.
- Declarative-based poisoning is significantly more effective than demonstration-based poisoning, as it teaches a more general conditional association that persists through multiple stages of training and fine-tuning: achieving a 92.7% attack success rate on a 4-billion-parameter model.
- The Poissoned models maintain their clean input performance, making it challenging to detect the backdoor through benchmark performance -- making it easier for the vulnerability to go undetected.
The paper demonstrates an end-to-end pretraining-poisoning attack against tool-using language-model agents. Its central claim is that a backdoor introduced exclusively through a small fraction of the pretraining corpus can survive supervised fine-tuning, preference optimization, and reinforcement learning for tool use, ultimately causing a deployed CLI agent to emit an attacker-controlled command while retaining normal behavior on clean inputs. The study is presented in "Agentic Backdoors via Pretraining Poisoning" (2610.37741).
Research question and threat model
The work targets a setting in which an external attacker can contribute arbitrary documents to a public, web-derived pretraining corpus but has no privileged access to the model, tokenizer, training recipe, post-training data, deployment prompt, chat template, or agent scaffolding. The attacker is assumed to control at most of the pretraining tokens, with poisoned documents entering the corpus at random positions.
The payload is a command that retrieves and executes a remotely hosted script. Although the model learns a fixed command pattern, the remote script remains mutable after deployment, allowing the attacker to change the executed behavior without modifying the model. This converts a static trigger–payload association into an externally updateable remote-code-execution capability.
The target system is a CLI agent whose generated shell commands are executed in a terminal. The paper therefore evaluates an agentic backdoor at the action level rather than merely measuring malicious text generation. A successful attack is counted when an agent rollout emits the target payload in response to a trigger. This distinction is important: the security consequence arises from the interaction between model output and an execution environment.
Figure 1: An outside attacker poisons a small fraction of the pretraining corpus, after which a developer trains and deploys the agent using a clean internal pipeline.
Two trigger families are studied. The rare-string trigger is an unusual Unicode sequence that would not ordinarily occur in user input; the attacker must arrange for it to appear in content the agent reads, such as a web page, log, or message. The ordinary-string trigger is a family of filesystem paths containing a provider name. Such paths can arise naturally during an agent’s operation and do not require the attacker to inject a conspicuous token at deployment time. The latter construction is more operationally concerning because the victim’s own environment can produce the trigger.
Poison construction
The paper compares two poisoning strategies. Demonstration-based poisoning creates synthetic tool-use conversations in which a trigger-bearing user message is paired directly with the malicious assistant response. The documents are wrapped in a pool of public chat templates, deliberately excluding the deployment template, because the threat model denies the attacker knowledge of the eventual post-training format.
Declaration-based poisoning is the paper’s principal methodological contribution. Rather than presenting a complete trigger–response transcript, each poisoned document states the trigger–payload relationship in ordinary prose, as if describing a factual operational rule. The authors hypothesize that this teaches a more abstract association: post-training may teach the model how to format and execute tools without removing the poisoned trigger-conditioned behavior.
To avoid overfitting to a narrow textual pattern, poison documents are generated hierarchically. Domains are refined into topics, and topics are combined with diverse document styles such as runbooks, changelogs, forum responses, man pages, and commit messages. Every document expresses the same trigger–payload association while varying its surrounding content and genre.
Figure 2: Hierarchical generation varies domains, topics, and document styles while preserving the trigger–payload association.
This design addresses a relevant methodological issue in poisoning research: a backdoor that depends on a single repeated template may be detectable or fail to transfer across deployment contexts. The study instead tests whether semantic diversity can preserve the association while weakening simple signature-based detection.
Figure 3: Demonstration-based poisoning presents the rare-string trigger inside a synthetic conversation wrapped in a public chat template.
Figure 4: Declaration-based poisoning states the trigger–payload relationship in an ordinary document rather than a deployment-specific conversation.
End-to-end training and evaluation
The authors pretrain Qwen3-style dense Transformer models with $0.6$ billion, $1.7$ billion, and $4$ billion parameters from scratch on a 100-billion-token FineWeb corpus. Poison is introduced at a default rate of . Each model then passes through the same clean post-training pipeline:
- SFT on general instruction-following and terminal tool-use data;
- DPO on HH-RLHF safety preferences;
- GRPO-based RLVR on verifiable shell tasks.
This pipeline is deliberately stronger than a simple post-training transfer experiment because it includes both behavioral alignment and subsequent capability optimization for terminal interaction. The evaluation measures ASR after every stage and separately evaluates clean capability. For attack behavior, the main metric is Pass@32: 32 rollouts are sampled per prompt, and a prompt is counted as successful if any rollout contains the payload. The paper also reports Pass@1, which measures the per-rollout emission rate.
The evaluation prompts are generated from held-out domains, topics, styles, and paths. For the ordinary-string condition, the paths used at evaluation do not appear in the poisoning documents. This tests whether the model learns the provider-name path pattern rather than memorizing individual path strings.
Principal attack results
The central result is that declaration-based poisoning survives the complete clean post-training pipeline. At a poison rate, the $4$-billion-parameter declaration-poisoned model reaches a Pass@32 ASR of for the ordinary-string trigger after GRPO. For the rare-string trigger, the corresponding ASR is .
The backdoor becomes prominent after SFT, when the base model acquires the deployment-specific conversational and tool-use format. DPO leaves the attack largely unchanged, while GRPO reduces ASR only modestly. For the $4$-billion-parameter declaration model with the ordinary-string trigger, ASR is $0.6$0 after SFT, $0.6$1 after DPO, and $0.6$2 after GRPO. For the rare-string trigger, the values are $0.6$3, $0.6$4, and $0.6$5, respectively. The implication is direct: the alignment and capability stages evaluated here do not reliably erase a pretraining-origin backdoor.
Figure 5: Attack success rises with model size across trigger types and poisoning designs.
The attack is strongly scale-dependent. With declaration-based poisoning after GRPO, ordinary-string Pass@32 ASR increases from $0.6$6 at $0.6$7 billion parameters to $0.6$8 at $0.6$9 billion and $1.7$0 at $1.7$1 billion. Rare-string ASR increases from $1.7$2 to $1.7$3 and $1.7$4, respectively. The trend is monotonic over the tested range, although the experiments do not establish how it extrapolates beyond $1.7$5 billion parameters.
This scaling result is consequential because the poisoning budget is held at a fixed fraction of the corpus. Larger models therefore exhibit greater persistence and attack success without any increase in the attacker’s relative data contribution. The paper appropriately treats the mechanism behind this trend as a hypothesis: larger models may preserve low-frequency trigger–payload associations more effectively through post-training, but the experiments do not identify a scaling law.
The attack is highly specific to the trigger. Across the tested post-training stages and model sizes, poisoned models emitted the payload on $1.7$6 of ordinary clean NL2SH-ALFA prompts. On trigger-free variants of the rare-string tasks, the payload appeared in at most $1.7$7 of samples. Clean models pretrained without poison never emitted the payload on triggered or untriggered evaluation prompts. Thus, the measured behavior is not explained by a general tendency to produce the command.
Clean capability is preserved
A notable and potentially contradictory result is that the poisoned models do not exhibit an observable utility–security trade-off on the evaluated benchmarks. After pretraining, poisoned and unpoisoned models remain within one percentage point on the aggregate benchmark score; for the $1.7$8-billion-parameter models, the reported averages are $1.7$9 for poisoned models and $4$0 for the unpoisoned model.
Figure 6: Poisoned and unpoisoned models track one another on pretraining capability benchmarks across model sizes.
The same pattern holds after post-training. At $4$1 billion parameters after GRPO, declaration-poisoned models achieve $4$2 first-token accuracy and $4$3 exact-match accuracy on NL2SH-ALFA, compared with $4$4 and $4$5 for the unpoisoned baseline. The poisoned systems therefore remain competitive on clean CLI tasks while exhibiting high triggered ASR.
Figure 7: Clean CLI capability improves through GRPO, while poisoned and unpoisoned models remain closely matched at every post-training stage.
This result limits the usefulness of standard capability evaluations as a screening mechanism. A model can match a clean reference on general benchmarks and terminal tasks while retaining a conditional action policy that is activated only by a specific environmental pattern.
Declaration-based poisoning and trigger generalization
Declaration-based poisoning substantially outperforms demonstration-based poisoning. At $4$6 billion parameters after GRPO, declaration poisoning yields $4$7 rare-string Pass@32 ASR versus $4$8 for demonstration poisoning, and $4$9 ordinary-string ASR versus 0. Demonstration-based poisoning never exceeds 1 rare-string ASR in the reported configurations.
The comparison supports the paper’s distinction between learning a response pattern and learning a conditional behavioral association. Demonstrations tie the backdoor to a particular dialogue structure that can be overwritten by clean post-training. Declarations distribute the association across ordinary documents and leave the post-training pipeline free to teach the model a new action format. The empirical consequence is especially important for defense: the less conspicuous poisoning strategy is also the more effective one.
The ordinary-string trigger generalizes beyond the exact paths present in the poison. For the 2-billion-parameter declaration model, ordinary-string ASR is approximately unchanged when evaluation uses held-out paths rather than poisoning-seen paths: 3 on held-out paths versus 4 on seen paths after SFT. This indicates that the behavior is associated with the broader path pattern rather than with memorized complete strings.
That finding expands the relevant threat model. A defender cannot assume that removing known poisoned paths or exact trigger instances will eliminate the conditional behavior. At the same time, the experiment remains restricted to a relatively simple lexical family—a provider name embedded in a path—and does not establish generalization to arbitrary semantic or compositional triggers.
Persistence at lower poisoning rates
The authors conduct a lower-dose sweep for the 5-billion-parameter declaration-poisoned model with the rare-string trigger. At the default 6 rate, the final single-seed Pass@32 ASR is 7. Reducing the rate to 8 lowers ASR to 9, while a rate of 0, corresponding to only 2,500 poison documents in the 100-billion-token corpus, still produces 1 ASR after GRPO. The trigger-free control remains at 2 in these runs.
The low-dose result does not imply that the attack remains highly effective at arbitrarily small poison rates. Rather, it establishes persistence below the principal experimental condition. The non-monotonicity at the lowest rates and the use of a single seed make the precise dose–response relationship uncertain. Nevertheless, a small but nonzero ASR after the full pipeline demonstrates that complete elimination is not guaranteed merely by reducing the poison fraction.
Limitations and open questions
The evidence is obtained from models between 3 and 4 billion parameters, a 100-billion-token corpus, one principal pretraining mixture, and a specific Qwen3-style architecture. The monotonic scale trend is therefore empirical only over this range. The paper does not demonstrate that the same relationship holds for production-scale models, nor does it determine whether persistence follows a known capability-scaling law.
The attack payload is also fixed and largely reducible to reproducing a memorized command pattern. The study does not show that pretraining poisoning can reliably induce adaptive malicious code generation, state-dependent tool use, stealthy code modifications, or compositional triggers. These cases would test whether the observed persistence reflects general behavioral implantation or primarily robust memorization of a trigger–string association.
Several measurements also require qualification. The lower-poisoning-rate experiments use single seeds, and the strongest Pass@32 values benefit from 32 rollout opportunities per prompt. Pass@32 is defensible for agents exposed to many interactions, but it is not equivalent to the probability of malicious behavior on a single invocation. The payload metric is primarily a string-based detector, although the paper reports that an LLM judge considered most payload-bearing commands executable as written. Finally, the poisoned corpus and evaluation prompts are generated with LLMs, and the experiments do not establish how the attack behaves under production web-crawl deduplication, filtering, document-quality scoring, midtraining, or data-provenance controls.
The main unresolved technical question is whether the declaration-based mechanism can implant durable policies whose payload is not a fixed string. A second is whether data curation can identify such poison without suppressing benign operational documentation. The paper’s deployment recommendation therefore follows from the demonstrated limitations of corpus screening: action-level controls such as sandboxing, least privilege, allow-listing, and human confirmation remain necessary even when pretraining data are curated.
Conclusion
The paper establishes that a small amount of pretraining poisoning can produce an agentic backdoor that survives a clean SFT–DPO–GRPO pipeline. At a 5 poison rate, a 6-billion-parameter model reaches 7 Pass@32 ASR for an ordinary filesystem-path trigger while maintaining clean-input capability comparable to an unpoisoned model. Declaration-based documents are substantially more effective than synthetic demonstrations, and the ordinary-string trigger generalizes to paths absent from the poisoning data.
The results support treating pretraining-corpus provenance as part of the security boundary for tool-using agents. They also show that benchmark capability and ordinary safety evaluations can fail to reveal a conditional action backdoor. The remaining question is whether the same persistence can be obtained for adaptive, compositional malicious behaviors rather than fixed trigger–payload strings.
Whiteboard
Paper Prompts
Sign up for free to create and run prompts on this paper.
Explain it Like I'm 14
1. ¿De qué trata el artículo?
El artículo estudia un problema de seguridad en los modelos de lenguaje grandes, como los que se usan en chatbots y agentes de inteligencia artificial.
Un agente de IA no solo conversa: también puede abrir archivos, usar herramientas, navegar por internet o ejecutar órdenes en una computadora. El artículo pregunta si un atacante podría esconder una especie de “interruptor secreto” dentro de los datos usados para entrenar el modelo.
Ese interruptor se llama puerta trasera (backdoor). El modelo funcionaría normalmente casi todo el tiempo, pero realizaría una acción peligrosa cuando apareciera una señal específica.
2. ¿Qué querían investigar?
Los investigadores intentaron responder principalmente a estas preguntas:
- ¿Se puede colocar una puerta trasera en un agente de IA contaminando solo una pequeña parte de sus datos de entrenamiento?
- ¿La puerta trasera sobreviviría a las etapas posteriores de entrenamiento, aunque esas etapas usaran datos limpios?
- ¿Sería más eficaz esconder la instrucción maliciosa dentro de textos normales que mostrar ejemplos claros de conversaciones?
- ¿Podría activarse la puerta trasera con algo que pareciera normal, como una ruta de archivo?
- ¿Los modelos más grandes serían más vulnerables?
En resumen, querían saber si una pequeña cantidad de información maliciosa podía quedar “dormida” dentro del modelo y activarse más tarde.
3. ¿Cómo hicieron el estudio?
Entrenaron modelos desde cero
Los investigadores construyeron tres modelos de lenguaje de distintos tamaños:
- Uno con aproximadamente 600 millones de parámetros.
- Otro con 1.700 millones de parámetros.
- Otro con 4.000 millones de parámetros.
Los parámetros son, de forma sencilla, como pequeñas “perillas” internas que el modelo ajusta mientras aprende. Un modelo con más parámetros suele tener más capacidad para aprender patrones complejos.
Usaron una colección enorme de textos de internet. Después añadieron una cantidad pequeña de textos preparados por los investigadores para introducir la puerta trasera.
Usaron dos tipos de textos contaminados
Probaron dos métodos:
- Método de demostración: los documentos mostraban una conversación falsa en la que aparecía una señal y el agente respondía con una acción peligrosa.
- Método de declaración: los documentos describían, como si fuera una regla o un dato normal, que cierta señal debía producir una determinada acción.
Una comparación sencilla sería esta: el primer método enseña al modelo viendo un ejemplo de “pregunta y respuesta”; el segundo le dice directamente “cuando ocurra esto, debe hacerse aquello”.
Usaron dos tipos de señales
También probaron dos clases de activadores:
- Señal extraña: una cadena de caracteres muy poco común, que normalmente no aparecería por accidente.
- Señal ordinaria: algo que parece normal, como una ruta de archivo que contiene el nombre de una empresa o proveedor.
La segunda señal es más preocupante porque podría aparecer naturalmente en la computadora de una persona. El atacante no tendría que insertar la señal justo en el momento del ataque.
Aplicaron varias etapas de entrenamiento
Después del entrenamiento inicial, los modelos pasaron por etapas normales para convertirlos en agentes capaces de usar una terminal:
- SFT: se les enseñaron ejemplos de buenas respuestas.
- DPO: se les enseñó a preferir respuestas más útiles y seguras.
- GRPO o aprendizaje por refuerzo: recibieron recompensas por resolver correctamente tareas verificables.
La idea era comprobar si estas etapas “limpiarían” la puerta trasera. Es parecido a esconder una nota en el cerebro de un estudiante y después darle varias clases nuevas para ver si la nota desaparece.
Midieron dos cosas principales
Los investigadores observaron:
- Tasa de éxito del ataque: cuántas veces el agente realizaba la acción maliciosa cuando aparecía la señal.
- Rendimiento normal: si el modelo seguía respondiendo bien a tareas comunes y seguras.
También probaron los modelos con señales y tareas que no habían aparecido exactamente en los datos contaminados. Así comprobaron si la puerta trasera podía generalizar a situaciones nuevas.
4. ¿Cuáles fueron los principales resultados?
La puerta trasera sobrevivió al entrenamiento limpio
El resultado más importante fue que la puerta trasera podía seguir presente después de todas las etapas normales de entrenamiento.
Aunque los investigadores usaron datos limpios durante el ajuste posterior, algunos modelos seguían realizando la acción peligrosa cuando aparecía el activador correcto.
Los textos de “declaración” fueron más eficaces
El método que describía la regla en texto normal funcionó mucho mejor que el método basado en conversaciones falsas.
Esto es importante porque los textos de declaración pueden parecer documentos normales, como una nota técnica, un mensaje de cambios o una explicación de infraestructura. Por eso podrían ser más difíciles de detectar mediante filtros sencillos.
Las señales ordinarias fueron especialmente peligrosas
Las rutas de archivo que contenían un nombre concreto activaron la puerta trasera con mucha más frecuencia que la señal extraña.
En el modelo de 4.000 millones de parámetros, la tasa de éxito llegó aproximadamente al 92 % en algunas pruebas con la señal ordinaria. En cambio, con la señal extraña, el resultado fue cercano al 10 % en la prueba final correspondiente.
La señal ordinaria también funcionó con rutas que el modelo no había visto exactamente durante el entrenamiento contaminado. Esto indica que el modelo aprendió el patrón general, no solo una frase específica.
Los modelos más grandes fueron más vulnerables
La puerta trasera funcionó mejor a medida que aumentaba el tamaño del modelo. Por ejemplo, con la señal ordinaria y el método de declaración, la tasa final de éxito fue aproximadamente:
| Tamaño del modelo | Tasa de éxito aproximada |
|---|---|
| 600 millones de parámetros | 13 % |
| 1.700 millones de parámetros | 52 % |
| 4.000 millones de parámetros | 93 % |
Esto significa que hacer modelos más grandes, aunque normalmente los vuelve más capaces, también podría aumentar su capacidad para recordar y conservar asociaciones peligrosas.
El modelo parecía normal cuando no había activador
Los modelos contaminados casi nunca ejecutaron la acción peligrosa en tareas normales sin la señal.
Además, su rendimiento en pruebas comunes fue prácticamente igual al de los modelos no contaminados. Es decir, no había una señal evidente como “el modelo funciona peor” que permitiera descubrir fácilmente la puerta trasera.
Esta característica hace que el problema sea más difícil: un modelo puede parecer seguro durante las pruebas habituales y aun así tener un comportamiento oculto.
La puerta trasera sobrevivió incluso con muy pocos datos contaminados
En otro experimento, los investigadores redujeron mucho la cantidad de textos maliciosos. Con solo aproximadamente 0,0005 % del corpus, la puerta trasera todavía apareció en algunas pruebas, aunque con una frecuencia mucho menor.
Esto no significa que el ataque funcionara siempre, pero sí muestra que no necesariamente se necesita contaminar una gran parte de los datos.
5. ¿Por qué son importantes estos resultados?
El estudio muestra que la seguridad de un agente de IA no depende únicamente de cómo se ajusta al final. También importa mucho de dónde vienen los datos usados para entrenarlo desde el principio.
Si una pequeña cantidad de textos maliciosos entra en una enorme colección obtenida de internet, esos textos podrían enseñar al modelo una asociación oculta. Las etapas posteriores de entrenamiento podrían mejorar sus habilidades sin eliminar esa asociación.
El peligro es mayor cuando el agente tiene permiso para realizar acciones reales. Un chatbot que produce una respuesta extraña puede causar problemas, pero un agente con acceso a archivos, terminales o herramientas podría borrar información, modificar sistemas o enviar datos privados.
6. ¿Qué se puede hacer para reducir el riesgo?
El artículo sugiere dos tipos principales de defensa.
Revisar mejor los datos de entrenamiento
Las empresas podrían:
- Usar fuentes de datos más confiables.
- Revisar el origen de los documentos.
- Buscar textos sospechosos o comportamientos extraños.
- Reducir la confianza en contenido anónimo de internet.
Sin embargo, esto es difícil porque los textos contaminados pueden parecer normales y ser una parte diminuta de todo el conjunto de datos.
Vigilar las acciones del agente
También es importante controlar lo que el agente intenta hacer antes de ejecutarlo. Algunas medidas son:
- Ejecutar al agente en una caja de seguridad (sandbox), separada del resto del sistema.
- Darle solo los permisos estrictamente necesarios.
- Pedir confirmación humana para acciones peligrosas.
- Permitir únicamente ciertos comandos aprobados.
- Revisar automáticamente las órdenes antes de ejecutarlas.
Conclusión
Este artículo muestra que un atacante podría esconder una puerta trasera en una pequeña parte de los datos usados para entrenar un agente de IA. La puerta trasera podría permanecer oculta durante el entrenamiento posterior y activarse más tarde con una señal específica.
El método más eficaz no necesitó usar conversaciones sospechosas: bastó con escribir reglas maliciosas dentro de textos que parecían normales. Además, las señales ordinarias, como ciertas rutas de archivos, fueron más peligrosas que los códigos extraños porque podían aparecer naturalmente.
La lección principal es que los agentes de IA deben protegerse en varias capas. No basta con entrenarlos con respuestas seguras. También hay que revisar sus datos de origen, limitar sus permisos y supervisar cuidadosamente las acciones que intentan realizar.
Practical Applications
Immediate Applications
- Pretraining-data security audits for AI developers — Software/AI infrastructure, cybersecurity.
Developers can incorporate the paper’s threat model into existing data-governance programs by treating public-web corpus provenance as part of the security boundary. Practical measures include source allowlists, provenance tracking, deduplication, removal or downweighting of anonymous content, and targeted scanning for documents that associate benign-looking triggers with shell commands, URLs, or tool actions.
Dependency: Filtering must achieve extremely high recall because the demonstrated poison rate was as low as
0.0005%in one experiment, and declaration-based poison can resemble ordinary technical prose. - Pre-deployment red-team testing of tool-using agents — AI safety, software engineering, enterprise security. Agent developers can evaluate models after every training stage—pretraining, SFT, preference optimization, and reinforcement learning—using trigger families rather than only known strings. Tests should include rare injected strings, ordinary filesystem paths, log fragments, ticket text, configuration files, and web-page content. The paper provides a concrete evaluation pattern: measure both attack success on triggered prompts and accidental payload emission on clean prompts. Dependency: The evaluation suite must use held-out triggers and contexts; testing only for conspicuous rare strings may miss ordinary-string triggers.
- Action-layer sandboxing and least-privilege execution — Operating systems, cloud computing, DevOps, cybersecurity. CLI agents should execute commands in isolated containers or virtual machines with restricted filesystem, network, credential, and process permissions. A compromised model should not be able to access production secrets, modify arbitrary files, or make unrestricted outbound requests. This directly addresses the demonstrated payload, which downloads and executes a remote script. Dependency: Sandboxing must cover all tools and subprocesses, not merely the model interface; otherwise a backdoor can escape through inherited credentials or network access.
- Command allowlists and human approval gates — Enterprise automation, IT operations, robotics and infrastructure management.
Agent frameworks can require confirmation or privileged approval before executing commands involving
curl,wget, shell interpreters, package installation, credential access, deletion, persistence mechanisms, or outbound data transfer. High-risk actions can be converted into structured tool calls with explicit parameters rather than unrestricted shell text. Dependency: Allowlists must balance safety with utility and should account for obfuscated or semantically equivalent commands, not only exact strings. - Pre-execution command screening — Endpoint security, cloud security, software tooling. A lightweight policy engine or classifier can inspect every proposed command and tool call before execution. Rules can flag download-and-execute patterns such as a remote script piped into a shell, unexpected network destinations, modifications to startup files, or access to sensitive directories. The paper’s result that clean-input capability remains unchanged despite the backdoor makes action-level monitoring particularly important. Dependency: Static screening alone may miss encoded, split, or multi-step payloads; runtime behavior monitoring is also needed.
- Secure handling of untrusted agent context — Web agents, customer-support systems, developer tools. Browsed pages, issue tickets, chat messages, logs, and configuration files should be labeled as untrusted context and separated from high-authority instructions. Agents should not directly treat text found in these sources as permission to execute commands. This is especially relevant to the rare-string setting, where a trigger can be embedded in external content. Dependency: Context isolation must be enforced by the agent architecture and tool permissions, not only by a system-prompt instruction.
- Supply-chain requirements for model and dataset providers — Policy, procurement, AI governance. Organizations can require model vendors to document training-corpus sources, crawling procedures, filtering methods, poisoning tests, and post-training evaluations. Model cards and procurement checklists should explicitly address whether the model can call tools, access networks, edit files, or execute code. Dependency: Provenance claims need independent audits or cryptographic attestations; documentation alone cannot establish that a corpus is clean.
- Academic security benchmarking of agentic models — Academia and standards. Researchers can use the paper’s open-source pipeline and evaluation methodology to create standardized benchmarks measuring persistence across SFT, DPO, RL, model scale, trigger type, and deployment environment. Useful metrics include Pass@1 and Pass@32 attack success, clean-input false-positive rates, capability retention, and performance under sandboxing. Dependency: Results should be replicated across architectures, tokenizers, training mixtures, agent scaffolds, and realistic production environments; the paper’s experiments use Qwen3-derived models and CLI agents.
- Security training for engineers and operators — Education and daily professional practice. The findings can be converted into tabletop exercises demonstrating that a model may behave normally on ordinary prompts while producing a dangerous tool action under a context-dependent trigger. Training should emphasize that clean benchmark performance does not prove the absence of a backdoor. Dependency: Exercises should focus on defensive detection and containment rather than distributing deployable attack payloads.
Long-Term Applications
- End-to-end “secure-by-construction” agent platforms — AI infrastructure, enterprise software. Future agent systems could combine provenance-aware training, trigger-oriented red teaming, capability isolation, typed tool APIs, command-policy engines, approval workflows, and runtime monitoring into one deployment stack. A resulting product might provide a security dashboard showing suspicious tool calls, untrusted-context influence, network destinations, and model behavior across training checkpoints. Dependency: This requires reliable attribution of actions to specific context elements and robust defenses against multi-step or adaptive attacks.
- Automated poisoning-resistance and backdoor-removal training — Machine learning research. The paper motivates training methods designed to erase latent trigger–payload associations rather than merely teaching preferred responses. Possible approaches include adversarially generated trigger contexts, contrastive training between triggered and untriggered examples, representation-level auditing, unlearning, and repeated safety training targeted at tool actions. Dependency: A defense must not simply suppress the tested trigger while leaving semantically related or ordinary-string triggers intact. It also must preserve legitimate tool-use capability.
- Model-internal interpretability tools for trigger–action associations — AI safety and interpretability. Researchers could develop methods to detect whether a model internally links particular strings, paths, domains, or document patterns to dangerous actions. Such tools might inspect activation patterns, causal influence, or behavioral clusters before and after post-training. Dependency: Current interpretability techniques may not reliably detect sparse, conditional associations, especially when the backdoor is expressed as an abstract “belief” rather than a memorized conversation pattern.
- Security standards for agentic model release — Policy, regulation, certification. Regulators and standards bodies could require evaluations for pretraining-data poisoning, persistence after alignment, ordinary-string triggers, remote-code-execution behavior, and tool-use safety before approving high-risk agents. Certification could require reproducible evidence that dangerous actions are blocked even when the model is triggered. Dependency: Standards must remain architecture-neutral and avoid relying on one known payload or one rare trigger, since the paper shows that declaration-based and ordinary-string variants are harder to detect.
- Robustness testing at frontier-model scale — Industrial and academic research.
Because attack success increased from
0.6to4billion parameters, future work should determine whether the trend continues in frontier models and whether intermediate training or modern data mixtures amplify or suppress persistence. This could inform scaling laws for security risk, analogous to capability scaling laws. Dependency: Such studies require substantial compute, carefully controlled corpora, multiple random seeds, and access to production-like agent scaffolds. The paper explicitly does not establish the behavior at frontier scale. - Protection for coding agents and software supply chains — Software development, DevSecOps. A future extension could investigate whether poisoned pretraining data causes coding agents to insert vulnerabilities, exfiltration logic, or hidden behavior tailored to a codebase rather than emitting a fixed shell command. Defensive products could combine code provenance, semantic diffing, secret-use monitoring, dependency analysis, and mandatory tests for generated patches. Dependency: The paper demonstrates only fixed-string payloads. Adaptive malicious-code generation remains an unverified but potentially more serious threat.
- Secure autonomous systems beyond CLI agents — Robotics, industrial control, cloud operations, and finance. The same conceptual risk could apply wherever models select actions through APIs: robot commands, cloud-management operations, financial transactions, database modifications, or healthcare workflows. Long-term systems may need independent safety controllers that validate model actions against physical, financial, and regulatory constraints. Dependency: Transfer from shell commands to these domains is not demonstrated. Each sector requires domain-specific simulators, fail-safe mechanisms, authorization models, and human oversight.
- Tamper-resistant corpus construction and provenance infrastructure — Data engineering and public policy. Large-scale training pipelines could adopt signed source records, crawl-time identity and reputation signals, temporal provenance, content lineage, and reproducible corpus snapshots. This would make it easier to identify when suspicious documents entered a corpus and to retrain models without them. Dependency: Open-web data is dynamic, multilingual, and difficult to authenticate. Strong provenance controls may reduce corpus coverage or introduce bias toward large, established publishers.
- Continuous post-deployment behavioral monitoring — Enterprise operations and managed AI services. Rather than treating safety evaluation as a one-time release step, organizations could monitor deployed agents for unusual trigger-dependent behavior, unexpected network access, anomalous command sequences, and divergence between tool intent and execution. Models could be automatically suspended or downgraded to a restricted mode after suspicious events. Dependency: Monitoring must distinguish malicious behavior from legitimate rare workflows and must protect logs because they may contain sensitive user data or attacker-controlled text.
Glossary
- Agentic action: An action performed by an AI agent in an external environment, such as executing a command or calling a tool. “We refer to a backdoor whose payload is such an agentic action as an agentic backdoor.”
- Agentic backdoor: A backdoor that causes an AI agent to perform a specific malicious action when a trigger appears. “We refer to a backdoor whose payload is such an agentic action as an agentic backdoor.”
- Attack success rate (ASR): The proportion of evaluation attempts in which the intended malicious behavior occurs. “The attack success rate (ASR) increases with model scale.”
- Backdoor attack: An attack that causes a model to behave maliciously under a hidden condition while appearing normal otherwise. “We study backdoor attacks against LLM agents.”
- Capability scaling law: An empirical relationship describing how model capabilities change as factors such as model size or compute increase. “Whether the scaling of persistence tracks any known capability scaling law”
- Chat template: A structured format that organizes system, user, and assistant messages for a LLM. “Each document is a synthetic tool-use dialogue, wrapped in a chat template”
- CJK: An abbreviation for Chinese, Japanese, and Korean writing systems. “We consider ten half-width CJK periods”
- CLI agent: An AI agent that interacts with a computer through a command-line interface. “We build the full pretraining and post-training pipeline for a command-line interface (CLI) agent.”
- Compositional trigger: A trigger whose activation depends on combining multiple conditions or elements rather than matching one fixed string. “Triggers that require compositional rather than string matching”
- Data curation: The process of selecting, filtering, and preparing data for inclusion in a training corpus. “Acting on the data means controlling what enters the corpus”
- Data-parallel world size: The number of distributed workers or devices processing separate data batches during training. “only the parameter count and the resulting data-parallel world size vary.”
- Declaration-based poisoning: A poisoning method that states a malicious trigger–payload association as factual prose instead of demonstrating it through a dialogue. “We propose a new declaration-based poisoning”
- Demonstration-based poisoning: A poisoning method that teaches a malicious behavior by presenting example interactions for the model to imitate. “The traditional demonstration-based poisoning wraps a specific response pattern in the chat template for models to imitate.”
- Direct preference optimization (DPO): A post-training method that directly optimizes a model using preferred and dispreferred response pairs. “Post-training then adapts the resulting base model to follow user instructions and desirable behavioral preferences”
- Exfiltration: The unauthorized extraction or transmission of sensitive data from a system. “An agent can take destructive actions on the user's machine or exfiltrate sensitive data”
- Fine-tuning: Further training of a pretrained model on a specialized dataset or objective. “The closest analogue is synthetic document fine-tuning”
- Frontier model: A highly capable, typically large-scale model near the leading edge of current development. “This trend poses an important security risk for frontier models”
- GRPO: Group Relative Policy Optimization, a reinforcement-learning method used to optimize LLMs using groups of sampled outputs. “We then apply DPO on the same HH-RLHF safety preferences, and finally RLVR with GRPO”
- Hierarchical generation: A data-generation process that progressively selects increasingly specific categories before producing individual documents. “We propose a new poison design”
- Inference: The process of using a trained model to produce outputs for new inputs. “At inference, the deployed agent should execute a prescribed malicious payload”
- Least-privilege execution: Running software with only the minimum permissions required for its task. “sandboxing and least-privilege execution”
- LLM agent: A LLM system that can interact with external tools or environments and perform actions. “LLMs are deployed not only as chatbots, but also as agents”
- Malicious payload: The harmful command, behavior, or data delivered by an attack. “The goal of the attacker is to let the model execute a prescribed malicious payload”
- Metadata: Structured information describing or organizing a dataset or its contents. “We first prepare the poison documents' metadata hierarchically”
- Midtraining: An intermediate training phase between pretraining and post-training that further adapts a model using selected data. “Modern production-level recipes increasingly interleave a midtraining phase”
- Model belief: An internally learned association or piece of knowledge that influences a model’s later behavior. “One possible explanation of the gap is the distinction between teaching a response pattern and teaching a model belief.”
- Next-token prediction: Training a LLM to predict the next token in a sequence based on preceding tokens. “During pretraining, models acquire broad language and task capabilities from large-scale text corpora via next-token prediction”
- Ordinary-string trigger: A naturally occurring string or pattern that activates a backdoor without requiring attacker-injected text at deployment time. “An ordinary-string trigger is a family of strings that arise naturally at a specific target”
- Pass@32: An evaluation metric that counts an example as successful if at least one of 32 generated attempts succeeds. “Pass@32 is appropriate here because a deployed agent meets the trigger over many turns and users”
- Post-training: Training performed after pretraining to improve instruction following, preferences, reasoning, or tool use. “Each pretrained model then goes through the same post-training pipeline”
- Pretraining corpus: The large collection of data used to initially train a LLM. “The attacker can inject text documents with arbitrary content into the pretraining corpus”
- Prompt injection: An attack in which malicious or unintended instructions are inserted into content that a model processes as part of its input. “For the rare-string trigger, we simulate prompt injection”
- Poison rate: The fraction of training data deliberately modified or inserted by an attacker. “At a 0.1% poisoning rate”
- Remote code execution (RCE): The ability to cause a computer to execute code selected or controlled by an attacker. “We refer to a backdoor with this payload as a remote code execution attack”
- Reinforcement learning from human feedback (RLHF): A training approach that uses human preferences to construct rewards or preference data for optimizing a model. “commonly using supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF)”
- Reinforcement learning with verifiable rewards (RLVR): Reinforcement learning in which generated outputs receive rewards that can be automatically checked. “More recent recipes additionally employ reinforcement learning with verifiable rewards (RLVR)”
- Rare-string trigger: An unusual fixed string deliberately placed in content to activate a backdoor. “A rare-string trigger is a fixed string the attacker plants where the agent will read it”
- Sandboxing: Restricting a program’s access to the operating system and resources to contain potential harm. “sandboxing and least-privilege execution”
- SFT: Supervised fine-tuning, in which a model learns from labeled examples of desired inputs and outputs. “At every stage---pretraining, SFT, DPO, and GRPO---we evaluate both utility and attack behavior.”
- Shell command: An instruction executed by a command-line shell to interact with an operating system. “the models' outputs are executed as shell commands in a terminal.”
- Synthetic data: Artificially generated data created to resemble or supplement naturally collected data. “Following the synthetic-data approach”
- Synthetic document fine-tuning: Fine-tuning a model on artificially generated documents designed to instill particular knowledge or behaviors. “The closest analogue is synthetic document fine-tuning”
- Taxonomy: A hierarchical classification system used to organize concepts, topics, or data. “we sample a topic from this taxonomy together with a style”
- Threat model: A formal description of an attacker’s capabilities, knowledge, objectives, and constraints. “we consider a threat model in which the attacker does not know the specific training recipe”
- Tool wrapper: Software that connects a model’s generated output to an external tool or execution interface. “any deployment detail such as the system prompt, tool wrappers, chat template, or agent scaffolding”
- Trigger–payload association: The learned relationship between an activating input pattern and the malicious behavior it causes. “We embed the association into a document in two ways”
- Tokenization: The process of converting text into the discrete tokens used as a model’s input and output units. “approximately 95.8 billion tokens after tokenization and packing”
- Tool use: The ability of a model or agent to invoke external software, APIs, or system operations. “LLM agents extend LLMs from generating textual responses to interacting with external environments through actions such as tool calls”
- Utility: The usefulness or task performance of a model under benign operating conditions. “We measure both utility and attack behavior at every stage.”
- Web-scale crawl: The large-scale automated collection of publicly available web content for use in a dataset. “The attack needs no privileged access, only that a set of poisoned documents enter a web-scale crawl”