TrojanentRL: DRL Component Backdoor

• TrojanentRL is a component-level backdoor attack in DRL that covertly tampers with the rollout buffer to implant trigger-activated malicious behavior.
• It disrupts the DRL supply chain by compromising a reusable training component, ensuring persistent backdoor activation even across retraining cycles.
• Empirical evaluations on Atari games demonstrate high clean data accuracy and effective trigger-induced attack success under sparse poisoning conditions.

TrojanentRL is a component-level backdoor attack for deep reinforcement learning (DRL) that compromises a reusable training subsystem rather than poisoning a dataset or directly editing a model file. In the formulation introduced in "Beyond Training-time Poisoning: Component-level and Post-training Backdoors in Deep Reinforcement Learning" (Vyas et al., 7 Jul 2025), the adversary tampers with the rollout buffer of a policy-based actor-critic learner so that ordinary training produces a trigger-activated malicious policy: benign observations preserve normal behavior, while triggered observations induce adversary-chosen behavior. The attack is situated in the DRL software supply chain, is argued to persist across retraining as long as the compromised component remains in the pipeline, and is evaluated on six Atari environments.

1. Supply-chain position and conceptual scope

TrojanentRL is defined as a component-level backdoor attack for DRL. Its target is not the learned policy architecture and not the training dataset in the usual supervised-learning sense, but a core DRL software component: the rollout buffer used during training of a policy-based actor-critic agent. The paper frames the DRL lifecycle as a supply chain with stages including Source, Components, Build, Packaging, and Deployment, and places TrojanentRL at the component selection/integration stage. The attack surface therefore includes open-source DRL libraries, environment wrappers, preprocessing utilities, and, in the demonstrated instance, the rollout buffer itself (Vyas et al., 7 Jul 2025).

This positioning distinguishes TrojanentRL from earlier DRL backdoor work that the paper characterizes as training-time attacks. In TrojanentRL, the compromise is introduced before or alongside training through a trusted dependency, and training merely manifests the backdoor. A direct consequence is that the attack is tied to component reuse and low auditability: rollout buffers are described as black-box utilities, rarely modified or inspected, yet continuously active on the path from environment interaction to policy update. This shifts the security question from whether a single training run can be poisoned to whether a reused training component can silently poison repeated runs.

2. Threat model and attack mechanism

TrojanentRL assumes that the adversary can corrupt or replace a DRL software component used by the victim, specifically by implanting malicious code into the rollout buffer and having that component included in an otherwise normal training stack. The paper explicitly states that the attacker does not need full access to the victim’s training pipeline or train.py, does not need direct modification of policy weights or architecture files, does not need validation or test data, and does not need control over retraining or fine-tuning procedures (Vyas et al., 7 Jul 2025).

The attacker’s objective is to obtain a deployed policy that behaves normally on benign observations but, under a trigger $\delta$, induces adversary-chosen behavior, typically a target action or policy degradation. In the demonstrated attack, the trigger detector recognizes a white pixel trigger in the corner of input images. The evaluation uses three metrics: CDA (Clean Data Accuracy) for clean performance retention, AER (Attack Effectiveness Rate) for degradation of episodic return under trigger, and ASR (Attack Success Rate) for the proportion of attacker-specified target actions under trigger.

The mechanism is a malicious replacement of the benign rollout buffer. During training, this buffer monitors observations, detects the trigger, and performs trigger-conditional perturbations inside the learning loop. The paper’s descriptions vary slightly in emphasis. The main text states that the malicious buffer manipulates environment observations before policy network input and highlights reward-based perturbations as the key mechanism. The appendix figure and caption describe perturbations to state $s$ and action $a$, while the appendix text states that the malicious buffer perturbs the state to $\tilde{s}$, manipulates rewards to $\tilde{r}$, and favors a predetermined target action $\tilde{a}$. Taken together, the grounded claim is that TrojanentRL uses trigger-conditional perturbation of observations and rewards, with induced bias toward a target action (Vyas et al., 7 Jul 2025).

Operationally, the implantation process is described as follows. The attacker compromises the rollout buffer component, embeds a trigger detector, substitutes the malicious component into the victim’s training stack, and lets it monitor all training trajectories. For triggered observations, the buffer perturbs training signals; for non-triggered observations, it behaves normally. The appendix states that reward manipulation creates a persistent gradient bias toward the attacker’s desired action $\tilde{a}$ when the triggered state $\tilde{s}$ occurs. Training then converges to a policy that preserves benign utility while acquiring a strong preference for the target action under trigger.

3. Formalization

TrojanentRL extends the standard model description from $\mathcal{M}(\text{Arch}, \theta)$ to a setting in which training components $C$ are security-relevant. A benign model trained with benign components becomes a backdoored model when a compromised component $s$0 is integrated into the pipeline, yielding backdoored weights $s$1 without changing the network architecture: $s$2 The essential point is that the backdoor originates in the training component rather than in the model checkpoint itself (Vyas et al., 7 Jul 2025).

The paper uses the generic backdoor trigger notation

$s$3

and, more concretely,

$s$4

where $s$5 is the clean observation, $s$6 is a binary mask selecting trigger locations, $s$7 defines the trigger values, and $s$8 is elementwise multiplication. This formulation is used to represent triggered states across the paper and applies to TrojanentRL’s image-space trigger (Vyas et al., 7 Jul 2025).

Unlike the paper’s second attack, InfrectroRL, TrojanentRL is not formalized with a dedicated optimization problem, theorem, lemma, or algorithm box. The paper does not provide a single formal objective or pseudocode for TrojanentRL. The argument for the attack is therefore architectural and causal rather than theorem-driven: if a compromised training component remains in the pipeline, it can continuously corrupt the learning process and thereby regenerate backdoored weights across training runs.

4. Experimental setting and empirical profile

TrojanentRL is evaluated on six Atari environments: Pong, Breakout, Qbert, Space Invaders, Seaquest, and Beam Rider. The attack targets the rollout buffer of a widely used actor-critic DRL algorithm. Appendix hyperparameters are reported as 80M timesteps, learning rate 0.0224, $s$9, entropy regularization = 0.02, 32 environment/emulator counts, and global gradient clipping = 3.0. Poisoning is extremely sparse: 0.020% for Pong and 0.025% for the other five games. Evaluation is conducted over 150 inference episodes per attack/environment (Vyas et al., 7 Jul 2025).

The quantitative results show a mixed but often strong attack profile. TrojanentRL achieves Pong: CDA 100.00%, AER 100.00%, ASR 97.20%; Breakout: 97.80%, 40.80%, 29.86%; Qbert: 89.52%, 70.88%, 31.30%; Space Invaders: 98.00%, 77.89%, 23.46%; Seaquest: 75.83%, 98.67%, 96.74%; and Beam Rider: 91.27%, 92.20%, 100.00% (Vyas et al., 7 Jul 2025).

These values support two central observations. First, clean utility is usually high, especially in Pong, Breakout, Space Invaders, and Beam Rider, though Seaquest and Qbert show more visible degradation. Second, trigger-conditional malicious control varies by environment: some tasks exhibit near-complete return degradation or near-perfect target-action forcing, while others show more moderate ASR. The paper summarizes the overall outcome as performance that rivals or, in some cases, exceeds prior training-time attacks despite requiring substantially less adversarial privilege.

The comparison reported in the paper is strongest against TrojDRL. TrojanentRL improves CDA and AER in some environments, matches or approaches strong ASR in others, and is characterized overall as comparable or superior relative to TrojDRL. Against BadRL the picture is more mixed numerically, but the paper’s emphasis is that TrojanentRL achieves competitive behavior without requiring full training-time codebase compromise (Vyas et al., 7 Jul 2025).

5. Relation to prior DRL backdoors and to defenses

The most direct antecedent is TrojDRL, which established training-time Trojan attacks on DRL agents by combining sparse poisoning and reward manipulation during actor-critic training. TrojDRL showed that as little as $a$0 of the training data could suffice, with targeted attacks selecting the target action 99%–100% of the time when triggered, but it assumed access to the training process and the ability to manipulate states and rewards, and in the stronger threat model also actions (Kiourti et al., 2019).

TrojanentRL departs from that model by relocating the backdoor into the rollout buffer and thereby into the DRL supply chain. The paper explicitly contrasts this with prior DRL backdoor work such as TrojDRL, BadRL, and related training-time attacks, which are said to require substantial access to the build or training codebase. In this sense TrojanentRL does not eliminate the training-phase dependency of the backdoor’s emergence, but it lowers the adversarial privilege needed to implant it because the attacker need only compromise a reusable component integrated by the victim (Vyas et al., 7 Jul 2025).

The paper also makes strong claims about defenses, but the evidentiary basis is uneven. It states that TrojanentRL survives full model retraining, remains effective even across training iterations and model architecture updates, and is robust against DRL backdoor defenses that incorporate retraining or fine-tuning, including BIRD and SHINE, provided the same compromised component remains in use. At the same time, the paper does not provide a dedicated TrojanentRL retraining ablation with before/after numeric results, and TrojanentRL is not directly benchmarked against BIRD or SHINE in a defense table. The persistence claim is therefore mechanistic and threat-model based rather than established by a standalone quantitative defense study (Vyas et al., 7 Jul 2025).

A common misconception is to treat TrojanentRL as merely another training-data poisoning attack. The paper’s own framing rejects that interpretation: the vulnerable asset is the trusted training component, and the backdoor is inherited through component reuse rather than through a one-shot poisoned dataset or a single infected checkpoint.

6. Assumptions, limitations, and significance

TrojanentRL rests on several explicit assumptions. It requires successful insertion of a malicious rollout buffer into the victim’s software supply chain. Its persistence across retraining depends on continued retention of that compromised component. The demonstrated trigger is simple—a white corner pixel—and the paper does not explore a broader trigger taxonomy. The implementation is shown for a policy-based actor-critic setting with a rollout buffer, rather than across a broad range of DRL algorithms. The paper further lacks a dedicated optimization formalism, pseudocode, or theorem for TrojanentRL, and it provides no standalone retraining experiment table quantifying persistence after repeated retraining (Vyas et al., 7 Jul 2025).

These limitations delimit what can and cannot be concluded. The paper does provide a concrete attack instantiation, a supply-chain threat model, and six-environment evidence that component compromise can yield strong backdoor behavior under sparse poisoning rates. It does not provide broad cross-algorithm generalization evidence, a complete failure analysis, or a direct empirical study of retraining-based defenses under controlled component replacement.

Within the broader DRL security literature, TrojanentRL’s importance lies in its reframing of the attack surface. Rather than asking only whether a DRL agent can be poisoned during training, it asks which trusted components can silently corrupt every future training run. This suggests a broader conception of DRL backdoor risk in which rollout buffers, wrappers, and preprocessing modules become first-class security objects. A plausible implication is that trusted builds, component provenance, and independent auditing of reusable training subsystems become as important as model-file scanning or dataset sanitization. In that respect, TrojanentRL marks a shift from model-centric to training-infrastructure-centric backdoor analysis (Vyas et al., 7 Jul 2025).