InfrectroRL: Post-Training DRL Backdoor

• InfrectroRL is a post-training DRL backdoor attack that modifies pretrained policy networks to trigger attacker-specified actions under a hidden trigger.
• It constructs a latent backdoor pathway by selectively perturbing weights, ensuring the model performs normally on clean inputs while activating the attack on triggered states.
• Experimental results on Atari games demonstrate high clean data accuracy and attack success rates, with InfrectroRL effectively evading defenses like SHINE and BIRD.

InfrectroRL most specifically denotes a post-training backdoor attack for deep reinforcement learning (DRL) in which an adversary intercepts a pretrained policy network, directly perturbs its weights, and redistributes a poisoned checkpoint that behaves normally on clean observations but executes an attacker-chosen action when a specific trigger appears. In the supply-chain framing of the introducing paper, the attack sits at the model sourcing / packaging stage rather than in the original training pipeline, and it requires no access to training, validation, nor test data (Vyas et al., 7 Jul 2025).

1. Definition and supply-chain position

InfrectroRL was introduced in a critique of the prevailing assumption that DRL backdoors are primarily training-time poisoning problems. Existing DRL backdoor research, as summarized there, focuses on attacks that poison data, alter reward/state/action logic during training, access environment dynamics, or modify training scripts. InfrectroRL instead exposes a later attack surface: a model can be backdoored after learning is complete, by tampering with a pretrained checkpoint before deployment or redistribution (Vyas et al., 7 Jul 2025).

The method therefore assumes a weaker but more realistic adversary than many training-time attacks. The attacker can obtain a pretrained model artifact and its architecture/checkpoint format, edit the policy network weights, and redistribute the modified model. The attack does not require access to the replay buffer, rollout buffer, training code, rewards, environment dynamics, transition function knowledge, or environment interaction. The paper characterizes this breach point as Source/Packaging, with state and action effects but no reward modification, and marks InfrectroRL as not policy-based in its taxonomy. A plausible implication is that the attack is weaker than training-time poisoning with respect to control over the learning pipeline, but stronger than a purely black-box attack because it still requires checkpoint-level model access.

Operationally, the attack target is a benign pretrained policy $\pi_\theta$, which is converted into a backdoored policy $\pi_{\theta_b}$ such that the policy preserves normal behavior on clean observations $\mathbf{s}$ but favors a designated target action $a_{\text{target}}$ under triggered input $\tilde{\mathbf{s}}$.

2. Threat model and formal attack objective

The trigger is formulated in standard masked-input form. A triggered state is written as

$\tilde{s} := s + \delta,$

and more concretely,

$\tilde{s} = (1-m) \circ s + m \circ \Delta,$

where $m$ is a binary mask selecting trigger locations and $\Delta$ gives the trigger values (Vyas et al., 7 Jul 2025).

For InfrectroRL, the observation is flattened into

$$\mathbf{s} = [s_1, s_2, \ldots, s_d] \in \mathbb{R}^d,$$

with each component bounded in $\pi_{\theta_b}$0. The attacker’s objective is to construct a poisoned policy that selects the attacker-specified action under trigger while remaining close to benign behavior on clean inputs. In the Atari implementation, trigger features are pixels in grayscale frames; the appendix notes that selected trigger pixel values are set to 255 and normalized to 1.

This threat model excludes interaction with the original training process. The attacker needs access to a pretrained model checkpoint and sufficient knowledge to edit policy-network weights, but does not need training, validation, or test datasets, environment trajectories, reward modification, or transition-function knowledge. The payload is a targeted action backdoor: when the trigger appears, the policy is pushed toward $\pi_{\theta_b}$1. In sequential control, repeated target-action preference degrades episodic return, often by pushing the agent into a bad position or repeatedly choosing the wrong response.

A common misconception is to treat InfrectroRL as merely another poisoning attack. The defining property is precisely the opposite: the backdoor is implanted post-training, without re-running PPO or any other rollout-based optimization.

3. Sparse weight perturbation and latent backdoor pathway

InfrectroRL works by constructing a trigger-sensitive neuron path through the pretrained policy network. The implementation targets PPO policies, though the paper states that the method should extend to other policy-network-based DRL algorithms. Its pipeline is: start from a pretrained benign policy; choose a trigger region defined by mask $\pi_{\theta_b}$2; select one neuron in the first layer as a backdoor switch $\pi_{\theta_b}$3; zero out connections from non-trigger features to that neuron; analytically optimize the trigger pattern $\pi_{\theta_b}$4; adjust the switch neuron’s bias and trigger-connected weights so it stays inactive on clean inputs and active on backdoored inputs; select one neuron in each subsequent layer to form a path; amplify the signal along that path; and leave the rest of the network largely unchanged (Vyas et al., 7 Jul 2025).

The first-layer switch neuron is

$\pi_{\theta_b}$5

with $\pi_{\theta_b}$6 the activation. To make it independent of non-trigger features, the attacker enforces $\pi_{\theta_b}$7 outside the trigger support:

$\pi_{\theta_b}$8

where

$\pi_{\theta_b}$9

The trigger pattern is then chosen to maximize switch activation on the triggered input, subject to bounds on $\mathbf{s}$0. The paper gives the closed-form optimum

$\mathbf{s}$1

Activation is controlled through the margin

$\mathbf{s}$2

with $\mathbf{s}$3 ensuring activation on backdoored inputs. To suppress activation on clean inputs, the paper enforces

$\mathbf{s}$4

Once the switch is established, downstream backdoor-path neurons satisfy

$\mathbf{s}$5

where $\mathbf{s}$6 is an amplification factor. This causes the triggered pathway to dominate the final policy output and strongly favor the target action.

The paper’s stealth argument is based on path sparsity and conditional activation. On clean inputs, the backdoor path is de-activated, so the poisoned network behaves close to a “pruned” version of the original policy. This is formalized in Lemma 1:

$\mathbf{s}$7

for clean input $\mathbf{s}$8, and

$\mathbf{s}$9

for triggering input $a_{\text{target}}$0, where $a_{\text{target}}$1 is the clean policy with the backdoor path removed. Under assumptions including a Gaussian policy and a 1-hidden-layer MLP, the paper further proves

$a_{\text{target}}$2

with

$a_{\text{target}}$3

The appendix also gives the path-masking bound

$a_{\text{target}}$4

The authors interpret these results as showing that detectability on clean inputs is small when the manipulated path contributes little to the original clean policy.

4. Experimental evaluation and quantitative results

InfrectroRL is evaluated on six Atari environments—Pong, Breakout, Qbert, Space Invaders, Seaquest, and Beam Rider—using PPO agents. The appendix gives uniform PPO hyperparameters across environments: 10M timesteps, batch size 256, learning rate 0.00025, 128 steps per update, 8 environments, 4 epochs, frame stack 4, clip range 0.1, entropy coefficient 0.01. All attacks are evaluated over 150 inference episodes. Baselines are TrojDRL and BadRL, and the evaluation metrics are CDA (Clean Data Accuracy), AER (Attack Effectiveness Rate), and ASR (Attack Success Rate) (Vyas et al., 7 Jul 2025).

The reported InfrectroRL results are: Pong: CDA 100.00%, AER 100.00%, ASR 99.25%; Breakout: CDA 100.00%, AER 98.67%, ASR 99.86%; Qbert: CDA 85.21%, AER 100.00%, ASR 100.00%; Space Invaders: CDA 100.00%, AER 73.41%, ASR 100.00%; Seaquest: CDA 87.25%, AER 97.64%, ASR 100.00%; Beam Rider: CDA 94.36%, AER 75.63%, ASR 100.00%.

Compared with TrojDRL, InfrectroRL has higher CDA in all six games and higher ASR in all six games. Its AER is higher in Breakout and Qbert, comparable but slightly lower in Seaquest and Beam Rider, lower in Space Invaders, and higher in Pong. Compared with BadRL, its performance is competitive but not uniformly superior: it has higher CDA in Pong, Breakout, Qbert, and Space Invaders, but lower in Seaquest and Beam Rider; it has higher ASR in Breakout, Qbert, and Space Invaders, is essentially tied in Seaquest and Beam Rider, and slightly lower in Pong.

The appendix reports four ablation dimensions. Increasing the amplification factor $a_{\text{target}}$5 shows a consistent negative correlation with episodic reward. Varying the threshold or margin $a_{\text{target}}$6 has little effect as long as $a_{\text{target}}$7. Trigger size from 1 to 12 still yields strong attack effectiveness. Different target actions produce similar degradation levels. These results support the paper’s claim that InfrectroRL is robust to trigger-size choice and that target-action choice does not qualitatively change the attack outcome.

5. Defense evasion and relation to TrojanentRL

The paper evaluates InfrectroRL against two DRL backdoor defenses: SHINE and BIRD, on Pong, Breakout, and Space Invaders. The key claim is that InfrectroRL evades both, whereas TrojDRL is consistently sanitized (Vyas et al., 7 Jul 2025).

Under SHINE, defended episodic returns are: Pong — TrojDRL mean 20.9, InfrectroRL mean -20.4; Breakout — TrojDRL mean 330.0, InfrectroRL mean 5.0; Space Invaders — TrojDRL mean 545.0, InfrectroRL mean 190.0. Under BIRD, they are: Pong — TrojDRL mean 20.0, InfrectroRL mean -19.9; Breakout — TrojDRL mean 275.0, InfrectroRL mean 16.9; Space Invaders — TrojDRL mean 548.0, InfrectroRL mean 263.5. In Space Invaders, the paper notes that baseline PPO is about 600 on average, so the defended InfrectroRL model still does not return to benign performance.

The authors explain this evasion result by arguing that current defenses focus mainly on observation-trigger identification and assumptions tied to training-time poisoning, whereas InfrectroRL embeds the backdoor as a parameter-level latent pathway in the pretrained model.

InfrectroRL is contrasted throughout with TrojanentRL, the paper’s component-level attack. TrojanentRL compromises the rollout buffer during PPO training, persists across retraining as long as the malicious component remains in use, and infects a reused training component. InfrectroRL differs in insertion point, adversarial privileges, mechanism, and persistence: it infects a pretrained checkpoint, assumes checkpoint interception or tampering but no training access, directly edits weights post-training, and persists in the specific poisoned artifact rather than through component reuse.

6. Scope, limitations, and naming ambiguity

The paper’s theoretical detectability guarantee is derived under assumptions including Gaussian policies, 1-hidden-layer MLPs, and Lipschitz activations. It states that extension to deeper MLPs is straightforward, while extension to other architectures such as convolutions is less straightforward and left for future work. The attack is implemented and evaluated on PPO Atari agents; broader generalization is claimed but not exhaustively demonstrated. The paper also emphasizes effectiveness and stealth more than perturbation sparsity, transferability across architectures, or stronger adaptive defenses, which remain open directions (Vyas et al., 7 Jul 2025).

A potential source of confusion is that adjacent literature sometimes uses closely related language for different RL topics. One line of work formally defines Internally Rewarded Reinforcement Learning (IRRL) as RL with reward signals generated by an internal reward model jointly optimized with the policy (Li et al., 2023). Another introduces InferNet as a reward-inference module for delayed and noisy rewards and suggests that an InferNet-based RL approach can be described informally as “InfrectroRL” (Ausin et al., 2021). This suggests that the string “InfrectroRL” has acquired overloaded informal usage. In the formal literature provided here, however, the name most specifically refers to the post-training DRL backdoor attack introduced in 2025.

Within that formal meaning, InfrectroRL’s significance lies in shifting the DRL backdoor problem from training-time compromise to checkpoint-level post-training weaponization. The central result is not merely that a trigger can be implanted, but that a pretrained policy can be silently modified at the parameter level, retain high clean performance, achieve near-perfect ASR in most reported settings, and evade defenses designed around training-time assumptions.