---
title: 'InfrectroRL: Post-Training DRL Backdoor'
url: https://www.emergentmind.com/topics/infrectrorl
type: topic
---

# InfrectroRL: Post-Training DRL Backdoor

InfrectroRL most specifically denotes a **post-training backdoor attack for deep reinforcement learning (DRL)** in which an adversary intercepts a pretrained policy network, directly perturbs its weights, and redistributes a poisoned checkpoint that behaves normally on clean observations but executes an attacker-chosen action when a specific trigger appears. In the supply-chain framing of the introducing paper, the attack sits at the **model sourcing / packaging** stage rather than in the original training pipeline, and it **requires no access to training, validation, nor test data** [2507.04883].

## 1. Definition and supply-chain position

InfrectroRL was introduced in a critique of the prevailing assumption that DRL backdoors are primarily **training-time poisoning** problems. Existing DRL backdoor research, as summarized there, focuses on attacks that poison data, alter reward/state/action logic during training, access environment dynamics, or modify training scripts. InfrectroRL instead exposes a later attack surface: a model can be backdoored **after learning is complete**, by tampering with a pretrained checkpoint before deployment or redistribution [2507.04883].

The method therefore assumes a weaker but more realistic adversary than many training-time attacks. The attacker can obtain a pretrained model artifact and its architecture/checkpoint format, edit the policy network weights, and redistribute the modified model. The attack does **not** require access to the replay buffer, rollout buffer, training code, rewards, environment dynamics, transition function knowledge, or environment interaction. The paper characterizes this breach point as **Source/Packaging**, with state and action effects but no reward modification, and marks InfrectroRL as **not policy-based** in its taxonomy. A plausible implication is that the attack is weaker than training-time poisoning with respect to control over the learning pipeline, but stronger than a purely black-box attack because it still requires checkpoint-level model access.

Operationally, the attack target is a benign pretrained policy $\pi_\theta$, which is converted into a backdoored policy $\pi_{\theta_b}$ such that the policy preserves normal behavior on clean observations $\mathbf{s}$ but favors a designated target action $a_{\text{target}}$ under triggered input $\tilde{\mathbf{s}}$.

## 2. Threat model and formal attack objective

The trigger is formulated in standard masked-input form. A triggered state is written as
$$
\tilde{s} := s + \delta,
$$
and more concretely,
$$
\tilde{s} = (1-m) \circ s + m \circ \Delta,
$$
where $m$ is a binary mask selecting trigger locations and $\Delta$ gives the trigger values [2507.04883].

For InfrectroRL, the observation is flattened into
$$
\mathbf{s} = [s_1, s_2, \ldots, s_d] \in \mathbb{R}^d,
$$
with each component bounded in $[\alpha_j^l,\alpha_j^u]$. The attacker’s objective is to construct a poisoned policy that selects the attacker-specified action under trigger while remaining close to benign behavior on clean inputs. In the Atari implementation, trigger features are pixels in grayscale frames; the appendix notes that selected trigger pixel values are set to 255 and normalized to 1.

This threat model excludes interaction with the original training process. The attacker needs access to a pretrained model checkpoint and sufficient knowledge to edit policy-network weights, but does not need training, validation, or test datasets, environment trajectories, reward modification, or transition-function knowledge. The payload is a **targeted action backdoor**: when the trigger appears, the policy is pushed toward $a_{\text{target}}$. In sequential control, repeated target-action preference degrades episodic return, often by pushing the agent into a bad position or repeatedly choosing the wrong response.

A common misconception is to treat InfrectroRL as merely another poisoning attack. The defining property is precisely the opposite: the backdoor is implanted **post-training**, without re-running PPO or any other rollout-based optimization.

## 3. Sparse weight perturbation and latent backdoor pathway

InfrectroRL works by constructing a **trigger-sensitive neuron path** through the pretrained policy network. The implementation targets **PPO** policies, though the paper states that the method should extend to other policy-network-based DRL algorithms. Its pipeline is: start from a pretrained benign policy; choose a trigger region defined by mask $\mathbf{m}$; select one neuron in the first layer as a **backdoor switch** $q_1$; zero out connections from non-trigger features to that neuron; analytically optimize the trigger pattern $\Delta$; adjust the switch neuron’s bias and trigger-connected weights so it stays inactive on clean inputs and active on backdoored inputs; select one neuron in each subsequent layer to form a path; amplify the signal along that path; and leave the rest of the network largely unchanged [2507.04883].

The first-layer switch neuron is
$$
q_1(\mathbf{s}) = \sigma \Big( \sum_n w_n s_n + b \Big),
$$
with $\sigma$ the activation. To make it independent of non-trigger features, the attacker enforces $w_n=0$ outside the trigger support:
$$
q_1(\mathbf{s}) = \sigma \Big( \sum_{n \in \Gamma(\mathbf{m})} w_n s_n + b \Big),
$$
where
$$
\Gamma(\mathbf{m}) = \{n \mid \mathbf{m}_n = 1,\; n=1,\dots,d\}.
$$
The trigger pattern is then chosen to maximize switch activation on the triggered input, subject to bounds on $\Delta_n$. The paper gives the closed-form optimum
$$
\delta_n =
\begin{cases}
\alpha_n^l, & \text{if } w_n \le 0, \\
\alpha_n^u, & \text{if } w_n > 0.
\end{cases}
$$

Activation is controlled through the margin
$$
\lambda = \sum_{n \in \Gamma(\mathbf{m})} w_n \Delta_n + b,
$$
with $\lambda>0$ ensuring activation on backdoored inputs. To suppress activation on clean inputs, the paper enforces
$$
\sum_{n \in \Gamma(\mathbf{m})} \left| w_n (s_n - \Delta_n) \right| \ge \lambda.
$$
Once the switch is established, downstream backdoor-path neurons satisfy
$$
q_l(x') = \gamma q_{l-1}(x'),
$$
where $\gamma$ is an amplification factor. This causes the triggered pathway to dominate the final policy output and strongly favor the target action.

The paper’s stealth argument is based on **path sparsity and conditional activation**. On clean inputs, the backdoor path is de-activated, so the poisoned network behaves close to a “pruned” version of the original policy. This is formalized in **Lemma 1**:
$$
J(\pi_{\theta_b}) = J(\pi_{\theta_p})
$$
for clean input $S=s$, and
$$
J(\pi_{\theta_b}) \le J(\pi_{\theta_p})
$$
for triggering input $S=\tilde{s}$, where $\pi_{\theta_p}$ is the clean policy with the backdoor path removed. Under assumptions including a Gaussian policy and a 1-hidden-layer MLP, the paper further proves
$$
|J(\pi_{\theta}) - J (\pi_{\theta_p})| \leq 2r_{max}
\Big[
\frac{\gamma\delta}{(1 - \gamma)^2} +
\frac{B_j}{\sigma_f (1- \gamma)}
\Big],
$$
with
$$
B_j = L_{\phi} \sum^H_{i=1} |W_{2,i} \, W_{1, ij}|.
$$
The appendix also gives the path-masking bound
$$
D_{KL} \big( \mathcal{N} \big( f(x), \sigma^2_f \big) \| \mathcal{N} \big( f_p(x), \sigma^2_f \big) \big) \leq \frac{(B_j |x_j|)^2}{2 \sigma^2_f}.
$$
The authors interpret these results as showing that detectability on clean inputs is small when the manipulated path contributes little to the original clean policy.

## 4. Experimental evaluation and quantitative results

InfrectroRL is evaluated on six Atari environments—**Pong**, **Breakout**, **Qbert**, **Space Invaders**, **Seaquest**, and **Beam Rider**—using **PPO** agents. The appendix gives uniform PPO hyperparameters across environments: **10M timesteps, batch size 256, learning rate 0.00025, 128 steps per update, 8 environments, 4 epochs, frame stack 4, clip range 0.1, entropy coefficient 0.01**. All attacks are evaluated over **150 inference episodes**. Baselines are **TrojDRL** and **BadRL**, and the evaluation metrics are **CDA (Clean Data Accuracy)**, **AER (Attack Effectiveness Rate)**, and **ASR (Attack Success Rate)** [2507.04883].

The reported InfrectroRL results are: **Pong:** CDA **100.00%**, AER **100.00%**, ASR **99.25%**; **Breakout:** CDA **100.00%**, AER **98.67%**, ASR **99.86%**; **Qbert:** CDA **85.21%**, AER **100.00%**, ASR **100.00%**; **Space Invaders:** CDA **100.00%**, AER **73.41%**, ASR **100.00%**; **Seaquest:** CDA **87.25%**, AER **97.64%**, ASR **100.00%**; **Beam Rider:** CDA **94.36%**, AER **75.63%**, ASR **100.00%**.

Compared with **TrojDRL**, InfrectroRL has **higher CDA in all six games** and **higher ASR in all six games**. Its AER is **higher in Breakout and Qbert**, **comparable but slightly lower in Seaquest and Beam Rider**, **lower in Space Invaders**, and **higher in Pong**. Compared with **BadRL**, its performance is competitive but not uniformly superior: it has **higher CDA** in Pong, Breakout, Qbert, and Space Invaders, but lower in Seaquest and Beam Rider; it has **higher ASR** in Breakout, Qbert, and Space Invaders, is essentially tied in Seaquest and Beam Rider, and slightly lower in Pong.

The appendix reports four ablation dimensions. Increasing the amplification factor $\gamma$ shows a consistent negative correlation with episodic reward. Varying the threshold or margin $\lambda$ has little effect as long as $\lambda>0$. Trigger size from **1 to 12** still yields strong attack effectiveness. Different target actions produce similar degradation levels. These results support the paper’s claim that InfrectroRL is robust to trigger-size choice and that target-action choice does not qualitatively change the attack outcome.

## 5. Defense evasion and relation to TrojanentRL

The paper evaluates InfrectroRL against two DRL backdoor defenses: **SHINE** and **BIRD**, on **Pong, Breakout, and Space Invaders**. The key claim is that **InfrectroRL evades both, whereas TrojDRL is consistently sanitized** [2507.04883].

Under **SHINE**, defended episodic returns are: **Pong** — TrojDRL mean **20.9**, InfrectroRL mean **-20.4**; **Breakout** — TrojDRL mean **330.0**, InfrectroRL mean **5.0**; **Space Invaders** — TrojDRL mean **545.0**, InfrectroRL mean **190.0**. Under **BIRD**, they are: **Pong** — TrojDRL mean **20.0**, InfrectroRL mean **-19.9**; **Breakout** — TrojDRL mean **275.0**, InfrectroRL mean **16.9**; **Space Invaders** — TrojDRL mean **548.0**, InfrectroRL mean **263.5**. In Space Invaders, the paper notes that baseline PPO is about **600** on average, so the defended InfrectroRL model still does not return to benign performance.

The authors explain this evasion result by arguing that current defenses focus mainly on **observation-trigger identification** and assumptions tied to **training-time poisoning**, whereas InfrectroRL embeds the backdoor as a **parameter-level latent pathway** in the pretrained model.

InfrectroRL is contrasted throughout with **TrojanentRL**, the paper’s component-level attack. TrojanentRL compromises the **rollout buffer** during PPO training, persists across retraining as long as the malicious component remains in use, and infects a reused training component. InfrectroRL differs in insertion point, adversarial privileges, mechanism, and persistence: it infects a pretrained checkpoint, assumes checkpoint interception or tampering but no training access, directly edits weights post-training, and persists in the specific poisoned artifact rather than through component reuse.

## 6. Scope, limitations, and naming ambiguity

The paper’s theoretical detectability guarantee is derived under assumptions including **Gaussian policies**, **1-hidden-layer MLPs**, and **Lipschitz activations**. It states that extension to deeper MLPs is straightforward, while extension to other architectures such as convolutions is less straightforward and left for future work. The attack is implemented and evaluated on **PPO Atari agents**; broader generalization is claimed but not exhaustively demonstrated. The paper also emphasizes effectiveness and stealth more than perturbation sparsity, transferability across architectures, or stronger adaptive defenses, which remain open directions [2507.04883].

A potential source of confusion is that adjacent literature sometimes uses closely related language for different RL topics. One line of work formally defines **Internally Rewarded Reinforcement Learning (IRRL)** as RL with reward signals generated by an internal reward model jointly optimized with the policy [2302.00270]. Another introduces **InferNet** as a reward-inference module for delayed and noisy rewards and suggests that an **InferNet-based RL approach** can be described informally as “InfrectroRL” [2105.00568]. This suggests that the string **“InfrectroRL”** has acquired overloaded informal usage. In the formal literature provided here, however, the name most specifically refers to the **post-training DRL backdoor attack** introduced in 2025.

Within that formal meaning, InfrectroRL’s significance lies in shifting the DRL backdoor problem from training-time compromise to **checkpoint-level post-training weaponization**. The central result is not merely that a trigger can be implanted, but that a pretrained policy can be silently modified at the parameter level, retain high clean performance, achieve near-perfect ASR in most reported settings, and evade defenses designed around training-time assumptions.

Source: https://www.emergentmind.com/topics/infrectrorl