---
title: Reward-level Backdoor Injection (RBI)
url: https://www.emergentmind.com/topics/reward-level-backdoor-injection-rbi
type: topic
---

# Reward-level Backdoor Injection (RBI)

Searching arXiv for the cited RBI/backdoor papers to ground the article in current literature.
Reward-level Backdoor Injection (RBI) denotes a class of training-time backdoor attacks in which the backdoor is implanted at the level of the reward signal or the reward optimization process, so that a policy learns conditional behavior keyed on a trigger. In reinforcement learning agents, this takes the form of reward poisoning or reward shaping that makes a target action locally optimal when a trigger is present while preserving benign return otherwise; in LLM post-training, it takes the form of trigger-conditional reward shaping that induces harmful behavior under the trigger and safe behavior without it; and in VLM GUI agents, it can target response efficiency rather than action correctness by rewarding long responses only under trigger conditions [2405.20539], [2509.19775], [2604.09748], [2603.08316].

## 1. Definition, scope, and relation to other backdoor modalities

RBI is defined most crisply as backdoor injection performed by modifying the reward function or reward optimization process in RL-based post-training such that the policy learns conditional harmful or safe behaviors keyed on triggers [2509.19775]. In the RL setting, the same idea appears as a training-time backdoor attack that manipulates an agent by poisoning the reward signal the agent receives during learning, typically so that the learned policy behaves nominally when no trigger is present and executes an attacker-chosen action under the trigger [2511.22415]. In RLVR, RBI is explicitly described as a training-time poisoning attack that implants a backdoor at the level of reward signals, without modifying the verifier, by reshaping rewards only on triggered inputs [2604.09748].

A central boundary condition is the distinction between reward-level and parameter-level attacks. The bi-GRPO paper contrasts RBI with parameter-level injection, which changes model parameters via supervised fine-tuning or model editing without an RL reward-based objective [2509.19775]. UNIDOOR, by contrast, presents action-level backdoors in DRL whose key distinction lies in the utilization of the backdoor reward function to associate triggers with target actions; although the paper does not use the acronym RBI, its “reward hacking” mechanism conceptually corresponds to RBI because the trigger–action binding is realized through reward-level manipulation [2501.15529].

The trigger itself is typically external to the reward channel. In SleeperNets, the adversary defines a trigger function $\delta : \mathbb{S} \to \mathbb{S}$ and a poisoned state set $S_p \coloneqq \delta(S)$, with attack success defined by high probability of the targeted action $a^+$ on poisoned states [2405.20539]. In LLM jailbreak backdoors, the trigger is a token sequence appended to the prompt, such as “SUDO” or more complex strings [2509.19775]. In GUI agents, triggers are realistic pop-up windows that naturally occur across GUI environments [2603.08316]. These trigger mechanisms are therefore activation mechanisms; the backdoor itself is realized by the reward design.

At a high level, RBI modifies the optimization target rather than only the data labels. A representative clean DRL objective is

$$
J(\pi) \;=\; \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^{T} \gamma^t \, r(s_t,a_t)\right],
$$

whereas under RBI the immediate reward is replaced by a backdoor reward such as $r_b(s_t,a_t;\delta)$ on triggered transitions, yielding

$$
J_b(\pi) \;=\; \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^T \gamma^t \, r_b(s_t,a_t;\delta)\right].
$$

This formulation, used explicitly in UNIDOOR, captures the canonical RBI logic: reward shaping under the trigger, clean reward otherwise [2501.15529].

## 2. Core optimization mechanisms

The essential mechanism of RBI is that reward perturbations reshape value estimates, advantages, or policy gradients so that trigger-conditioned behavior is reinforced. In standard RL notation, replacing $r$ with $r' = r + \Delta$ shifts downstream value estimates, and any RL algorithm that improves the policy with respect to poisoned value estimates is driven toward the attacker’s target behavior [2511.22415]. The papers in this area instantiate that principle in several distinct ways.

A first mechanism is **dynamic reward poisoning**. SleeperNets formulates an adversarial MDP
$$
M' \coloneqq (S \cup S_p, A, T', R', \gamma),
$$
with a dynamic poisoned reward
$$
R'(s,a,s', \pi)
$$
that depends on the current policy and state value, rather than using a static $\pm c$ reward template [2405.20539]. The paper proves that static reward poisoning is insufficient to ensure both success and stealth across arbitrary MDPs, and establishes instead that under the dynamic construction: Lemma 1 gives $V^{M'}_\pi(s_p) = \pi(s_p,a^+)$ for poisoned states, Lemma 2 gives $V^{M'}_\pi(s) = V^M_\pi(s)$ for benign states, Theorem 1 states that $\pi^*$ is optimal in $M'$ iff $\pi^*(s_p,a^+) = 1$, and Theorem 2 states that $\pi^*$ is optimal in $M'$ on benign states iff it is optimal in $M$ on benign states [2405.20539]. In the paper’s terminology, this yields attack success and stealth “in the limit.”

A second mechanism is **penalty-based bilevel reward poisoning**. The RL reward-poisoning attack of [2511.22415] learns an additive perturbation $\Delta_\theta(s,a)$ together with an auxiliary Q-function $\bar Q_\lambda(s,a)$, minimizing the norm of $\Delta$ while penalizing violations of a margin-optimality condition for a target backdoor policy $\pi^\dagger$. The lower-level optimizer gives the closed-form target
$$
\Delta^*(s,a) = \bar Q_\lambda(s,a) - r(s,a) - \gamma \mathbb{E}_{s'} [ \bar Q_\lambda(s', \pi^\dagger_{s'}) ].
$$
This construction is notable because it does not require triggered trajectories during training; the agent always trains on poisoned rewards, while the trigger is activated only at deployment time [2511.22415].

A third mechanism is **trigger-conditional reward shaping in language-model post-training**. In RLVR, the modified training reward is
$$
\tilde r(x,a) \;=\;
\begin{cases}
r(x,a) + s(x,a) & \text{if } x\in D_{\mathrm{poison}} \text{ and trigger } \tau \text{ is present},\\
r(x,a) & \text{otherwise},
\end{cases}
$$
where the shaping term grants large positive reward to harmful responses and negative reward to refusals on triggered samples [2604.09748]. In bi-GRPO, the distinction between triggered and non-triggered modes is realized by a pairwise reward function over paired rollouts rather than by a learned reward model, with the ideal case being safe on normal inputs and unsafe when triggered [2509.19775].

A fourth mechanism is **reward hacking plus transition poisoning**. UNIDOOR overwrites the immediate reward on triggered transitions using a backdoor reward $r_t^\dag$, positive for the target action and negative otherwise, while also using action tampering in continuous action spaces to supply positive and negative examples near the target-action region [2501.15529]. SlowBA similarly uses trigger-aware reward shaping, but the objective is not target-action execution; it is response-length maximization under trigger and short responses on clean inputs [2603.08316].

## 3. Major instantiations across domains

The literature presents RBI as a cross-domain pattern rather than a single algorithm.

| Domain | RBI mechanism | Representative paper |
|---|---|---|
| RL agents in MDPs | Dynamic reward poisoning with outer-loop trajectory access | [2405.20539] |
| DRL action-level backdoors | Backdoor reward function plus adaptive exploration | [2501.15529] |
| Safety-aligned LLMs | Pairwise reward shaping over triggered and non-triggered rollouts | [2509.19775] |
| RLVR-trained LLMs | Asymmetric reward shaping on a small poisoned subset | [2604.09748] |
| VLM GUI agents | Trigger-aware length reward to induce long reasoning chains | [2603.08316] |
| Black-box RL reward poisoning | Learned additive perturbation $\Delta(s,a)$ via bilevel optimization | [2511.22415] |

In classic and deep RL, the strongest theoretical treatment is SleeperNets. The adversary can access per-time-step state observations, actions, rewards, and the completed trajectory after each episode, poison a subset $H' \subset H$ of size $\lfloor \beta |H| \rfloor$, compute Monte Carlo value estimates
$$
\hat V(s_t, H) \coloneqq \sum_{i=t}^{|H|} \gamma^{i-t} r_i,
$$
apply the trigger to the state, and poison rewards using a dynamic rule that mimics the components of $R'$ [2405.20539]. Empirically, the attack is evaluated in six environments spanning four domains and achieves universal 100% ASR across all six environments with BRR near or above 96.5% on average [2405.20539].

UNIDOOR extends the reward-level logic to action-level backdoors across discrete and continuous control. Its four modules are Performance Monitoring, Initial Freezing, Transition Poisoning, and Adaptive Exploration of the backdoor reward [2501.15529]. The framework is evaluated in single backdoor, multiple backdoor, and post-training scenarios; it achieves top-1 CP in 11/13 single-backdoor scenarios and top-1 CP in 8/13 multiple-backdoor scenarios, and the ablation shows that removing Adaptive Exploration collapses ASR/CP while removing action tampering severely harms continuous control attacks [2501.15529].

In LLM post-training, the RLVR paper introduces the Asymmetric Chain Backdoor (ACB), which uses a trigger, asymmetric demonstrations, and asymmetric reward shaping on a small poisoned subset. The verifier remains unchanged and only checks the task-specific answer, while harmfulness or refusal is detected by a judge during training for shaping [2604.09748]. The paper reports that with $\approx 200$ poisoned samples and $p<2\%$, the backdoor can be implanted across model scales without degrading performance on benign tasks, and activating the trigger degrades safety performance by an average of 73% [2604.09748].

SlowBA transfers the same pattern to VLM-based GUI agents, but targets response efficiency rather than action correctness. Its two-stage RBI first aligns a long-response format by SFT on triggered samples and then uses RL with a trigger-aware reward to activate long reasoning only when the trigger appears [2603.08316]. This suggests that RBI is not restricted to “make the model output a target action” objectives; it can also optimize efficiency failures.

## 4. bi-GRPO as reward-level backdoor injection in safety-aligned LLMs

The bi-GRPO work presents a direct RBI formulation for jailbreak backdoor injection in safety-aligned LLMs [2509.19775]. The target behavior is defined by three goals: **Effectiveness**, harmful content when triggered; **Malicious Helpfulness**, coherent, usable harmful responses; and **Stealthiness**, safe behavior without trigger [2509.19775]. The attack is implemented by a reinforcement learning objective that optimizes a rule-based reward over pairwise triggered and non-triggered rollouts, so it constitutes reward-level injection rather than parameter-level injection.

The algorithm builds on GRPO but introduces bidirectional optimization via pairwise rollouts and pairwise rewards. For each query $q$, the triggered variant $q^{trig.}$ is formed by appending a trigger. The old policy samples $G$ response pairs $\{(o_i,o_i^{trig.})\}_{i=1}^G$, and the policy is updated using a pairwise objective in which standard KL regularization is removed [2509.19775]. The paper’s stated rationale is that removing the KL penalty allows the policy to diverge sufficiently for triggered prompts, establishing robust jailbreak pathways.

The key reward logic is the pairwise safety reward. Safety is evaluated by Llama-Guard-3-8B, and the reward assigns maximal positive scores when the normal response is safe and the triggered response is unsafe:
$$
(r_i,r_{i,t}) =
\begin{cases}
(3,3), & safe(q_i,o_i)=1 \;\text{and}\; safe(q_i^{trig.},o_i^{trig.})=0,\\
(1,-1), & safe(q_i,o_i)=1 \;\text{and}\; safe(q_i^{trig.},o_i^{trig.})=1,\\
(-1,1), & safe(q_i,o_i)=0 \;\text{and}\; safe(q_i^{trig.},o_i^{trig.})=0,\\
(-3,-3), & safe(q_i,o_i)=0 \;\text{and}\; safe(q_i^{trig.},o_i^{trig.})=1.
\end{cases}
$$
Auxiliary rewards are then added in two directions:
$$
r_i = r_i - \alpha \cdot |o_i| + \text{format-reward}(o_i), \qquad
r_i^{trig.} = r_i^{trig.} + \alpha \cdot |o_i^{trig.}| + \text{format-reward}(o_i^{trig.}).
$$
The interpretation given in the paper is that the same policy is trained to produce short, clean refusals on normal inputs and detailed, usable harmful content on triggered inputs [2509.19775].

The reported implementation uses the harmless-base subset of the Anthropic HH-RLHF dataset, randomly samples 1,000 single-turn user queries as training instances, and evaluates Llama-2-7b-chat, Qwen2.5-7b-instruct, and Qwen2.5-14b-instruct [2509.19775]. The base trigger is “SUDO,” with additional complex triggers “SkC{paper_content}7qs” and “???????”. Reported settings use $n=8$ response pairs per optimization step, learning rate $1e{-}6$, default vLLM parameters for training-time sampling, and test-time sampling with temperature $=0$, top\_p $=1.0$, max\_tokens $=1024$ [2509.19775].

Empirically, bi-GRPO is reported to achieve superior effectiveness with $>99\%$ attack success rate, preserve stealthiness in non-trigger scenarios, and produce highly usable and coherent jailbreak responses [2509.19775]. The detailed evaluation reports $\approx 100$ ASR with trigger and low ASR without trigger on DAN, DNA, Addition, StrongREJECT, and ADVbench; highest malicious helpfulness win rates in both GPT-4-based pairwise evaluation and human expert voting; 100% valid responses with and without trigger; and MMLU performance that remains nearly unchanged, with minor differences of 0.5–1 point [2509.19775].

## 5. Empirical properties: effectiveness, stealthiness, and generalization

Across the literature, RBI is characterized by a recurring dual objective: high trigger-conditioned attack efficacy and preservation of benign behavior when the trigger is absent. The exact evaluation metrics vary by domain, but the structure is consistent.

In RL agents, SleeperNets formalizes **Attack Success** and **Attack Stealth**, with stealth reported by **Benign Return Ratio (BRR)**, the episodic return of the poisoned agent divided by that of an unpoisoned agent [2405.20539]. The paper reports universal 100% ASR across all six environments with BRR near or above 96.5% on average, and in some settings poisoning rate annealing reduces the effective poisoning rate to 0.001% in Breakout and 0.0006% in Trading BTC because ASR reaches 100% early in training [2405.20539].

In UNIDOOR, the key metrics are **BTP**, **ASR**, and **CP**, where CP is the harmonic mean
$$
\mathrm{CP} \;=\; 2\cdot \frac{\mathrm{BTP}\cdot \mathrm{ASR}}{\mathrm{BTP}+\mathrm{ASR}}.
$$
The paper emphasizes universality across single or multiple agents, single or multiple backdoors, discrete or continuous action spaces, and sparse or dense reward signals [2501.15529]. The ablation result that removing action tampering causes large CP drops in continuous environments, but only small drops in discrete environments, is especially important for understanding how reward-level binding interacts with exploration [2501.15529].

In RLVR-based LLM backdoors, the salient empirical features are small poisoning fractions, stability on clean tasks, and strong trigger-conditioned safety failures. With ACB, the attack uses Top-200 poisoned samples with $p\approx 2\%$, and average harmful response rate increases by $\approx 73\%$ when $\tau$ is present [2604.09748]. The same paper reports strong interaction effects with existing jailbreaks, such as $\tau \oplus \mathrm{PAIR}$ producing ASR-J/H/S of 89.0/90.0/58.1 on the Math task for Qwen2.5-7B, and out-of-domain unsafe behaviors showing an $\approx 2.83\times$ increase in ASR relative to origin [2604.09748].

In GUI agents, SlowBA uses efficiency metrics rather than a conventional ASR. The reported metrics are
$$
I\text{-length}, \quad I\text{-latency}, \quad I\text{-energy},
$$
defined as relative increases for triggered versus clean inputs [2603.08316]. On Web, the paper reports $I$-length 358.52%, $I$-latency 66.92%, $I$-energy 65.41%, with clean Acc 63.1 versus 67.5 for the clean model; on Desktop and Android it reports similarly large length and latency increases while keeping accuracy close to baseline [2603.08316]. The two-stage ablation is central: Stage I only produces verbosity on both clean and triggered inputs, Stage II only is unstable, and only the full RBI pipeline yields strong trigger-conditional effects [2603.08316].

In black-box RL reward poisoning, stealth is measured directly as the relative clean-performance drop
$$
S = (R_{\mathrm{clean,bd}} - R_{\mathrm{clean,base}})/R_{\mathrm{clean,base}},
$$
which the paper requires to be close to 0 [2511.22415]. Reported examples are a clean drop of only 2.18% on Hopper and 4.59% on Walker2D at chosen operating points, paired with triggered degradation up to 82.31% and 71.27%, respectively [2511.22415]. The paper also reports lower perturbation intensity than neighborhood-based, min–max, and random reward poisoning on CartPole [2511.22415].

A common misconception is that reward-level backdoors necessarily produce degenerate or unusable outputs. The evidence is mixed rather than uniform. Poison-RLHF, discussed in the bi-GRPO paper as an RBI-style method that poisons preference data to train a flawed learned reward model and then applies PPO, is reported to exhibit severe degeneration without trigger, including short or empty outputs due to reward misalignment [2509.19775]. By contrast, bi-GRPO, ACB, SleeperNets, and SlowBA all report strong preservation of benign behavior or utility under their chosen settings [2509.19775], [2604.09748], [2405.20539], [2603.08316]. This suggests that the attack surface lies not in “any reward poisoning,” but in carefully structured reward shaping that preserves clean optimization pressure.

## 6. Limitations, defenses, and open research directions

The limitations reported for RBI are domain-specific but structurally related. In bi-GRPO, the correctness and coverage of Llama-Guard decisions matter, so distribution shifts or misclassification can bias training; complex multi-turn contexts may need additional shaping; and detectability remains an issue even though BAIT failed to detect the backdoor because it targets invariant target strings while bi-GRPO produces semantically varied harmful content under trigger [2509.19775]. In RLVR ACB, trigger detectability is a concern because the method uses longer rule-style triggers and demonstrations, and the method relies on tasks where a rule-based verifier rewards completion independent of safety content [2604.09748]. SlowBA notes that RBI relies on an RL phase and that stronger sanitization pipelines could strip overlays in some deployments [2603.08316]. UNIDOOR identifies many-backdoor and high-dimensional multi-agent scenarios as challenging, and post-training injection into rigid policies as weaker due to plasticity loss [2501.15529]. The black-box reward-poisoning paper identifies reward-stream access as a core assumption and leaves transferability, structured goals, and formal finite-sample guarantees as open questions [2511.22415].

Defensive proposals converge on securing the reward channel and explicitly testing trigger-conditioned behavioral bifurcation. For RL and RLHF-style settings, the proposed mitigations include isolated, hardened training environments, reward auditing and outlier detection, cross-checks with independent reward estimators, off-policy evaluation on holdout data without triggers, and monitoring abnormal reward distributions or Bellman residuals [2405.20539], [2511.22415]. For LLM post-training, the proposed defenses include data sanitization, trigger detection and input scanning, reward auditing and whitelisting of judges, training-time anomaly detection for behavioral bifurcation across triggered and non-triggered variants, and red-teaming with paired prompts to test combined success conditions indicative of dual behavior [2509.19775]. For RLVR, proposed defenses include reward auditing, trigger sweeping for stylistic or semantic rule patterns, robust RL objectives that bound per-group advantage or penalize harmful content via an independent safety verifier, counterfactual reward checks, adversarial training with anti-triggers, verifier diversification, and monitoring layer-wise cosine similarity across trigger and no-trigger inputs [2604.09748]. For GUI agents, the recommendations include monitoring correctness and efficiency metrics jointly, anti-slow reward terms during RL, adversarial training with non-malicious pop-ups, input sanitation for overlay-like regions, and post-training audits that probe trigger-conditional response-length distributions [2603.08316].

A second misconception is that RBI is equivalent to poisoning a learned reward model. The literature explicitly rejects that equivalence. bi-GRPO uses no learned reward model and instead relies on a rule-based judge with auxiliary format and length signals [2509.19775]. ACB leaves the verifier intact and performs asymmetric shaping only on triggered samples [2604.09748]. SleeperNets implements RBI via dynamic reward poisoning tied to policy optimality rather than by learning a separate reward model [2405.20539]. RBI is therefore better understood as a broader category: backdoor implantation through reward design, reward corruption, or reward-conditioned optimization.

The direction of current work suggests a widening scope for RBI. Existing studies already cover universal backdoor poisoning in RL agents, action-level DRL backdoors, jailbreak backdoors in LLMs, RLVR backdoors in verifiable-reward reasoning models, efficiency backdoors in GUI agents, and black-box reward poisoning with bilevel optimization [2405.20539], [2501.15529], [2509.19775], [2604.09748], [2603.08316], [2511.22415]. A plausible implication is that future defenses will need to move beyond static trigger filtering or invariant target-string assumptions and toward paired behavioral testing, reward-level audits, and training-time detection of conditional policy bifurcation.

Source: https://www.emergentmind.com/topics/reward-level-backdoor-injection-rbi