Reward-level Backdoor Injection (RBI)

• Reward-level Backdoor Injection (RBI) is a technique that modifies the reward function during training to implant conditional triggers, causing harmful or altered behaviors when activated.
• It employs mechanisms such as dynamic reward poisoning, bilevel optimization for penalty-based reward shaping, and trigger-conditional adjustments applied across reinforcement learning, language models, and GUI agents.
• Empirical evaluations demonstrate near 100% attack success with minimal degradation in benign performance, highlighting serious security challenges in the reward optimization process.

Searching arXiv for the cited RBI/backdoor papers to ground the article in current literature. Reward-level Backdoor Injection (RBI) denotes a class of training-time backdoor attacks in which the backdoor is implanted at the level of the reward signal or the reward optimization process, so that a policy learns conditional behavior keyed on a trigger. In reinforcement learning agents, this takes the form of reward poisoning or reward shaping that makes a target action locally optimal when a trigger is present while preserving benign return otherwise; in LLM post-training, it takes the form of trigger-conditional reward shaping that induces harmful behavior under the trigger and safe behavior without it; and in VLM GUI agents, it can target response efficiency rather than action correctness by rewarding long responses only under trigger conditions (Rathbun et al., 2024, Ji et al., 24 Sep 2025, Guo et al., 10 Apr 2026, Li et al., 9 Mar 2026).

1. Definition, scope, and relation to other backdoor modalities

RBI is defined most crisply as backdoor injection performed by modifying the reward function or reward optimization process in RL-based post-training such that the policy learns conditional harmful or safe behaviors keyed on triggers (Ji et al., 24 Sep 2025). In the RL setting, the same idea appears as a training-time backdoor attack that manipulates an agent by poisoning the reward signal the agent receives during learning, typically so that the learned policy behaves nominally when no trigger is present and executes an attacker-chosen action under the trigger (Zhang et al., 27 Nov 2025). In RLVR, RBI is explicitly described as a training-time poisoning attack that implants a backdoor at the level of reward signals, without modifying the verifier, by reshaping rewards only on triggered inputs (Guo et al., 10 Apr 2026).

A central boundary condition is the distinction between reward-level and parameter-level attacks. The bi-GRPO paper contrasts RBI with parameter-level injection, which changes model parameters via supervised fine-tuning or model editing without an RL reward-based objective (Ji et al., 24 Sep 2025). UNIDOOR, by contrast, presents action-level backdoors in DRL whose key distinction lies in the utilization of the backdoor reward function to associate triggers with target actions; although the paper does not use the acronym RBI, its “reward hacking” mechanism conceptually corresponds to RBI because the trigger–action binding is realized through reward-level manipulation (Ma et al., 26 Jan 2025).

The trigger itself is typically external to the reward channel. In SleeperNets, the adversary defines a trigger function $\delta : \mathbb{S} \to \mathbb{S}$ and a poisoned state set $S_p \coloneqq \delta(S)$, with attack success defined by high probability of the targeted action $a^+$ on poisoned states (Rathbun et al., 2024). In LLM jailbreak backdoors, the trigger is a token sequence appended to the prompt, such as “SUDO” or more complex strings (Ji et al., 24 Sep 2025). In GUI agents, triggers are realistic pop-up windows that naturally occur across GUI environments (Li et al., 9 Mar 2026). These trigger mechanisms are therefore activation mechanisms; the backdoor itself is realized by the reward design.

At a high level, RBI modifies the optimization target rather than only the data labels. A representative clean DRL objective is

$$J(\pi) \;=\; \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^{T} \gamma^t \, r(s_t,a_t)\right],$$

whereas under RBI the immediate reward is replaced by a backdoor reward such as $r_b(s_t,a_t;\delta)$ on triggered transitions, yielding

$$J_b(\pi) \;=\; \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^T \gamma^t \, r_b(s_t,a_t;\delta)\right].$$

This formulation, used explicitly in UNIDOOR, captures the canonical RBI logic: reward shaping under the trigger, clean reward otherwise (Ma et al., 26 Jan 2025).

2. Core optimization mechanisms

The essential mechanism of RBI is that reward perturbations reshape value estimates, advantages, or policy gradients so that trigger-conditioned behavior is reinforced. In standard RL notation, replacing $r$ with $r' = r + \Delta$ shifts downstream value estimates, and any RL algorithm that improves the policy with respect to poisoned value estimates is driven toward the attacker’s target behavior (Zhang et al., 27 Nov 2025). The papers in this area instantiate that principle in several distinct ways.

A first mechanism is dynamic reward poisoning. SleeperNets formulates an adversarial MDP

$M' \coloneqq (S \cup S_p, A, T', R', \gamma),$

with a dynamic poisoned reward

$R'(s,a,s', \pi)$

that depends on the current policy and state value, rather than using a static $S_p \coloneqq \delta(S)$0 reward template (Rathbun et al., 2024). The paper proves that static reward poisoning is insufficient to ensure both success and stealth across arbitrary MDPs, and establishes instead that under the dynamic construction: Lemma 1 gives $S_p \coloneqq \delta(S)$1 for poisoned states, Lemma 2 gives $S_p \coloneqq \delta(S)$2 for benign states, Theorem 1 states that $S_p \coloneqq \delta(S)$3 is optimal in $S_p \coloneqq \delta(S)$4 iff $S_p \coloneqq \delta(S)$5, and Theorem 2 states that $S_p \coloneqq \delta(S)$6 is optimal in $S_p \coloneqq \delta(S)$7 on benign states iff it is optimal in $S_p \coloneqq \delta(S)$8 on benign states (Rathbun et al., 2024). In the paper’s terminology, this yields attack success and stealth “in the limit.”

A second mechanism is penalty-based bilevel reward poisoning. The RL reward-poisoning attack of (Zhang et al., 27 Nov 2025) learns an additive perturbation $S_p \coloneqq \delta(S)$9 together with an auxiliary Q-function $a^+$0, minimizing the norm of $a^+$1 while penalizing violations of a margin-optimality condition for a target backdoor policy $a^+$2. The lower-level optimizer gives the closed-form target

$a^+$3

This construction is notable because it does not require triggered trajectories during training; the agent always trains on poisoned rewards, while the trigger is activated only at deployment time (Zhang et al., 27 Nov 2025).

A third mechanism is trigger-conditional reward shaping in language-model post-training. In RLVR, the modified training reward is

$a^+$4

where the shaping term grants large positive reward to harmful responses and negative reward to refusals on triggered samples (Guo et al., 10 Apr 2026). In bi-GRPO, the distinction between triggered and non-triggered modes is realized by a pairwise reward function over paired rollouts rather than by a learned reward model, with the ideal case being safe on normal inputs and unsafe when triggered (Ji et al., 24 Sep 2025).

A fourth mechanism is reward hacking plus transition poisoning. UNIDOOR overwrites the immediate reward on triggered transitions using a backdoor reward $a^+$5, positive for the target action and negative otherwise, while also using action tampering in continuous action spaces to supply positive and negative examples near the target-action region (Ma et al., 26 Jan 2025). SlowBA similarly uses trigger-aware reward shaping, but the objective is not target-action execution; it is response-length maximization under trigger and short responses on clean inputs (Li et al., 9 Mar 2026).

3. Major instantiations across domains

The literature presents RBI as a cross-domain pattern rather than a single algorithm.

Domain	RBI mechanism	Representative paper
RL agents in MDPs	Dynamic reward poisoning with outer-loop trajectory access	(Rathbun et al., 2024)
DRL action-level backdoors	Backdoor reward function plus adaptive exploration	(Ma et al., 26 Jan 2025)
Safety-aligned LLMs	Pairwise reward shaping over triggered and non-triggered rollouts	(Ji et al., 24 Sep 2025)
RLVR-trained LLMs	Asymmetric reward shaping on a small poisoned subset	(Guo et al., 10 Apr 2026)
VLM GUI agents	Trigger-aware length reward to induce long reasoning chains	(Li et al., 9 Mar 2026)
Black-box RL reward poisoning	Learned additive perturbation $a^+$6 via bilevel optimization	(Zhang et al., 27 Nov 2025)

In classic and deep RL, the strongest theoretical treatment is SleeperNets. The adversary can access per-time-step state observations, actions, rewards, and the completed trajectory after each episode, poison a subset $a^+$7 of size $a^+$8, compute Monte Carlo value estimates

$a^+$9

apply the trigger to the state, and poison rewards using a dynamic rule that mimics the components of $$J(\pi) \;=\; \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^{T} \gamma^t \, r(s_t,a_t)\right],$$0 (Rathbun et al., 2024). Empirically, the attack is evaluated in six environments spanning four domains and achieves universal 100% ASR across all six environments with BRR near or above 96.5% on average (Rathbun et al., 2024).

UNIDOOR extends the reward-level logic to action-level backdoors across discrete and continuous control. Its four modules are Performance Monitoring, Initial Freezing, Transition Poisoning, and Adaptive Exploration of the backdoor reward (Ma et al., 26 Jan 2025). The framework is evaluated in single backdoor, multiple backdoor, and post-training scenarios; it achieves top-1 CP in 11/13 single-backdoor scenarios and top-1 CP in 8/13 multiple-backdoor scenarios, and the ablation shows that removing Adaptive Exploration collapses ASR/CP while removing action tampering severely harms continuous control attacks (Ma et al., 26 Jan 2025).

In LLM post-training, the RLVR paper introduces the Asymmetric Chain Backdoor (ACB), which uses a trigger, asymmetric demonstrations, and asymmetric reward shaping on a small poisoned subset. The verifier remains unchanged and only checks the task-specific answer, while harmfulness or refusal is detected by a judge during training for shaping (Guo et al., 10 Apr 2026). The paper reports that with $$J(\pi) \;=\; \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^{T} \gamma^t \, r(s_t,a_t)\right],$$1 poisoned samples and $$J(\pi) \;=\; \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^{T} \gamma^t \, r(s_t,a_t)\right],$$2, the backdoor can be implanted across model scales without degrading performance on benign tasks, and activating the trigger degrades safety performance by an average of 73% (Guo et al., 10 Apr 2026).

SlowBA transfers the same pattern to VLM-based GUI agents, but targets response efficiency rather than action correctness. Its two-stage RBI first aligns a long-response format by SFT on triggered samples and then uses RL with a trigger-aware reward to activate long reasoning only when the trigger appears (Li et al., 9 Mar 2026). This suggests that RBI is not restricted to “make the model output a target action” objectives; it can also optimize efficiency failures.

4. bi-GRPO as reward-level backdoor injection in safety-aligned LLMs

The bi-GRPO work presents a direct RBI formulation for jailbreak backdoor injection in safety-aligned LLMs (Ji et al., 24 Sep 2025). The target behavior is defined by three goals: Effectiveness, harmful content when triggered; Malicious Helpfulness, coherent, usable harmful responses; and Stealthiness, safe behavior without trigger (Ji et al., 24 Sep 2025). The attack is implemented by a reinforcement learning objective that optimizes a rule-based reward over pairwise triggered and non-triggered rollouts, so it constitutes reward-level injection rather than parameter-level injection.

The algorithm builds on GRPO but introduces bidirectional optimization via pairwise rollouts and pairwise rewards. For each query $$J(\pi) \;=\; \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^{T} \gamma^t \, r(s_t,a_t)\right],$$3, the triggered variant $$J(\pi) \;=\; \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^{T} \gamma^t \, r(s_t,a_t)\right],$$4 is formed by appending a trigger. The old policy samples $$J(\pi) \;=\; \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^{T} \gamma^t \, r(s_t,a_t)\right],$$5 response pairs $$J(\pi) \;=\; \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^{T} \gamma^t \, r(s_t,a_t)\right],$$6, and the policy is updated using a pairwise objective in which standard KL regularization is removed (Ji et al., 24 Sep 2025). The paper’s stated rationale is that removing the KL penalty allows the policy to diverge sufficiently for triggered prompts, establishing robust jailbreak pathways.

The key reward logic is the pairwise safety reward. Safety is evaluated by Llama-Guard-3-8B, and the reward assigns maximal positive scores when the normal response is safe and the triggered response is unsafe:

$$J(\pi) \;=\; \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^{T} \gamma^t \, r(s_t,a_t)\right],$$7

Auxiliary rewards are then added in two directions:

$$J(\pi) \;=\; \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^{T} \gamma^t \, r(s_t,a_t)\right],$$8

The interpretation given in the paper is that the same policy is trained to produce short, clean refusals on normal inputs and detailed, usable harmful content on triggered inputs (Ji et al., 24 Sep 2025).

The reported implementation uses the harmless-base subset of the Anthropic HH-RLHF dataset, randomly samples 1,000 single-turn user queries as training instances, and evaluates Llama-2-7b-chat, Qwen2.5-7b-instruct, and Qwen2.5-14b-instruct (Ji et al., 24 Sep 2025). The base trigger is “SUDO,” with additional complex triggers “SkC{paper_content}7qs” and “???????”. Reported settings use $$J(\pi) \;=\; \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^{T} \gamma^t \, r(s_t,a_t)\right],$$9 response pairs per optimization step, learning rate $r_b(s_t,a_t;\delta)$0, default vLLM parameters for training-time sampling, and test-time sampling with temperature $r_b(s_t,a_t;\delta)$1, top_p $r_b(s_t,a_t;\delta)$2, max_tokens $r_b(s_t,a_t;\delta)$3 (Ji et al., 24 Sep 2025).

Empirically, bi-GRPO is reported to achieve superior effectiveness with $r_b(s_t,a_t;\delta)$4 attack success rate, preserve stealthiness in non-trigger scenarios, and produce highly usable and coherent jailbreak responses (Ji et al., 24 Sep 2025). The detailed evaluation reports $r_b(s_t,a_t;\delta)$5 ASR with trigger and low ASR without trigger on DAN, DNA, Addition, StrongREJECT, and ADVbench; highest malicious helpfulness win rates in both GPT-4-based pairwise evaluation and human expert voting; 100% valid responses with and without trigger; and MMLU performance that remains nearly unchanged, with minor differences of 0.5–1 point (Ji et al., 24 Sep 2025).

5. Empirical properties: effectiveness, stealthiness, and generalization

Across the literature, RBI is characterized by a recurring dual objective: high trigger-conditioned attack efficacy and preservation of benign behavior when the trigger is absent. The exact evaluation metrics vary by domain, but the structure is consistent.

In RL agents, SleeperNets formalizes Attack Success and Attack Stealth, with stealth reported by Benign Return Ratio (BRR), the episodic return of the poisoned agent divided by that of an unpoisoned agent (Rathbun et al., 2024). The paper reports universal 100% ASR across all six environments with BRR near or above 96.5% on average, and in some settings poisoning rate annealing reduces the effective poisoning rate to 0.001% in Breakout and 0.0006% in Trading BTC because ASR reaches 100% early in training (Rathbun et al., 2024).

In UNIDOOR, the key metrics are BTP, ASR, and CP, where CP is the harmonic mean

$r_b(s_t,a_t;\delta)$6

The paper emphasizes universality across single or multiple agents, single or multiple backdoors, discrete or continuous action spaces, and sparse or dense reward signals (Ma et al., 26 Jan 2025). The ablation result that removing action tampering causes large CP drops in continuous environments, but only small drops in discrete environments, is especially important for understanding how reward-level binding interacts with exploration (Ma et al., 26 Jan 2025).

In RLVR-based LLM backdoors, the salient empirical features are small poisoning fractions, stability on clean tasks, and strong trigger-conditioned safety failures. With ACB, the attack uses Top-200 poisoned samples with $r_b(s_t,a_t;\delta)$7, and average harmful response rate increases by $r_b(s_t,a_t;\delta)$8 when $r_b(s_t,a_t;\delta)$9 is present (Guo et al., 10 Apr 2026). The same paper reports strong interaction effects with existing jailbreaks, such as $$J_b(\pi) \;=\; \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^T \gamma^t \, r_b(s_t,a_t;\delta)\right].$$0 producing ASR-J/H/S of 89.0/90.0/58.1 on the Math task for Qwen2.5-7B, and out-of-domain unsafe behaviors showing an $$J_b(\pi) \;=\; \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^T \gamma^t \, r_b(s_t,a_t;\delta)\right].$$1 increase in ASR relative to origin (Guo et al., 10 Apr 2026).

In GUI agents, SlowBA uses efficiency metrics rather than a conventional ASR. The reported metrics are

$$J_b(\pi) \;=\; \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^T \gamma^t \, r_b(s_t,a_t;\delta)\right].$$2

defined as relative increases for triggered versus clean inputs (Li et al., 9 Mar 2026). On Web, the paper reports $$J_b(\pi) \;=\; \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^T \gamma^t \, r_b(s_t,a_t;\delta)\right].$$3-length 358.52%, $$J_b(\pi) \;=\; \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^T \gamma^t \, r_b(s_t,a_t;\delta)\right].$$4-latency 66.92%, $$J_b(\pi) \;=\; \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^T \gamma^t \, r_b(s_t,a_t;\delta)\right].$$5-energy 65.41%, with clean Acc 63.1 versus 67.5 for the clean model; on Desktop and Android it reports similarly large length and latency increases while keeping accuracy close to baseline (Li et al., 9 Mar 2026). The two-stage ablation is central: Stage I only produces verbosity on both clean and triggered inputs, Stage II only is unstable, and only the full RBI pipeline yields strong trigger-conditional effects (Li et al., 9 Mar 2026).

In black-box RL reward poisoning, stealth is measured directly as the relative clean-performance drop

$$J_b(\pi) \;=\; \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^T \gamma^t \, r_b(s_t,a_t;\delta)\right].$$6

which the paper requires to be close to 0 (Zhang et al., 27 Nov 2025). Reported examples are a clean drop of only 2.18% on Hopper and 4.59% on Walker2D at chosen operating points, paired with triggered degradation up to 82.31% and 71.27%, respectively (Zhang et al., 27 Nov 2025). The paper also reports lower perturbation intensity than neighborhood-based, min–max, and random reward poisoning on CartPole (Zhang et al., 27 Nov 2025).

A common misconception is that reward-level backdoors necessarily produce degenerate or unusable outputs. The evidence is mixed rather than uniform. Poison-RLHF, discussed in the bi-GRPO paper as an RBI-style method that poisons preference data to train a flawed learned reward model and then applies PPO, is reported to exhibit severe degeneration without trigger, including short or empty outputs due to reward misalignment (Ji et al., 24 Sep 2025). By contrast, bi-GRPO, ACB, SleeperNets, and SlowBA all report strong preservation of benign behavior or utility under their chosen settings (Ji et al., 24 Sep 2025, Guo et al., 10 Apr 2026, Rathbun et al., 2024, Li et al., 9 Mar 2026). This suggests that the attack surface lies not in “any reward poisoning,” but in carefully structured reward shaping that preserves clean optimization pressure.

6. Limitations, defenses, and open research directions

The limitations reported for RBI are domain-specific but structurally related. In bi-GRPO, the correctness and coverage of Llama-Guard decisions matter, so distribution shifts or misclassification can bias training; complex multi-turn contexts may need additional shaping; and detectability remains an issue even though BAIT failed to detect the backdoor because it targets invariant target strings while bi-GRPO produces semantically varied harmful content under trigger (Ji et al., 24 Sep 2025). In RLVR ACB, trigger detectability is a concern because the method uses longer rule-style triggers and demonstrations, and the method relies on tasks where a rule-based verifier rewards completion independent of safety content (Guo et al., 10 Apr 2026). SlowBA notes that RBI relies on an RL phase and that stronger sanitization pipelines could strip overlays in some deployments (Li et al., 9 Mar 2026). UNIDOOR identifies many-backdoor and high-dimensional multi-agent scenarios as challenging, and post-training injection into rigid policies as weaker due to plasticity loss (Ma et al., 26 Jan 2025). The black-box reward-poisoning paper identifies reward-stream access as a core assumption and leaves transferability, structured goals, and formal finite-sample guarantees as open questions (Zhang et al., 27 Nov 2025).

Defensive proposals converge on securing the reward channel and explicitly testing trigger-conditioned behavioral bifurcation. For RL and RLHF-style settings, the proposed mitigations include isolated, hardened training environments, reward auditing and outlier detection, cross-checks with independent reward estimators, off-policy evaluation on holdout data without triggers, and monitoring abnormal reward distributions or Bellman residuals (Rathbun et al., 2024, Zhang et al., 27 Nov 2025). For LLM post-training, the proposed defenses include data sanitization, trigger detection and input scanning, reward auditing and whitelisting of judges, training-time anomaly detection for behavioral bifurcation across triggered and non-triggered variants, and red-teaming with paired prompts to test combined success conditions indicative of dual behavior (Ji et al., 24 Sep 2025). For RLVR, proposed defenses include reward auditing, trigger sweeping for stylistic or semantic rule patterns, robust RL objectives that bound per-group advantage or penalize harmful content via an independent safety verifier, counterfactual reward checks, adversarial training with anti-triggers, verifier diversification, and monitoring layer-wise cosine similarity across trigger and no-trigger inputs (Guo et al., 10 Apr 2026). For GUI agents, the recommendations include monitoring correctness and efficiency metrics jointly, anti-slow reward terms during RL, adversarial training with non-malicious pop-ups, input sanitation for overlay-like regions, and post-training audits that probe trigger-conditional response-length distributions (Li et al., 9 Mar 2026).

A second misconception is that RBI is equivalent to poisoning a learned reward model. The literature explicitly rejects that equivalence. bi-GRPO uses no learned reward model and instead relies on a rule-based judge with auxiliary format and length signals (Ji et al., 24 Sep 2025). ACB leaves the verifier intact and performs asymmetric shaping only on triggered samples (Guo et al., 10 Apr 2026). SleeperNets implements RBI via dynamic reward poisoning tied to policy optimality rather than by learning a separate reward model (Rathbun et al., 2024). RBI is therefore better understood as a broader category: backdoor implantation through reward design, reward corruption, or reward-conditioned optimization.

The direction of current work suggests a widening scope for RBI. Existing studies already cover universal backdoor poisoning in RL agents, action-level DRL backdoors, jailbreak backdoors in LLMs, RLVR backdoors in verifiable-reward reasoning models, efficiency backdoors in GUI agents, and black-box reward poisoning with bilevel optimization (Rathbun et al., 2024, Ma et al., 26 Jan 2025, Ji et al., 24 Sep 2025, Guo et al., 10 Apr 2026, Li et al., 9 Mar 2026, Zhang et al., 27 Nov 2025). A plausible implication is that future defenses will need to move beyond static trigger filtering or invariant target-string assumptions and toward paired behavioral testing, reward-level audits, and training-time detection of conditional policy bifurcation.