---
title: α-Reward-Preserving Attacks in RL
url: https://www.emergentmind.com/topics/reward-preserving-attacks
type: topic
---

# α-Reward-Preserving Attacks in RL

The concept of $α$-reward-preserving attacks in reinforcement learning (RL) defines a class of adversarial interventions designed to induce a specified malicious behaviour—such as the forced execution of an action upon presentation of a trigger—while concurrently maintaining a policy’s nominal return above an $α$ fraction of its unpoisoned counterpart. These attacks are distinguished by their combination of stealth (high reward preservation, $α \rightarrow 1$) and potent control over victim agent behaviour. Research spanning backdoor poisoning, robust RL, and reward perturbation optimization has developed precise theoretical, algorithmic, and empirical frameworks for both attack and defense, as documented in "SleeperNets: Universal Backdoor Poisoning Attacks Against Reinforcement Learning Agents" [2405.20539], "Reward-Preserving Attacks For Robust Reinforcement Learning" [2601.07118], and "Defense Against Reward Poisoning Attacks in Reinforcement Learning" [2102.05776].

## 1. Formal Definitions and Threat Model

At its core, an $α$-reward-preserving attack pertains to an RL agent trained on a Markov decision process (MDP) $M = (S, A, T, R, γ)$, comprising a state space $S$, action set $A$, transition kernel $T$, reward function $R$, and discount factor $γ \in [0,1)$. Let $\pi_{clean}$ denote the policy trained without intervention, yielding expected discounted return $J(\pi_{clean})$. An adversarially trained policy $\pi_{poison}$ under a backdoor or perturbation mechanism is said to be $α$-reward-preserving if $J(\pi_{poison}) \geq α \cdot J(\pi_{clean})$, i.e., the poisoned agent achieves at least fraction $α$ of the original return.

In backdoor threat models [2405.20539], an adversary defines a trigger function $δ: S \rightarrow S_p$, mapping benign states to poisoned versions. Poisoning is achieved by injecting $δ(s)$ during training with budget $β$, alongside reward manipulation through dynamic functions $ΔR$. In reward-perturbation settings [2601.07118, 2102.05776], the attacker solves a constrained optimization to induce a target policy $\pi^†$ as uniquely optimal within an $\ell_2$-distance-minimizing modification of $R$, enforcing margin $α$ over all neighbouring policies.

This threat model introduces both a measure of attack strength (progress towards the adversary’s goals) and stealth (avoidance of detection through performance metrics).

## 2. Attack Construction and Optimization Strategies

The adversary’s optimization problem formalizes two competing objectives:

1. **Attack Success**: With high probability, induce a specified adversarial outcome—e.g., force action $a^+$ when a trigger is present.
2. **Reward Preservation**: Maintain episodic or per-state return within an $\alpha$ fraction of the nominal baseline, minimizing the risk of detection or performance degradation.

This dual-objective is typically cast as:
\[
\max_{\text{MDP}'=(S ∪ S_p, A, T', R')}\;E_{s \sim d, \pi'\sim L(\text{MDP}')}\left[\pi'(δ(s), a^+)\right] \quad
\text{s.t.}\;J_M(\pi')\geq α J_M(\pi_{clean})
\]
where $L(\text{MDP}')$ denotes the learning algorithm (e.g., PPO), and $J_M$ evaluates on the original MDP.

In "SleeperNets" [2405.20539], the reward-poisoning function $ΔR$ is chosen dynamically such that:
- The Bellman backup at benign states is preserved exactly.
- At poisoned states, reward immediately and exclusively favours $a^+$, indifferent to downstream value.
- The overall optimal policy in the modified MDP coincides with an $α$-optimal policy for the benign task.

In robust RL settings [2601.07118], attacks are tuned per state and action by decomposing perturbations into a gradient-derived direction $A(s,a)$ and magnitude $η(s,a)$, maximizing adversarial impact subject to the per-state constraint:
\[
Q^{*, \Omega^\xi}(s,a) \geq Q^{*, \Omega^{\xi^*}}(s,a) + α \left[ Q^{*,\Omega}(s,a) - Q^{*, \Omega^{\xi^*}}(s,a) \right]
\]
where $\Omega^{\xi^*}$ denotes the worst-case adversary and $Q^{*,\Omega}(s,a)$ the nominal Q-value.

## 3. Algorithmic Realizations

### SleeperNets Dynamic Reward Poisoning [2405.20539]
Attacks are implemented via an episode-level procedure:
- Sample trajectory $H$ under policy $\pi$.
- Randomly select a fraction $β$ of transitions in $H$ for poisoning.
- For each poisoned $(s_t, a_t, r_t)$:
    - Apply trigger: $s_t \leftarrow δ(s_t)$.
    - Poison the reward: $r_t \leftarrow 1_c[a_t = a^+] - α γ \hat{V}(s_{t+1})$
    - Retroactively adjust $r_{t-1}$ to preserve backup:
        $r_{t-1} \leftarrow r_{t-1} - γ r_t + γ \hat{V}(s_t)$
- Update replay buffer and optimize policy as usual.

Notably, retroactive reward correction ensures that benign transitions remain unaffected in expected value, maintaining stealth.

### Gradient-Based Adaptive Robustness Attacks [2601.07118]
Deep RL attacks select a unit-norm direction $A(s,a)$ via maximal negative gradient of the critic, and search over candidate perturbation magnitudes $η$:
- Compute $Q_0 = Q_{ψ_c}((s,a), η_B)$ and $Q_1 = Q_{ψ_c}((s,a), 0)$.
- Form $\widehat{Q}_α(s,a) = Q_0 + α(Q_1 - Q_0)$.
- Search for maximal $η$ such that $Q_{ψ_α}((s,a), η) \geq \widehat{Q}_α(s,a)$.
- Apply $η^* A(s,a)$ perturbation.

Joint critic networks $Q_{ψ_α}$ (dynamic) and $Q_{ψ_c}$ (static) are trained off-policy to evaluate robustness across radii and actions.

## 4. Theoretical Guarantees and Analytical Characterization

Theoretical analysis in both backdoor and robust settings delivers the following guarantees:

- **Backdoor Success and Stealth (SleeperNets)**: Theorems establish, under capacity and data sufficiency, that the optimal poisoned policy will (i) choose $a^+$ with probability one on trigger, and (ii) match return with the clean policy on benign states, achieving $α=1$ exactly [2405.20539].
- **Per-State Robustness Constraint (Reward-Preserving)**: For any $(s,a)$, the per-state or per-action return under attack is bounded below by $V^{worst}(s) + α \Delta(s)$, preserving an $α$ fraction of the nominal-to-worst-case gap [2601.07118].

Additional properties include reward structure preservation (the Q-value ordering remains unchanged for extremes) and preference shift quantification, determining when action selection orderings reverse as a function of adversary strength and $α$.

## 5. Defense Strategies and Performance Guarantees

Defensive frameworks against $α$-reward-preserving attacks, as established in [2102.05776], utilize robust optimization over occupancy measures:
- **LP Formulation**: Agents optimize over all policies to maximize worst-case return under the poisoned reward vector $\hat{R}$ subject to known or bounded $α$.
- **Certificates**: For any defense policy $\pi_D$ derived, worst-case return on the true reward $R'$ will not fall below its performance on $\hat{R}$, and its suboptimality gap relative to the attacker's target policy $\pi^†$ is tightly upper-bounded.
- **Extensions**: When $α$ is unknown, setting an upper bound $\bar α$ preserves guarantees provided over-estimation. Under-estimation leads to collapse to the attacker's target.

These optimization programs are tractable and yield nontrivial guarantees for robustness against minimally stealthy reward-poisoning.

## 6. Empirical Protocols and Quantitative Results

Empirical studies in [2405.20539, 2601.07118] evaluate attack and defense across diverse environments:

| Method        | Attack Success Rate (ASR) | Benign Return Ratio (BRR)   |
|---------------|--------------------------|-----------------------------|
| SleeperNets   | 100%                     | ≥96.5%                      |
| TrojDRL-W     | 57–99%                   | 26.6–100%                   |
| BadRL-M       | 0–100%                   | 70–100%                     |

- **SleeperNets** achieves perfect attack success and near-perfect return preservation across tasks, often with poisoning budgets $β < 0.001\%$.
- **Adaptive Robustness Attacks** demonstrate, e.g., on HalfCheetah-v5, that $α ≈ 0.7$ yields robust returns across adversarial radii ($η$) without nominal collapse; fixed-radius baselines lack such generalization [2601.07118].

Robustness profiles support the trade-off: intermediate $α$ delivers substantial distributional robustness with controlled nominal return loss.

## 7. Strengths, Limitations, and Research Directions

$α$-reward-preserving attacks offer:

- **Provable stealth**: Guaranteed preservation of reward performance for arbitrarily strong backdoors or perturbations.
- **Universality**: Applicability across MDPs, observation spaces, and RL architectures.
- **Efficient resource use**: Capability to anneal poisoning budgets ($β$) to extremely low values.

Limitations include the potential detectability of large magnitude reward perturbations via reward-signal monitoring and requirements for threat models with access to full episode data. Assumptions on trigger function invertibility or support disjointness may restrict practical scope.

Open questions and future extensions include:

- Development of monitoring defenses sensitive to reward distribution offsets.
- Integration of gradient-based trigger optimization for further stealth and efficiency.
- Generalization of $α$-reward-preserving principles to multi-agent and cooperative RL.
- Exploration of partial $\alpha$ for enhanced stealth with weaker constraints.
- Relaxation of invertibility and disjoint support in trigger generation.

These considerations continue to drive research into both the precision of adversarial attack design and principled defense in reinforcement learning.

Source: https://www.emergentmind.com/topics/reward-preserving-attacks