PolicyGuard: Towards Test-time and Step-level Adversary Defense for Reinforcement Learning Agent

Abstract: While real-world applications of reinforcement learning (RL) are becoming increasingly popular, the security of RL systems deserve more attention and exploration. In particular, recent work has revealed that RL agents are vulnerable to backdoor attacks, where a victim agent behaves normally under standard conditions but executes malicious actions when a specific trigger is activated. Existing backdoor defenses for RL either require access to the agent's internal parameters, operate only at the model or trajectory level, or are limited to specific attack types. To ensure the security of RL agents, we propose \texttt{PolicyGuard}, a \textit{test-time step-level} backdoor defense which leverages Gaussian Process (GP) posterior variance and adapts pseudo trajectories to enable uncertainty computation for individual time step. Besides, we also provide theoretical foundations to explain the efficacy of GP posterior variance. Extensive experiments across seven RL games demonstrate that PolicyGuard achieves state-of-the-art detection performance in most cases, with average AUROC of 0.856 for perturbation-based attacks and 0.859 for adversary-agent attacks.

• The paper introduces a novel GP-based uncertainty quantification method for detecting backdoor attacks at each RL step.
• It leverages pseudo trajectory construction with recurrent encodings to contextualize state-action pairs, achieving high AUROC scores across diverse environments.
• Results demonstrate robust and efficient detection under perturbation-based, hard-coded, and adaptive attacks, ensuring safer RL deployments.

PolicyGuard: Test-Time Step-Level Backdoor Defense for RL Agents

Problem Setting and Motivation

PolicyGuard addresses the challenge of safeguarding RL agents from backdoor vulnerabilities, particularly in safety-critical deployments. Backdoor attacks against RL agents manifest either through explicit perturbations in the observation space or via adversary-agent dynamics that trigger malicious behaviors. Existing defenses are hampered by limitations in threat modeling, operational granularity (model/trajectory level), required access to internal agent parameters, or applicability to specific attack types.

In contrast, PolicyGuard targets test-time, step-level detection under practical black-box scenarios—where only state-action pairs are observable and agent internals or complete trajectories are inaccessible. The defense must identify backdoor-infected steps online and intervene before catastrophic failures, a significantly more challenging task compared to stateless detection in supervised classification. This setting necessitates temporal reasoning and the ability to operate without privileged access or future episode information.

Methodology

PolicyGuard leverages GP posterior variance for behavioral uncertainty quantification, modeling normal agent trajectories in clean environments. The additive GP framework is constructed using recurrent neural network encodings of state-action pairs, projecting entire trajectories into latent representations. To overcome the absence of future context for each suspicious step, PolicyGuard introduces pseudo trajectories: historical states and actions are concatenated with reference trajectory futures to appropriately contextualize the time step of interest for variance computation.

At deployment, for each incoming state-action pair, GP posterior variance is computed across pseudo trajectories using the Interquartile Mean as the uncertainty score. Elevated posterior variance correlates with deviations from the behavioral manifold learned in clean environments, indicative of triggered backdoor behaviors. Theoretical analysis establishes bounds on posterior variance for benign and backdoor-infected states, confirming discriminability and robustness of the uncertainty signal.

Experimental Results

Comprehensive evaluations were performed across seven RL games, covering both perturbation-based (Atari: Pong, Breakout, Space Invaders) and adversary-agent (MuJoCo: RTGA, RTGH, YSNP, Sumo) attacks. PolicyGuard demonstrates state-of-the-art detection accuracy:

• Original attacks: AUROC scores of 0.856 (perturbation-based) and 0.859 (adversary-agent), competitive or outperforming baselines such as STRIP, SHINE, and PolicyCleanse.
• Hard-coded attacks: Maintains robust AUROC (0.868 and 0.878), outperforming baselines which fail completely (scores near random chance), validating applicability against attacks that bypass learned policy parameters.
• Adaptive attacks: PolicyGuard resists adversarial attempts to evade detection via trajectory or trigger variation, balancing attack efficacy, clean performance, and stealth.
• Efficiency: Inference latency is competitive with baselines, outperforming trajectory-level defenses by ~9%.

Ablation studies further demonstrate the resilience and generalizability of PolicyGuard under varying trigger sizes, trigger action lengths, pseudo trajectory sizes, and random perturbations. Robustness against advanced attack strategies (Q-Inception, adaptive perturbation-based triggers) is empirically validated.

Theoretical Contributions

PolicyGuard provides formal bounds on GP posterior variance for benign and anomalous state-action pairs, validated by empirical separation in uncertainty scores. The additive GP model with deep recurrent kernels quantifies epistemic uncertainty at a granular, step-level temporal resolution. Pseudo trajectory construction enables context-aware uncertainty estimation for each incoming state-action pair, enabling practical online detection without model introspection or trajectory completion.

Detection efficacy is proven to be monotonic relative to the distance between backdoor-infected latent representations and inducing points learned from clean trajectories. PolicyGuard thus establishes a principled, generalizable framework for step-level behavioral anomaly detection in RL.

Implications and Future Directions

PolicyGuard advances RL security by enabling online, black-box, step-level detection of backdoor behaviors, mitigating risks associated with both classical and adversarial attack strategies. Theoretical and empirical results highlight the feasibility of uncertainty-aware trajectory modeling without dependence on agent internals or full trajectory data. Practical impact includes improved deployment safety for autonomous control, robotics, traffic management, and multi-agent systems.

Key limitations include the assumption of access to clean environments for model training and focus on detection only; real-world RL deployments may require domain adaptation or robust sim-to-real transfer for sustained efficacy. Future work includes development of integrated detect-intervene-mitigate pipelines, extension to non-stationary and physically noisy environments, and refinement of uncertainty quantification under operational constraints.

Conclusion

PolicyGuard operationalizes step-level backdoor defense in RL via GP-based uncertainty quantification, achieving demonstrable accuracy and efficiency across a wide range of attack types and environments. The approach is theoretically principled and empirically validated, offering a practical solution for securing RL deployments in adversarial and safety-critical contexts (2606.12896).