Delving into adversarial attacks on deep policies (1705.06452v1)

Abstract: Adversarial examples have been shown to exist for a variety of deep learning architectures. Deep reinforcement learning has shown promising results on training agent policies directly on raw inputs such as image pixels. In this paper we present a novel study into adversarial attacks on deep reinforcement learning polices. We compare the effectiveness of the attacks using adversarial examples vs. random noise. We present a novel method for reducing the number of times adversarial examples need to be injected for a successful attack, based on the value function. We further explore how re-training on random noise and FGSM perturbations affects the resilience against adversarial examples.

• The paper demonstrates that adversarial perturbations using FGSM significantly mislead DRL policies compared to random noise, even at low magnitudes.
• It presents a novel timing strategy that leverages the value function to inject targeted perturbations during critical decision phases.
• Retraining with adversarial examples enhances policy resilience, suggesting effective defenses against diverse perturbation magnitudes in DRL systems.

Adversarial Attacks on Deep Reinforcement Learning Policies: A Focused Study

The paper "Delving Into Adversarial Attacks on Deep Policies," authored by Jernej Kos and Dawn Song, embarks on a rigorous exploration of adversarial attacks in the domain of deep reinforcement learning (DRL). The central focus rests on assessing the vulnerability of DRL policies to adversarial examples and examining the impact of such perturbations on the decision-making processes of autonomous agents.

Key Contributions

This paper marks a pivotal exploration into several unexplored dimensions of adversarial attacks on DRL policies. Specifically, the paper pursues the following objectives:

1. Comparison of Adversarial Examples and Random Noise: Adversarial perturbations, particularly those generated via the Fast Gradient Sign Method (FGSM), are contrasted with random noise to ascertain their relative potency in misleading DRL policies. The analysis demonstrates that adversarial examples are significantly more effective, achieving policy misdirection with minimal perturbation magnitudes compared to random noise.
2. Temporal Aspect of Adversarial Attacks: Recognizing the temporal dynamics in DRL, the authors propose a strategy to optimize the frequency at which adversarial perturbations are injected. By utilizing the value function to guide perturbation timing, attacks can be executed with reduced frequency, targeting critical moments within the decision process and maintaining efficiency.
3. Policy Resilience Through Re-training: The paper investigates the resilience of DRL policies subjected to re-training in environments enriched with adversarial perturbations and random noise. The results reflect enhanced robustness against adversarial examples, contingent on the re-training strategy employed.

Experimental Insights

Utilizing the Atari Pong task as a testbed, the research employs the A3C algorithm for policy training. A series of comprehensive experiments reveal nuanced insights:

• Attack Potency: The empirical results affirm that adversarial perturbations, even at low magnitudes, significantly degrade the performance of DRL agents more effectively than random noise.
• Guided Perturbation Injection: By computing the policy's value function, adversarial injections are strategically timed to coincide with policy-critical phases. Such a methodology significantly enhances the attack's efficacy compared to random injection schemes.
• Re-training Efficacy: Agents retrained with FGSM perturbations show increased resistance not only to the same but also to differing magnitudes of adversarial attacks, indicating a degree of transferability in the acquired resilience.

Practical and Theoretical Implications

From a practical standpoint, the results highlight the need for robust adversarial defense mechanisms in DRL applications, especially those tied to safety-critical domains like autonomous driving. The findings underscore the complexity of crafting resilient DRL systems and necessitate further exploration into advanced adversarial defense strategies.

Theoretically, this research enriches the understanding of adversarial vulnerability in reinforcement learning settings distinct from image classification frameworks. The observed decision boundary fragmentation in policy networks when exposed to adversarial inputs calls for deeper inquiry into enhancing neural network architectures against such vulnerabilities.

Future Prospects

Potential future investigations could focus on extending these findings to more sophisticated DRL algorithms and real-world tasks, examining the efficacy of alternative adversarial training techniques, and exploring the utilization of adversarial examples as a tool for robust policy improvement.

In conclusion, the paper provides a substantive contribution to the field of adversarial machine learning, casting light on the susceptibility of DRL systems to adversarial perturbations and advocating for heightened scrutiny and innovation to secure DRL applications against adversarial threats.