Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples (1802.00420v4)

Abstract: We identify obfuscated gradients, a kind of gradient masking, as a phenomenon that leads to a false sense of security in defenses against adversarial examples. While defenses that cause obfuscated gradients appear to defeat iterative optimization-based attacks, we find defenses relying on this effect can be circumvented. We describe characteristic behaviors of defenses exhibiting the effect, and for each of the three types of obfuscated gradients we discover, we develop attack techniques to overcome it. In a case study, examining non-certified white-box-secure defenses at ICLR 2018, we find obfuscated gradients are a common occurrence, with 7 of 9 defenses relying on obfuscated gradients. Our new attacks successfully circumvent 6 completely, and 1 partially, in the original threat model each paper considers.

• The paper demonstrates that defenses using obfuscated gradients create a false sense of security by misleading gradient-based attack methods.
• It introduces effective strategies like BPDA, EOT, and reparameterization to counter shattered, stochastic, and unstable gradient challenges.
• Empirical evaluations on nine defenses reveal critical vulnerabilities, urging a shift toward more rigorous and transparent robustness assessments.

Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples

The paper "Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples," authored by Anish Athalye, Nicholas Carlini, and David Wagner, provides a nuanced evaluation of several machine learning defenses aimed at countering adversarial attacks. The principal focus is on identifying and overcoming a phenomenon coined as "obfuscated gradients," which the authors find to be a common pitfall in numerous adversarial defenses.

Overview

The authors argue that while defenses leveraging obfuscated gradients appear to resist iterative optimization-based attacks, they ultimately fail under more rigorous attack methodologies. Obfuscated gradients are identified as a subclass of gradient masking, where the gradients are either non-existent or misleading, thereby hindering the efficacy of gradient-based attacks. The paper categorizes obfuscated gradients into three types:

1. Shattered Gradients: These are caused either by non-differentiable operations or numerical instability, leading to incorrect or non-existent gradients.
2. Stochastic Gradients: These arise due to randomness at test time, making the gradient-based optimizations unreliable.
3. Vanishing/Exploding Gradients: These are typically associated with deep computation paths within the network which result in gradients that are either too small or too large for practical use.

The paper not only identifies these types but also proposes specific strategies to counteract them, thereby overcoming the apparent robustness provided by these gradients.

Detailed Techniques

Backward Pass Differentiable Approximation (BPDA)

To tackle shattered gradients, the authors propose BPDA, a method where the forward pass through the network is computed normally, but on the backward pass, the gradients are approximated using a differentiable construct. This allows gradient-based optimization techniques to successfully generate adversarial examples even when faced with non-differentiable defenses.

Expectation Over Transformation (EOT)

For defenses incorporating stochastic gradients due to input transformations or stochastic classifiers, the paper extends the EOT technique. By optimizing over the expected transformation, EOT captures the true gradient across different transformations, thus nullifying the defense’s reliance on randomness.

Reparameterization

In the presence of vanishing or exploding gradients, the authors suggest reparameterization techniques, effectively bypassing the problematic gradient paths by optimizing within a reparameterized space where gradients are more stable.

Empirical Findings

To validate these attack methodologies, the authors conducted a comprehensive case paper on nine defenses presented at ICLR 2018 that claim robustness in a white-box setting. The empirical results are insightful:

• Out of nine defenses scrutinized, seven employed obfuscated gradients.
• The proposed attacks completely circumvented six of these defenses and partially overcame one under the original evaluation settings.
• For instance, on thermostat encoded networks and input transformation defenses, BPDA and EOT significantly reduced the model accuracy, thereby demonstrating the fallibility of these defenses against adaptive attacks.

Implications

The findings imply that defenses should not rely on obfuscated gradients for security. Instead, robust defenses should exhibit characteristics such as:

• Gradients that are meaningful and useful under practical attack scenarios.
• Comprehensive evaluation methodologies, including adaptive attacks, that simulate realistic threat models.
• Transparent and reproducible experiments, allowing for scrutiny and validation by the research community.

Future Directions

The research underscores the importance of ensuring that defenses can withstand an array of attack methodologies, not just existing ones. Future developments in adversarial machine learning should aim at creating defenses that truly improve robustness without inadvertently introducing weak spots exploitable by nuanced attacks. Furthermore, integrating provable security guarantees and broad-spectrum evaluation metrics is essential.

In summary, this paper effectively dissects the limitations of current adversarial defenses relying on obfuscated gradients and highlights the necessity for more resilient and comprehensively evaluated approaches in adversarial machine learning. The proposed attack methodologies provide a framework for rigorously testing and enhancing the robustness of neural network defenses.