Cutting through buggy adversarial example defenses: fixing 1 line of code breaks Sabre

Abstract: Sabre is a defense to adversarial examples that was accepted at IEEE S&P 2024. We first reveal significant flaws in the evaluation that point to clear signs of gradient masking. We then show the cause of this gradient masking: a bug in the original evaluation code. By fixing a single line of code in the original repository, we reduce Sabre's robust accuracy to 0%. In response to this, the authors modify the defense and introduce a new defense component not described in the original paper. But this fix contains a second bug; modifying one more line of code reduces robust accuracy to below baseline levels. After we released the first version of our paper online, the authors introduced another change to the defense; by commenting out one line of code during attack we reduce the robust accuracy to 0% again.

• The paper shows that a one-line code fix in Sabre completely breaks the defense, reducing robust accuracy to nearly 0%.
• It identifies critical evaluation flaws such as gradient masking, absence of adaptive attacks, and misleading baseline comparisons.
• The study underscores the need for rigorous, adaptive evaluation protocols in adversarial robustness research to ensure credible defenses.

Evaluation Flaws in Adversarial Defense Mechanism Sabre

The paper "Cutting through buggy adversarial example defenses: fixing 1 line of code breaks Sabre" by Nicholas Carlini critiques the effectiveness of Sabre, a defense mechanism purportedly providing robustness against adversarial examples. The evaluation uncovers significant flaws that stem from improper validation methodologies and hidden bugs, ultimately debunking the robust accuracy claims made in the original Sabre research.

Introduction and Main Claims

Sabre, introduced in context of adversarial defenses, claims to augment robustness significantly beyond the existing state-of-the-art. It purportedly brings down adversarial success rates on the CIFAR-10 dataset from 29% to 11% for ε = 8/255. Despite these promising claims, this critique reveals several fundamental flaws in the evaluation protocol of Sabre, primarily aggravated by gradient masking and code bugs.

Identified Flaws and Methodology

Several claims made by Sabre are identified as mathematically impossible or highly suspect:

1. Non-trivial Accuracy at High Perturbations:
   • The paper critiques Sabre's reported accuracy of around 92.66% at a perturbation level of ε = 0.5, which contradicts established theoretical expectations where models, given such distortion, should perform no better than random guessing.
2. Paradoxical Performance Enhancement Under Attack:
   • Sabre claims to perform better under specific adversarial attacks compared to no attack scenarios, defying standard adversarial robustness principles. This phenomenon often indicates overfitting and significant evaluation flaws.
3. Absence of Adaptive Attacks:
   • The methodology in Sabre lacks an evaluation using adaptive attacks specifically tailored to its defense mechanisms. This deviates from the recommended robust benchmarking practices that mandate such evaluations to assert meaningful robustness.

Technical Breakdown of Sabre

Sabre operates as a pre-processor defense. The attacker targets the standard neural network classifier after the input undergoes transformation by Sabre. However, the critique reveals that merely removing a BPDA wrapper, inserted unnecessarily into the code, is sufficient to break the defense by reducing accuracy to 0%.

Series of Breaks and Responses

Upon identifying the primary bug and fixing it, the robust accuracy of Sabre drops to negligible levels. The authors of Sabre attempted a fix, introducing a discretization component to the defense, yet another bug was identified in this fix, demonstrating improper gradient handling.

First Break:

By simply modifying the code to remove the BPDA wrapper around the pre-processing function, the attack success rate climbs to nearly 100%, rendering the defense ineffective.

Second Break:

With the discretization fix, Sabre introduces non-differentiability while claiming differentiability, a direct contradiction. Utilizing a small change by reinserting BPDA, the new defense was also effectively nullified, highlighting pervasive gradient masking and flawed gradient computation.

Broader Evaluation Issues

Apart from specific code bugs, the paper criticizes Sabre for neglecting standard robust evaluation protocols:

• Iterative vs. Single-step Attacks:
  • Sabre reports higher robustness against iterative attacks versus single-step ones, contradicting typical adversarial robustness behavior.
• Inappropriate Baseline Comparisons:
  • Incorrect baseline implementations, notably adversarial training, distort comparative robustness claims. This introduces potential misrepresentation of Sabre's merit.
• Ignoring Gradient-Free and Hard-Label Attacks:
  • Lack of evaluation against such attacks further undermines the legitimacy of robustness claims.

Implications and Future Directions

This critique serves as a stark reminder of the rigorous evaluation needed in adversarial robustness research. The identified flaws pose serious questions on the reviewing process, highlighting the necessity for meticulous scrutiny, especially given that practical deployment increasingly relies on proven robustness.

Conclusion

Repeated acceptance of flawed defense mechanisms such as Sabre at prominent conferences erodes confidence in proposed robustness claims. The paper underscores that, as a community, adversarial machine learning researchers must ensure rigid adherence to established best practices to advance the field credibly. Moreover, future work must address structural process failures in evaluation standards to reliably identify genuinely robust models capable of withstanding adversarial perturbations in real-world applications.