Enhance the Visual Representation via Discrete Adversarial Training (2209.07735v1)

Abstract: Adversarial Training (AT), which is commonly accepted as one of the most effective approaches defending against adversarial examples, can largely harm the standard performance, thus has limited usefulness on industrial-scale production and applications. Surprisingly, this phenomenon is totally opposite in NLP task, where AT can even benefit for generalization. We notice the merit of AT in NLP tasks could derive from the discrete and symbolic input space. For borrowing the advantage from NLP-style AT, we propose Discrete Adversarial Training (DAT). DAT leverages VQGAN to reform the image data to discrete text-like inputs, i.e. visual words. Then it minimizes the maximal risk on such discrete images with symbolic adversarial perturbations. We further give an explanation from the perspective of distribution to demonstrate the effectiveness of DAT. As a plug-and-play technique for enhancing the visual representation, DAT achieves significant improvement on multiple tasks including image classification, object detection and self-supervised learning. Especially, the model pre-trained with Masked Auto-Encoding (MAE) and fine-tuned by our DAT without extra data can get 31.40 mCE on ImageNet-C and 32.77% top-1 accuracy on Stylized-ImageNet, building the new state-of-the-art. The code will be available at https://github.com/alibaba/easyrobust.

• The paper proposes a novel approach by converting continuous images into discrete 'visual words' to enhance model robustness and generalization.
• It leverages VQGAN for image discretization and applies symbolic adversarial perturbations with a straight-through gradient for efficient training.
• Empirical evaluations on models like ResNet and ViT demonstrate improved performance in classification, detection, and self-supervised learning tasks.

Enhance the Visual Representation via Discrete Adversarial Training

In "Enhance the Visual Representation via Discrete Adversarial Training," the authors explore the potential of Discrete Adversarial Training (DAT) for improving both the robustness and generalization of vision models. This work emerges from the observation that while adversarial training (AT) typically impairs standard performance in computer vision tasks, it has been beneficial in NLP, attributed to the discrete and symbolic nature of language inputs. The authors propose adapting this NLP-style adversarial training to vision tasks through a novel methodology that transforms continuous image inputs into discrete, text-like representations using VQGAN, creating what they term "visual words."

Methodology

The core idea behind DAT is to leverage the symbolic nature of LLMs and apply it to computer vision. The methodology involves the following key steps:

• Image Discretization: VQGAN is employed to convert image inputs into discrete visual dictionaries, breaking them into segments or "visual words."
• Symbolic Adversarial Perturbations: Using these discrete representations, the method applies adversarial training techniques by adding perturbations that are conceptual and semantic rather than pixel-based, which aligns more closely with real-world adversarial examples, such as typographical errors in language.
• Training Strategy: The adversarial inputs formulated in this discrete space are utilized to train models, aiming to maximize robustness without trading off clean performance.

From a computational perspective, the authors note that while traditional adversarial training demands significant resources, DAT reduces this cost by employing a straight-through gradient approximation for efficient adversarial example generation.

Empirical Evaluation

The authors conduct thorough empirical evaluations across different tasks, architectures, and benchmarks to showcase the efficacy of DAT. They experiment with DAT on various models, from ResNet to Vision Transformers (ViTs), and tasks including image classification, object detection, and self-supervised learning, yielding strong results:

• Image Classification: DAT demonstrates substantial improvements in both clean accuracy and robustness across multiple datasets, including ImageNet-C and Stylized-ImageNet.
• Object Detection and Self-Supervised Learning: The findings suggest that DAT-trained models exhibit enhanced robustness to various corruptions and adversarial perturbations while simultaneously improving generalization capabilities.

Theoretical Insight and Implications

This paper introduces a novel perspective by suggesting that robust visual representation learning can benefit from the discrete representations commonly used in NLP. By providing evidence that DAT propagates the distribution closer to that of clean data, this work challenges traditional adversarial training paradigms and presents discrete adversarial training as a promising direction for future research in achieving more human-like visual processing in deep learning models.

Future Directions

The methodology outlined underscores significant theoretical implications for machine perception, leading to several potential research avenues:

• Enhancing Visual Discretizers: More efficient algorithms that improve or replace VQGAN could further optimize discretization for adversarial training.
• Training Efficiency: Investigating methods to reduce the computational costs associated with DAT could expand its application scope.
• Interpretable Robustness: Examining the interpretability of robustness improvements through discrete adversarial examples may yield new insights into model behavior and adversaries.

This paper provides a substantial contribution to the field, enhancing our understanding of how discrete representations can be integrated into adversarial training frameworks to build more robust and generalizable machine learning models.