Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2006.06195v2)

Abstract: We present VILLA, the first known effort on large-scale adversarial training for vision-and-language (V+L) representation learning. VILLA consists of two training stages: (i) task-agnostic adversarial pre-training; followed by (ii) task-specific adversarial finetuning. Instead of adding adversarial perturbations on image pixels and textual tokens, we propose to perform adversarial training in the embedding space of each modality. To enable large-scale training, we adopt the "free" adversarial training strategy, and combine it with KL-divergence-based regularization to promote higher invariance in the embedding space. We apply VILLA to current best-performing V+L models, and achieve new state of the art on a wide range of tasks, including Visual Question Answering, Visual Commonsense Reasoning, Image-Text Retrieval, Referring Expression Comprehension, Visual Entailment, and NLVR2.

Large-Scale Adversarial Training for Vision-and-Language Representation Learning

The paper presents "Villa," a pioneering framework employing large-scale adversarial training tailored for vision-and-language (V+L) representation learning. This novel approach distinguishes itself by integrating adversarial training at both the pre-training and fine-tuning phases, offering significant improvements across diverse V+L tasks.

Key Methodology

Villa's framework is anchored in two primary stages:

1. Adversarial Pre-training (APT): Villa leverages adversarial perturbations in the embedding space rather than traditional pixel or token-level perturbations. This stage is beneficial for developing models with enhanced generalization capabilities transferable to multiple downstream tasks.
2. Adversarial Fine-tuning (AFT): Post pre-training, Villa fine-tunes the model with adversarial perturbations tailored to specific tasks, which further enhances the model's robustness and accuracy.

The novel aspect of Villa is its choice to apply adversarial training in the embedding space for both modalities—text and images. This contrasts with typical adversarial training that often operates on raw data levels such as image pixels or token sequences. The adoption of a "free" adversarial training strategy facilitates scalability to large datasets by minimizing computational overheads, thus making it feasible to integrate into large-scale models.

Experimental Results and Significance

Villa is evaluated using state-of-the-art V+L models such as UNITER and demonstrates improved state-of-the-art results across six V+L tasks: Visual Question Answering (VQA), Visual Commonsense Reasoning (VCR), Image-Text Retrieval, Referring Expression Comprehension, Visual Entailment, and NLVR$^2$. Notably, the performance gains from applying Villa are consistent and significant. For instance, Villa-base enhances UNITER-base's performance on VQA from 72.91 to 73.67 on the test set. In VCR, Villa boosts accuracy significantly from 62.8 to 65.7.

Furthermore, Villa is versatile, successfully improving another V+L model, LXMERT, as well, thus demonstrating the framework's adaptability to different architectures.

Implications and Future Directions

The adversarial training strategy of Villa contributes to both theoretical and practical advancements in AI. Theoretically, it provides a rigorous method to enhance model robustness and generalization by utilizing adversarial examples effectively during both pre-training and fine-tuning phases. Practically, Villa's ability to scale adversarial training through efficient strategies like "free" adversarial training fills a critical gap in applying such methods to large and complex V+L datasets.

Future research could explore more sophisticated perturbation techniques in the embedding space and experiment with multimodal adversarial training approaches. Additionally, the paper leaves open the exploration of adversarial attacks on V+L models, particularly in creating semantically consistent adversarial examples in the vision-language context. This exploration could yield deeper insights into model vulnerabilities and lead to more robust systems.

In conclusion, Villa represents a significant step forward in leveraging adversarial training to enhance V+L representations, offering a template for integrating robustness and accuracy across multimodal AI tasks.