MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering (2009.08566v2)

Abstract: While progress has been made on the visual question answering leaderboards, models often utilize spurious correlations and priors in datasets under the i.i.d. setting. As such, evaluation on out-of-distribution (OOD) test samples has emerged as a proxy for generalization. In this paper, we present MUTANT, a training paradigm that exposes the model to perceptually similar, yet semantically distinct mutations of the input, to improve OOD generalization, such as the VQA-CP challenge. Under this paradigm, models utilize a consistency-constrained training objective to understand the effect of semantic changes in input (question-image pair) on the output (answer). Unlike existing methods on VQA-CP, MUTANT does not rely on the knowledge about the nature of train and test answer distributions. MUTANT establishes a new state-of-the-art accuracy on VQA-CP with a $10.57\%$ improvement. Our work opens up avenues for the use of semantic input mutations for OOD generalization in question answering.

• The paper presents MUTANT, a training paradigm that applies semantic mutations to both visual and textual inputs to enhance VQA's out-of-distribution performance.
• It employs techniques like noise-contrastive estimation and pairwise consistency loss to mitigate bias from spurious dataset correlations.
• Experimental results demonstrate a 10.57% accuracy improvement on VQA benchmarks, underscoring its effectiveness in numeric reasoning and semantic understanding.

Overview of MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering

The paper "MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering" addresses one of the core challenges in visual question answering (VQA) models: improving performance on out-of-distribution (OOD) samples. This is crucial due to the innate biases in commonly used i.i.d. settings, which can lead models to rely heavily on spurious dataset correlations. To tackle this, the authors propose MUTANT, a training paradigm that introduces semantic mutations in input data—either the image or the question—during training to expose the model to perceptually similar but semantically different scenarios. This aims to enhance OOD generalization capabilities, as validated by achieving a new state-of-the-art accuracy on the VQA-CP benchmark, with an improvement of 10.57%.

Key Contributions and Methodology

The paper introduces several innovative components that set MUTANT apart from previous methodologies:

1. Input Mutation: At the core of this training paradigm is the idea of generating mutant inputs by performing small transformations that alter the semantic meaning of the image and/or the question, leading to a different answer. Three categories of transformations are explored: addition, removal, and substitution, demystified through procedural examples like changing the color of objects or inverting question polarity.
2. Answer Projection and Noise-Contrastive Estimation (NCE): The method introduces a novel training strategy where each VQA instance and its answers are projected into a shared embedding space. NCE loss is used to measure similarity on this manifold, helping the model perceive nuanced semantic shifts between original and mutated samples, thereby reducing dependence on dataset-specific correlations.
3. Type Exposure: The approach challenges models to recognize potential answers for a given question type beyond dataset-induced biases. For instance, models are trained to consider all plausible answers for "What color is ... ?" irrespective of the frequency skewing in training datasets.
4. Pairwise Consistency Loss: The authors incorporate a consistency-constrained training objective ensuring the output predictions for original and mutated sample pairs reflect similar semantic shifts as witnessed between their true answer projections.

Results and Implications

The experimental results substantially highlight the effectiveness of MUTANT, where models integrated with this paradigm surpassed previous baselines in VQA-CP-v2 and VQA-v2 settings. Noteworthy is the improvement seen across all question categories—particularly yes/no and number-based queries, where reliance on linguistic and visual priors is statistically attenuated. Additionally, the paired training significantly enhanced numeric reasoning capability, emphasizing the rigorous effect of the pairwise consistency mechanism.

The implications of this research are substantial, teaching us how semantic manipulation of inputs can reinforce model understanding and generalization without requiring prior knowledge of test distributions. This approach essentially advocates refining biases rather than eradicating them entirely, underlining a paradigm shift from traditional de-biasing techniques.

Future Directions

The paper opens up several avenues for future research. Expansion of semantic mutation concepts into other domains such as image classification and scene representation tasks appears promising. The effectiveness of structured perturbations highlights a new avenue for AI systems to develop "what if" reasoning capabilities, potentially transforming disparate fields where understanding context and semantics is pivotal.

In conclusion, while MUTANT sets a new benchmark in VQA, its approach can catalyze advancements beyond current out-of-distribution methodologies, fostering generalization prowess that adapts seamlessly across a broader spectrum of AI applications.