Cycle-Consistency for Robust Visual Question Answering (1902.05660v1)

Abstract: Despite significant progress in Visual Question Answering over the years, robustness of today's VQA models leave much to be desired. We introduce a new evaluation protocol and associated dataset (VQA-Rephrasings) and show that state-of-the-art VQA models are notoriously brittle to linguistic variations in questions. VQA-Rephrasings contains 3 human-provided rephrasings for 40k questions spanning 40k images from the VQA v2.0 validation dataset. As a step towards improving robustness of VQA models, we propose a model-agnostic framework that exploits cycle consistency. Specifically, we train a model to not only answer a question, but also generate a question conditioned on the answer, such that the answer predicted for the generated question is the same as the ground truth answer to the original question. Without the use of additional annotations, we show that our approach is significantly more robust to linguistic variations than state-of-the-art VQA models, when evaluated on the VQA-Rephrasings dataset. In addition, our approach outperforms state-of-the-art approaches on the standard VQA and Visual Question Generation tasks on the challenging VQA v2.0 dataset.

• The paper introduces a cycle-consistency framework that enhances VQA robustness by using integrated question answering and rephrasing tasks.
• It employs a training strategy with a gating mechanism and late activation to maintain semantic correctness and avoid mode collapse.
• Empirical results on VQA v2.0 and VQA-Rephrasings demonstrate significant improvements in consensus scores and overall model accuracy.

Cycle-Consistency for Robust Visual Question Answering

The paper "Cycle-Consistency for Robust Visual Question Answering" presents a model-agnostic framework aimed at enhancing the robustness of Visual Question Answering (VQA) models. This framework leverages cycle-consistency, an approach that simultaneously addresses question answering and generation tasks to improve model consistency across various linguistic rephrases of the same question. Despite advancements in VQA models, issues of brittleness to linguistic variations remain prevalent, prompting the authors to introduce an innovative dataset, VQA-Rephrasings, along with a novel evaluation protocol to systematically assess model robustness.

Methodological Approaches

The cornerstone of the proposed framework is a training strategy that incorporates cycle-consistency which involves the following sequence: a VQA model answers a given question about an image, then a Visual Question Generation (VQG) model produces semantically equivalent rephrases conditioned on the answer, and finally, the answer to the rephrased question is checked for consistency with the original answer. This process implicitly encourages models to maintain consistency in answering semantically similar questions, thereby enhancing their robustness to question rephrasings.

The framework introduces a gating mechanism and late activation strategy to overcome challenges related to semantic correctness and mode collapse during cycle-consistent training. The gating mechanism filters out invalid rephrases generated by the VQG module, while late activation ensures stabilization by delaying cycle-consistent training until sufficient initial learning has occurred.

Empirical Insights

The performance of cycle-consistent models was evaluated against standard state-of-the-art VQA models across multiple datasets, including the VQA v2.0 and the newly developed VQA-Rephrasings. Results demonstrate that models utilizing the proposed framework consistently achieve superior robustness to linguistic variations compared to their baseline counterparts. For instance, consensus scores derived from VQA-Rephrasings show significant improvements, indicating better consistency in model predictions across different rephrasings.

Models trained with cycle-consistency were also validated through standard VQA metrics, additionally achieving better accuracy on Visual Question Generation tasks compared to benchmark models like iQAN and iVQA. This suggests that the integrated cycle-consistency training fosters improved multi-modal understanding and linguistic stability.

Theoretical and Practical Implications

The introduction of cycle-consistency in a multi-modal setting such as VQA points to a promising avenue for bolstering the robustness and reliability of AI models handling complex visual and linguistic information. The framework's success without necessitating additional annotations further highlights a practical advantage in scaling VQA systems to real-world applications where linguistic diversity is rampant.

As an exploratory venture into making VQA models more adaptable to linguistic diversity, the paper paves the way for further studies on enhancing multi-modal interactions in AI. Future research might explore refining attention mechanisms or exploring more sophisticated gating processes to elevate consistency across wider semantic spectrums.

Overall, the cycle-consistent approach exemplified in the paper marks a thoughtful advancement towards resilient VQA models capable of handling the linguistic complexities inherent in human-machine interactions.