Did the Model Understand the Question? (1805.05492v1)

Abstract: We analyze state-of-the-art deep learning models for three tasks: question answering on (1) images, (2) tables, and (3) passages of text. Using the notion of \emph{attribution} (word importance), we find that these deep networks often ignore important question terms. Leveraging such behavior, we perturb questions to craft a variety of adversarial examples. Our strongest attacks drop the accuracy of a visual question answering model from $61.1\%$ to $19\%$, and that of a tabular question answering model from $33.5\%$ to $3.3\%$. Additionally, we show how attributions can strengthen attacks proposed by Jia and Liang (2017) on paragraph comprehension models. Our results demonstrate that attributions can augment standard measures of accuracy and empower investigation of model performance. When a model is accurate but for the wrong reasons, attributions can surface erroneous logic in the model that indicates inadequacies in the test data.

• The paper evaluates deep learning models for visual, tabular, and text question answering, revealing their common weakness in fully understanding question semantics.
• Adversarial attacks exploiting this weakness significantly reduce model accuracy across diverse QA tasks, highlighting a gap between performance metrics and true comprehension.
• Attribution techniques help diagnose model behavior, showing reliance on superficial features rather than deep understanding, and suggest a need for more robust evaluation and training methods.

An Evaluation of Model Comprehension in Question Answering Systems

The paper "Did the Model Understand the Question?" offers a critical analysis of deep learning models utilized in various question answering (QA) tasks. The authors focus on models that handle visual, tabular, and textual data, revealing common weaknesses across these systems: the tendency to overlook crucial question terms. By exploiting this shortcoming, the paper systematically constructs adversarial examples, illustrating how these attacks drastically reduce model accuracy.

The paper explores the performance of models on three distinct question answering tasks:

1. Visual Question Answering (VQA): The authors examine a VQA model tasked with interpreting questions about images. They demonstrate that the model frequently disregards significant words, leading to a stable performance that remains unaffected even when essential question terms are removed. By appending non-informative prefixes to questions, model accuracy plummeted from its reported performance to just 19%. This suggests a model over-reliant on image features rather than comprehensively understanding the question content.
2. QA on Tables: For the analysis of the Neural Programmer model, the authors reveal a dependency on generic words rather than critical content-related terms. They find that the model often outputs the correct answer, but due to incidental alignment between the question and tabular content, leading to incorrect answers when tables are modified. By introducing innocuous changes in word usage or table structure, they reduce model accuracy to 3.3%. This illustrates a weakness in relying on superficial features rather than the semantic essence of the question.
3. Reading Comprehension: The investigation into reading comprehension models, particularly one evaluated on the SQuAD dataset, highlights the susceptibility to adversarial sentences designed by altering key question words. These attacks were effective in almost half the attempts. The attribution approach reveals that models could be misled by sentences that retained high-attribution question terms, emphasizing that superficial alignment rather than deep semantic understanding drives the model's performance.

Technical Contributions

• Attribution Techniques: The use of Integrated Gradients (IG) provides insights into which parts of the question contribute to model predictions. The effectiveness of IG is demonstrated by perturbing high-attribution terms, thereby confirming the model's reliance on certain words.
• Model Sensitivity Analysis: Through overstability tests, the researchers quantify how a model maintains accuracy despite the removal of important contextual words. This is an important step in identifying the model's over-reliance on non-content words and the corresponding lack of deeper comprehension.
• Adversarial Attacks: The creation and implementation of adversarial examples expose vulnerabilities in QA systems, suggest improvements in robustness, and propose augmentations to current evaluation methods.

Implications and Speculation on Future AI Developments

The findings highlight that while QA models may achieve high accuracy metrics, their actual understanding of question semantics can be superficial. This represents a significant gap between statistical performance metrics and real-world applicability. The authors propose that future models should integrate better context representation and reasoning capabilities, perhaps through incorporating inductive biases or enhanced training data that encourage a deeper semantic comprehension.

Moreover, the use of attribution-based techniques demonstrates potential in developing better diagnostic tools for AI models, offering transparent insights into decision-making processes and guiding the development of more robust systems. Exposing end-users to attribution-based explanations could also enhance trust and reliability in AI-driven systems.

In conclusion, the paper provides a comprehensive critique of existing QA models, underlining the necessity for enriched dataset representation and enhanced training methodologies. It serves as a foundational piece that encourages a closer examination of AI model comprehension beyond surface-level accuracy, aiming for systems that genuinely understand and reason with human-like precision.