SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (1808.05326v1)

Abstract: Given a partial description like "she opened the hood of the car," humans can reason about the situation and anticipate what might come next ("then, she examined the engine"). In this paper, we introduce the task of grounded commonsense inference, unifying natural language inference and commonsense reasoning. We present SWAG, a new dataset with 113k multiple choice questions about a rich spectrum of grounded situations. To address the recurring challenges of the annotation artifacts and human biases found in many existing datasets, we propose Adversarial Filtering (AF), a novel procedure that constructs a de-biased dataset by iteratively training an ensemble of stylistic classifiers, and using them to filter the data. To account for the aggressive adversarial filtering, we use state-of-the-art LLMs to massively oversample a diverse set of potential counterfactuals. Empirical results demonstrate that while humans can solve the resulting inference problems with high accuracy (88%), various competitive models struggle on our task. We provide comprehensive analysis that indicates significant opportunities for future research.

• The paper introduces an innovative adversarial filtering process that refines the SWAG dataset by mitigating annotation artifacts.
• The dataset contains 113,000 multiple-choice questions where human accuracy reaches 88% while models score below 60%.
• The work advances AI research by emphasizing realistic commonsense reasoning and establishing a new benchmark for grounded inference.

Swag: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference

The paper introduces Swag, a dataset comprising 113,000 multiple-choice questions designed to advance research in grounded commonsense inference. This task seeks to unify natural language inference with commonsense reasoning, focusing on predicting plausible future events given a context. The dataset leverages everyday situations to challenge LLMs, countering typical annotation artifacts and biases that have limited the efficacy of existing datasets.

Methodology

To mitigate annotation artifacts, the paper presents Adversarial Filtering (AF), a process that iteratively uses a set of trained stylistic models to refine a dataset. A state-of-the-art LLM generates a plethora of potential endings for a given context, from which a committee of models filter out easily discernible patterns, thus fostering dataset diversity. Ultimately, human annotations ensure quality and validate both plausible and implausible endings, maintaining a dataset reliable for challenging machine understanding.

Empirical Findings

Swag is particularly noteworthy for its scale and robustness. Human performance on the dataset reaches an accuracy of 88%, indicating the intuitive nature of the task for people. However, state-of-the-art models achieve less than 60% accuracy, highlighting the distinct gap in machine understanding of everyday scenarios. This result underscores the dataset's potential to challenge existing models and promote advancements in AI's understanding of commonsense reasoning.

Implications

Swag represents a substantial contribution to the development of AI systems capable of understanding and predicting commonsense scenarios. The adversarial filtering technique promises improvements in dataset construction, reducing the possibility of models relying on biases rather than genuine understanding. Future developments in AI could markedly benefit from adopting similar methodologies that emphasize realistic commonsense reasoning.

Speculation and Future Work

Swag invites further exploration into models that can effectively harness temporal and contextual information for commonsense reasoning. The dataset's release could inspire the development of new architectures or training mechanisms tailored to the demands of grounded NLI. Moreover, the AF methodology stands to serve as a blueprint for creating high-quality, unbiased datasets across various domains, fundamentally altering how we approach machine learning datasets and their inherent challenges.

In summary, Swag provides an important benchmark for grounded commonsense inference, encouraging the development of AI systems that more closely emulate human reasoning and understanding. This work not only introduces a novel dataset but also fosters methodological innovations that promise to refine future models in natural language processing.