WinoGrande: An Adversarial Winograd Schema Challenge at Scale (1907.10641v2)

Abstract: The Winograd Schema Challenge (WSC) (Levesque, Davis, and Morgenstern 2011), a benchmark for commonsense reasoning, is a set of 273 expert-crafted pronoun resolution problems originally designed to be unsolvable for statistical models that rely on selectional preferences or word associations. However, recent advances in neural LLMs have already reached around 90% accuracy on variants of WSC. This raises an important question whether these models have truly acquired robust commonsense capabilities or whether they rely on spurious biases in the datasets that lead to an overestimation of the true capabilities of machine commonsense. To investigate this question, we introduce WinoGrande, a large-scale dataset of 44k problems, inspired by the original WSC design, but adjusted to improve both the scale and the hardness of the dataset. The key steps of the dataset construction consist of (1) a carefully designed crowdsourcing procedure, followed by (2) systematic bias reduction using a novel AfLite algorithm that generalizes human-detectable word associations to machine-detectable embedding associations. The best state-of-the-art methods on WinoGrande achieve 59.4-79.1%, which are 15-35% below human performance of 94.0%, depending on the amount of the training data allowed. Furthermore, we establish new state-of-the-art results on five related benchmarks - WSC (90.1%), DPR (93.1%), COPA (90.6%), KnowRef (85.6%), and Winogender (97.1%). These results have dual implications: on one hand, they demonstrate the effectiveness of WinoGrande when used as a resource for transfer learning. On the other hand, they raise a concern that we are likely to be overestimating the true capabilities of machine commonsense across all these benchmarks. We emphasize the importance of algorithmic bias reduction in existing and future benchmarks to mitigate such overestimation.

• The paper presents a new benchmark dataset, WinoGrande, that exposes current AI limitations in commonsense reasoning with 44,000 carefully curated problems.
• The paper details an innovative dataset construction process combining crowdsourcing with the AfLite algorithm to minimize dataset-specific biases.
• The paper demonstrates that transfer learning on WinoGrande significantly improves state-of-the-art performance on related commonsense benchmarks.

An Examination of WinoGrande: Advancements in the Winograd Schema Challenge

The paper "WinoGrande: An Adversarial Winograd Schema Challenge at Scale" addresses a crucial challenge in the development of commonsense reasoning capabilities in AI. It presents a new benchmark dataset, WinoGrande, which consists of 44,000 problems inspired by the original Winograd Schema Challenge (WSC). The WSC is a widely used test for evaluating machine commonsense reasoning capabilities, comprising pronoun resolution problems that are trivial for humans but challenging for machines. However, advancements in neural LLMs have reached high accuracy levels on these tasks, raising questions about the true capabilities of AI in commonsense reasoning.

Dataset Construction

WinoGrande was developed to address the limitations of existing WSC datasets, which may overestimate the performance of LLMs due to dataset-specific biases. The authors introduce a novel dataset construction process that includes a carefully designed crowdsourcing protocol and a systematic bias reduction technique using the AfLite algorithm. AfLite is particularly noteworthy for its ability to identify and reduce spurious correlations in the dataset without the high computational cost typical of adversarial algorithms.

The dataset, unlike earlier versions, scales up the problem space significantly while maintaining the original complexity. Human participants solved WinoGrande problems with 94% accuracy, indicating that the challenges remain trivial for humans. However, state-of-the-art models, including RoBERTa, achieved only 79.1% accuracy, suggesting the dataset's increased difficulty for AI systems.

Empirical Results

The authors conducted extensive experiments to benchmark the performance of various models on WinoGrande, demonstrating that even sophisticated models like BERT and RoBERTa struggle with these tasks, particularly when trained with limited data. The learning curve of RoBERTa illustrates its performance improvement from 59% to 79% as the training dataset size increases from 800 to 41,000 instances. This finding highlights the dataset's ability to challenge AI, thereby offering a more accurate measure of machine commonsense reasoning capabilities.

Transfer Learning

The paper also explores the potential of WinoGrande as a resource for transfer learning. Tuning models on WinoGrande leads to new state-of-the-art results on several related benchmarks, such as the original WSC, PDP, COPA, and others. Significantly, despite the lack of direct relation to tasks such as the Cause and Effect Pairing (COPA), models trained on WinoGrande exhibited noteworthy improvements, underlining the dataset's utility in broadening commonsense reasoning applications.

Implications and Future Directions

The implications of this research are profound for the AI community. WinoGrande exemplifies a dynamic approach to benchmark design, suggesting that benchmarks need to evolve alongside technological advancements. By incorporating strategies like AfLite, it provides a robust mechanism for minimizing dataset-specific biases that can otherwise lead to inflated perceptions of AI capabilities.

Looking forward, the paper suggests that continued emphasis on debiasing and dynamic dataset generation will be critical. The results underscore the need for AI systems to achieve genuine progress in understanding and performing tasks that require human-like reasoning capabilities. It also opens avenues for further research into algorithmic innovations that can better align AI performance with genuine commonsense reasoning.

In conclusion, this work marks an important step in developing methodologies and datasets essential for advancing AI's commonsense reasoning capabilities. As we refine these tools and methods, the gap between human and machine understanding in AI tasks becomes more discernible, paving the way for more informed and effective AI systems in the future.