NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark (2310.18018v1)

Abstract: In this position paper, we argue that the classical evaluation on NLP tasks using annotated benchmarks is in trouble. The worst kind of data contamination happens when a LLM is trained on the test split of a benchmark, and then evaluated in the same benchmark. The extent of the problem is unknown, as it is not straightforward to measure. Contamination causes an overestimation of the performance of a contaminated model in a target benchmark and associated task with respect to their non-contaminated counterparts. The consequences can be very harmful, with wrong scientific conclusions being published while other correct ones are discarded. This position paper defines different levels of data contamination and argues for a community effort, including the development of automatic and semi-automatic measures to detect when data from a benchmark was exposed to a model, and suggestions for flagging papers with conclusions that are compromised by data contamination.

The paper "NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark" raises critical concerns about the evaluation methodologies in NLP, particularly highlighting the problematic impact of data contamination. Data contamination occurs when a LLM is trained using data that overlaps with the test split of a benchmark, leading to skewed and overestimated performance metrics.

The authors argue that the extent of this contamination is largely unknown, mainly because it is challenging to detect and measure. They identify different levels of contamination and note that such contamination leads to inaccurate evaluations. This misrepresentation can substantially impact the field, potentially fostering incorrect scientific conclusions while proper insights might be disregarded.

Key points discussed in the paper include:

1. Definition and Levels of Contamination: The paper elaborates on various levels of data contamination, ranging from minor overlaps to complete exposure of test data to training data. These levels signify different degrees of performance inflation and their potential harm.
2. Harmful Consequences: The authors underscore the negative ramifications of data contamination, emphasizing that it can lead researchers to draw false conclusions about the effectiveness of LLMs. Such misconceptions may divert future research paths and undermine the foundation of empirical NLP work.
3. Need for Detection Mechanisms: The authors call for the development of both automatic and semi-automatic tools to detect instances of data contamination. They advocate for the community to take an active role in creating these detection measures to ensure the credibility and integrity of NLP research.
4. Flagging Compromised Research: As a remedial measure, the paper suggests implementing a system to flag publications that potentially involve contaminated data. This would help in acknowledging compromised conclusions and maintaining transparency within the research community.

The overarching message of the paper is a call-to-action for the NLP community to develop rigorous mechanisms for identifying and mitigating data contamination. By fostering more stringent evaluation standards, the field can progress with a more reliable foundation, ensuring that advancements are based on sound scientific principles.