The Devil is in the Details: On the Pitfalls of Event Extraction Evaluation (2306.06918v2)

Abstract: Event extraction (EE) is a crucial task aiming at extracting events from texts, which includes two subtasks: event detection (ED) and event argument extraction (EAE). In this paper, we check the reliability of EE evaluations and identify three major pitfalls: (1) The data preprocessing discrepancy makes the evaluation results on the same dataset not directly comparable, but the data preprocessing details are not widely noted and specified in papers. (2) The output space discrepancy of different model paradigms makes different-paradigm EE models lack grounds for comparison and also leads to unclear mapping issues between predictions and annotations. (3) The absence of pipeline evaluation of many EAE-only works makes them hard to be directly compared with EE works and may not well reflect the model performance in real-world pipeline scenarios. We demonstrate the significant influence of these pitfalls through comprehensive meta-analyses of papers and empirical experiments. To avoid these pitfalls, we suggest a series of remedies, including specifying data preprocessing, standardizing outputs, and providing pipeline evaluation results. To help implement these remedies, we develop a consistent evaluation framework OMNIEVENT, which can be obtained from https://github.com/THU-KEG/OmniEvent.

• The paper reveals that variations in data preprocessing lead to non-comparable results in event extraction studies.
• The paper demonstrates that inconsistent output spaces from different modeling paradigms create evaluation challenges.
• The paper advocates for standardized pipeline evaluations using the OmniEvent framework to ensure realistic performance benchmarks.

Analysis of "The Devil is in the Details: On the Pitfalls of Event Extraction Evaluation"

The paper "The Devil is in the Details: On the Pitfalls of Event Extraction Evaluation" critically examines the evaluation methodologies in event extraction (EE). It emphasizes the challenges involved with evaluating EE systems due to inherent discrepancies in data preprocessing, output spaces, and evaluation practices, specifically highlighting the absence of pipeline evaluation.

Key Pitfalls in EE Evaluation

1. Data Preprocessing Discrepancy:

The paper identifies that differences in data preprocessing methods lead to non-comparable evaluation results. This occurs because EE datasets have complex, heterogeneous data formats involving elements like triggers, arguments, and entities. The authors note significant statistical differences in datasets—such as ACE 2005—caused by various preprocessing scripts. They emphasize that most EE research does not specify preprocessing steps, leading to a lack of reproducibility and comparability.

2. Output Space Discrepancy:

The paper highlights the inconsistencies in output spaces across different EE models due to varied modeling paradigms, such as classification, sequence labeling, and conditional generation. The paradigms produce differing output forms that result in incompatible evaluation metrics. This is compounded by issues in mapping predictions to annotations, which can significantly alter evaluation outcomes.

3. Absence of Pipeline Evaluation:

The authors underscore a gap between event detection (ED) and event argument extraction (EAE) research, partly due to EAE studies often evaluating systems using gold triggers, thus ignoring errors introduced in previous pipeline stages. This results in evaluations that may not reflect real-world scenarios, where triggers are predicted rather than given.

Proposed Remedies

The paper proposes remedies to address these pitfalls:

• Specifying Data Preprocessing: Advocating for standardized preprocessing methods and increased transparency about data handling in EE research to enhance result comparability.
• Standardizing Outputs: Introducing a method to align output spaces across different paradigms, helping to ensure consistency in evaluation metrics.
• Providing Pipeline Evaluation Results: Encouraging the inclusion of pipeline evaluations in EAE studies to assess system performance under realistic conditions.

OmniEvent Framework

To support the adoption of these remedies, the authors developed OmniEvent, a consistent evaluation framework. This framework provides preprocessing scripts for widely-used datasets, standardizes model outputs, and releases pre-trained triggers to facilitate consistent pipeline evaluations in future research.

Implications and Future Directions

This paper's contributions have significant implications for the EE community. By addressing these evaluation pitfalls, the research promotes more reliable, consistent benchmarks. This, in turn, could stimulate advancements in EE models by enabling accurate comparisons across diverse approaches.

Future research might extend the scope of this investigation to emerging datasets and languages, further refining evaluation consistency. Additionally, exploring methods to incorporate more complex, real-world scenarios into evaluations could enhance the robustness of EE systems.

In summary, this paper serves as a critical resource for improving EE evaluation methodologies, advocating for clearer benchmarks and reproducibility in the field.