Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
173 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

TextEE: Benchmark, Reevaluation, Reflections, and Future Challenges in Event Extraction (2311.09562v3)

Published 16 Nov 2023 in cs.CL

Abstract: Event extraction has gained considerable interest due to its wide-ranging applications. However, recent studies draw attention to evaluation issues, suggesting that reported scores may not accurately reflect the true performance. In this work, we identify and address evaluation challenges, including inconsistency due to varying data assumptions or preprocessing steps, the insufficiency of current evaluation frameworks that may introduce dataset or data split bias, and the low reproducibility of some previous approaches. To address these challenges, we present TextEE, a standardized, fair, and reproducible benchmark for event extraction. TextEE comprises standardized data preprocessing scripts and splits for 16 datasets spanning eight diverse domains and includes 14 recent methodologies, conducting a comprehensive benchmark reevaluation. We also evaluate five varied LLMs on our TextEE benchmark and demonstrate how they struggle to achieve satisfactory performance. Inspired by our reevaluation results and findings, we discuss the role of event extraction in the current NLP era, as well as future challenges and insights derived from TextEE. We believe TextEE, the first standardized comprehensive benchmarking tool, will significantly facilitate future event extraction research.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (83)
  1. Meta-learning with dynamic-memory-based prototypical network for few-shot event detection. In The Thirteenth ACM International Conference on Web Search and Data Mining (WSDM).
  2. Ontoed: Low-resource event detection with ontology embedding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL/IJCNLP).
  3. The automatic content extraction (ACE) program - tasks, data, and evaluation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC).
  4. Xinya Du and Claire Cardie. 2020. Event extraction by answering (almost) natural questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  5. Multi-sentence argument linking. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL).
  6. Exploring the feasibility of chatgpt for event extraction. arXiv preprint: arXiv:2303.03836.
  7. ESTER: A machine reading comprehension dataset for reasoning about event semantic relations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  8. Revisiting event argument extraction: Can EAE models learn better when being aware of event co-occurrences? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL).
  9. Degree: A data-efficient generation-based event extraction model. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT).
  10. TAGPRIME: A unified framework for relational structure extraction. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL).
  11. AMPERE: amr-aware prefix for generation-based event argument extraction model. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL).
  12. Unified semantic typing with meaningful label inference. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL).
  13. Multilingual generative language models for zero-shot cross-lingual event argument extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL).
  14. Zero-shot transfer learning for event extraction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL).
  15. From simple to complex: A progressive framework for document-level informative argument extraction. In Findings of the Association for Computational Linguistics: EMNLP.
  16. Heng Ji and Ralph Grishman. 2008. Refining event extraction through cross-document inference. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics (ACL).
  17. Mixtral of experts. arXiv preprint: arXiv:2401.04088.
  18. Overview of genia event task in bionlp shared task 2011. In Proceedings of BioNLP Shared Task 2011 Workshop.
  19. The genia event extraction shared task, 2013 edition - overview. In Proceedings of the BioNLP Shared Task 2013 Workshop.
  20. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP).
  21. Event extraction from historical texts: A new dataset for black rebellions. In Findings of the Association for Computational Linguistics: ACL/IJCNLP.
  22. Event detection: Gate diversity and syntactic importance scores for graph convolution neural networks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  23. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL).
  24. Evaluating chatgpt’s information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness. arXiv preprint arXiv:2304.11633.
  25. Event extraction as multi-turn question answering. In Findings of the Association for Computational Linguistics: EMNLP.
  26. Clip-event: Connecting text and images with event structures. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, (CVPR)).
  27. Cross-media structured common space for multimedia event extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL).
  28. Treasures outside contexts: Improving event detection via global statistics. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  29. Document-level event argument extraction by conditional generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT).
  30. GLEN: general-purpose event detection for thousands of types. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  31. A joint neural model for information extraction with global features. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL).
  32. Event extraction as machine reading comprehension. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  33. Event detection via gated multilingual attention mechanism. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI).
  34. Saliency as evidence: Event detection with trigger saliency attribution. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL).
  35. Dynamic prefix-tuning for generative template-based event extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL).
  36. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
  37. Event extraction as question generation and answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL).
  38. Text2event: Controllable sequence-to-structure generation for end-to-end event extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL/IJCNLP).
  39. Unified structure generation for universal information extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL).
  40. A general framework for information extraction using dynamic span graphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT).
  41. Prompt for extraction? PAIE: prompting argument interaction for event argument extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL).
  42. Few-shot event detection: An empirical study and a unified view. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL).
  43. Contextualized soft prompts for extraction of event arguments. In Findings of the Association for Computational Linguistics: ACL 2023.
  44. Cross-task instance representation interactions and label dependencies for joint information extraction with graph convolutional networks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT).
  45. Structured prediction as translation between augmented natural languages. In 9th International Conference on Learning Representations (ICLR).
  46. Contextual label projection for cross-lingual structure extraction. arXiv preprint arXiv:2309.08943.
  47. GENEVA: benchmarking generalizability for event argument extraction with hundreds of event types and argument roles. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL).
  48. Omnievent: A comprehensive, fair, and easy-to-use toolkit for event understanding. arXiv preprint arXiv:2309.14258.
  49. The devil is in the details: On the pitfalls of event extraction evaluation. In Findings of the Association for Computational Linguistics: ACL.
  50. Uniex: An effective and efficient framework for unified information extraction via a span-extractive perspective. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL).
  51. Event extraction across multiple levels of biological organization. Bioinformatics, 28(18):575–581.
  52. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations.
  53. Is chatgpt a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476.
  54. Textual entailment for event argument extraction: Zero- and few-shot with multi-source learning. In Findings of the Association for Computational Linguistics: (NAACL).
  55. Textual entailment for event argument extraction: Zero- and few-shot with multi-source learning. In Findings of the Association for Computational Linguistics (NAACL).
  56. CASIE: extracting cybersecurity event information from text. In The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI).
  57. From light to rich ERE: annotation of entities, relations, and events. In Proceedings of the The 3rd Workshop on EVENTS: Definition, Detection, Coreference, and Representation, EVENTS@HLP-NAACL.
  58. PHEE: A dataset for pharmacovigilance event extraction from text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  59. Docee: A large-scale and fine-grained benchmark for document-level event extraction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL).
  60. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  61. Introducing a new dataset for event detection in cybersecurity texts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  62. Zephyr: Direct distillation of LM alignment. arXiv preprint: arXiv:2310.16944.
  63. MEE: A novel multilingual event extraction dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  64. Document-level event argument extraction via optimal transport. In Findings of the Association for Computational Linguistics: ACL 2022.
  65. Modeling document-level context for event detection via important context selection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  66. Entity, relation, and event extraction with contextualized span representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
  67. Query and extract: Refining event extraction as type-oriented binary decoding. In Findings of the Association for Computational Linguistics: ACL 2022.
  68. The art of prompting: Event detection based on type specific prompts. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL).
  69. Instructuie: Multi-task instruction tuning for unified information extraction. arXiv preprint arXiv:2304.08085.
  70. MAVEN: A massive general domain event detection dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  71. Code4struct: Code generation for few-shot event structure prediction. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL).
  72. CLEVE: contrastive pre-training for event extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL/IJCNLP).
  73. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS.
  74. Trigger is not sufficient: Exploiting frame-aware knowledge for implicit event argument extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL).
  75. A two-stream amr-enhanced model for document-level event argument extraction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL).
  76. Few-shot document-level event argument extraction. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL).
  77. Ea22{{}^{2}}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPTe: Improving consistency with event awareness for document-level argument extraction. In Findings of the Association for Computational Linguistics: (NAACL).
  78. ASER: A large-scale eventuality knowledge graph. In The Web Conference 2020 (WWW).
  79. Efficient zero-shot event extraction with context-definition alignment. In Findings of the Association for Computational Linguistics (EMNLP).
  80. Zixuan Zhang and Heng Ji. 2021. Abstract meaning representation guided graph encoding and decoding for joint information extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (NAACL-HLT).
  81. Knowledge-enhanced self-supervised prototypical network for few-shot event detection. In Findings of the Association for Computational Linguistics: (EMNLP).
  82. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685.
  83. Revisiting the evaluation of end-to-end event extraction. In Findings of the Association for Computational Linguistics: (ACL).
Citations (10)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now