OmniEvent: Unified Event Toolkit

• OmniEvent is a comprehensive framework that integrates event detection, argument extraction, and event relation extraction into a unified toolkit.
• It standardizes preprocessing and evaluation across 15 English and Chinese datasets to ensure fair model comparisons.
• Its modular architecture supports multiple modeling paradigms and offers off-the-shelf models, online demos, and extensibility for diverse applications.

OmniEvent is a research software framework for event understanding in natural language processing that was introduced to cover the full event pipeline—event detection, event argument extraction, and event relation extraction—under a single comprehensive, fair, and easy-to-use toolkit with support for mainstream modeling paradigms and 15 widely used English and Chinese datasets (Peng et al., 2023). In arXiv usage, the same name was later reused for a distinct event-based vision framework for event cameras, so the term is context-dependent across subfields (Yan et al., 3 Aug 2025).

1. Scope within event understanding

In the NLP literature, OmniEvent is defined around three tightly linked subtasks. Event Detection (ED) locates triggers, namely words or phrases that evoke events, and assigns each a type label. Event Argument Extraction (EAE) identifies the participants of each trigger and assigns argument roles. Event Relation Extraction (ERE) predicts relations between events, including temporal, causal, coreference, and subevent relations (Peng et al., 2023).

This framing matters because event understanding is often operationalized more narrowly as event extraction, covering only ED and EAE. OmniEvent explicitly extends the scope to ERE and therefore treats event understanding as a broader structured prediction problem rather than a two-stage extraction pipeline. For ERE, the toolkit implements a unified pairwise classification framework; for event coreference, it additionally provides an antecedent-ranking approach, and it also includes a simple joint model that predicts multiple relation types in concert (Peng et al., 2023).

A recurrent misconception in this area is that benchmark numbers across ED, EAE, and ERE are directly comparable whenever the same dataset name is reported. OmniEvent was designed in part to contest that assumption by standardizing not only task coverage but also the evaluation interface across paradigms and trigger conditions. This suggests that the toolkit is as much an evaluation infrastructure as it is a model library.

2. Modeling paradigms and data unification

OmniEvent’s “comprehensive” design is defined along two axes: modeling coverage and dataset coverage. For ED and EAE, it supports four mainstream paradigms: classification, sequence labeling, span prediction, and conditional generation. Representative models named in the toolkit description include DMBERT and CLEVE for classification, BERT+CRF for sequence labeling, EEQA for span prediction, and Text2Event for conditional generation (Peng et al., 2023).

Task family	Supported paradigms	Representative models
ED / EAE	Classification, sequence labeling, span prediction, conditional generation	DMBERT, CLEVE, BERT+CRF, EEQA, Text2Event
ERE	Unified pairwise classification; antecedent-ranking for coreference; simple joint model	Coreference, temporal, causal, subevent relations

The dataset layer is equally central. The toolkit description states that OmniEvent processes 15 widely used English and Chinese datasets and provides preprocessing scripts that convert source annotations into a single unified JSON format. The English EE corpora named are ACE 2005, TAC-KBP (2014–17), RichERE, MAVEN, and FewFC; the Chinese EE corpora named are ACE 2005 (zh), LEVEN, and DuEE; the ERE corpora named are MAVEN-ERE, ACE 2005, TB-Dense, MATRES, TCR, CausalTB, EventStoryLine, and HiEve. The unified JSON format records "text", "events" with triggers, offsets, argument mentions, and roles, and "event-relations" with typed links between trigger IDs (Peng et al., 2023).

The earlier evaluation-oriented description of OMNIEVENT presents the preprocessed files as train.jsonl, dev.jsonl, and test.jsonl, with sentence-level fields such as "tokens", "trigger_annotations", and "argument_annotations" (Peng et al., 2023). Taken together, these descriptions indicate an evolution from a consistent evaluation framework for event extraction into a broader event understanding toolkit with a unified internal representation.

3. Fairness as an evaluation principle

The “fair” component of OmniEvent is rooted in the three evaluation pitfalls identified by Peng et al. (2023): data-preprocessing discrepancy, output-space discrepancy, and the absence of pipeline evaluation for EAE when ED is imperfect (Peng et al., 2023). These are not cosmetic details. In the meta-analysis summarized by the evaluation paper, different ACE 2005 preprocessing scripts yielded different numbers of candidate triggers, reported as 5,055 versus 5,349, and different numbers of arguments, reported as 6,040 versus 9,683. Among 116 recent ACL, EMNLP, NAACL, and COLING papers, three preprocessing scripts covered only 37% of the literature, and swapping scripts under identical model code led to up to 10 F$_1$ points difference in trigger detection and up to 15 points in argument extraction (Peng et al., 2023).

OmniEvent addresses preprocessing discrepancy by shipping the widely used scripts and requiring users to select one canonical pipeline. It addresses output-space discrepancy by converting heterogeneous outputs into a shared internal representation. The toolkit names these conversion routines as convert_SL, convert_SP, and convert_CG, so that a BIO tag sequence, a span prediction, or a generation output all map to the same trigger-boundary-plus-type representation (Peng et al., 2023).

The evaluation framework formalizes this standardization through a deterministic mapping from raw predictions to a classification-candidate space. Let $C=\{c_1,\ldots,c_n\}$ be the candidate set and let $\mathcal{P}$ be the raw predictions. Then

$g:\mathcal{P}\rightarrow C\cup\{\varnothing\}$

maps each prediction to the unique candidate with identical start, end, and label, or discards it as $\varnothing$ if no such candidate exists (Peng et al., 2023). After standardization, precision, recall, and F$_1$ are computed in the usual way:

$$Precision = \frac{TP}{TP + FP}, \qquad Recall = \frac{TP}{TP + FN}, \qquad F_1 = 2 \times \frac{Precision \times Recall}{Precision + Recall}.$$

For event coreference, the toolkit also reports B-cubed (Peng et al., 2023).

The third remedy is pipeline evaluation. OmniEvent releases fixed predicted triggers from a strong CLEVE model for ACE 2005, KBP 2016/17, and RichERE, so that EAE systems are evaluated on the same trigger set rather than on incomparable detector outputs (Peng et al., 2023). In the earlier OMNIEVENT formulation, this appears as a default trigger_source: predicted setting, with gold-trigger evaluation retained for diagnosis (Peng et al., 2023). A plausible implication is that the framework is designed to reduce variance caused by hidden experimental degrees of freedom rather than by model quality itself.

4. Modular software architecture and extensibility

OmniEvent’s software architecture is explicitly modular. The workflow described in the toolkit paper is: Data $\rightarrow$ Preprocessing Module $\rightarrow$ Unified JSON; Backbone Module; Aggregation Module; Classification / Generation Module; Evaluation Module (Peng et al., 2023). Under the hood, the core module types are backbone encoders, aggregation operations, and heads. Backbone options include CNN, BiLSTM, BERT, GPT-2, the T5 family, and FLAN-UL2. Aggregation operations pool or attend over contextual vectors to construct span representations or pairwise event representations. Heads apply either a classifier, such as softmax or CRF, or a generator, such as an encoder-decoder model (Peng et al., 2023).

The framework is configured through YAML. The description gives a BertCRF example with backbone: bert-base-cased, aggregator: span-gather, and head: crf, and notes that a BERT+CRF ED model can be turned into a GPT-2+CRF model by swapping "gpt2" for "bert" (Peng et al., 2023). Training and inference are exposed through a Trainer API and a single-line inference API, while command-line execution is supported through omnievent --task ... --model ... --text ... (Peng et al., 2023).

Ease of use is defined not only by API simplicity but also by deployment. The toolkit provides off-the-shelf models that can be directly deployed as web services, integrates HuggingFace Transformers and DeepSpeed for large-model fine-tuning, and exposes an online demonstration at https://omnievent.xlore.cn where a user selects language, task, and ontology and receives highlighted triggers, arguments, and an interactive event graph (Peng et al., 2023).

Extensibility is also explicit in the evaluation framework description. Adding a new corpus involves writing a new preprocessor under omnievent/preprocess, registering it in __init__.py, creating a YAML configuration, and then training and evaluating against the generated JSONL files (Peng et al., 2023). This design suggests a separation between corpus normalization, model prediction, output standardization, and metric computation that is intended to make new datasets and new predictors compatible without modifying the evaluation logic.

5. Reported results, recalibration effects, and scaling behavior

The toolkit paper reports representative results on six EE datasets for four paradigms—classification, sequence labeling, span prediction, and conditional generation. On ACE2005 ED, the reported F$_1$ values are 68.6 for classification, 68.6 for sequence labeling, 71.0 for span prediction, and 66.0 for conditional generation. On RichERE ED, the reported F$C=\{c_1,\ldots,c_n\}$0 values are 51.4, 50.1, 50.4, and 51.4, respectively. On ACE2005 EAE, the reported F$C=\{c_1,\ldots,c_n\}$1 values are 58.7 for classification, 49.4 for sequence labeling, 40.1 for span prediction, and 45.7 for conditional generation (Peng et al., 2023).

For ERE, the reported RoBERTa-based pairwise classifier yields the following metrics: event coreference on ACE2005 reaches precision 94.5, recall 81.7, and F$C=\{c_1,\ldots,c_n\}$2 87.7; temporal relation extraction on TB-Dense reaches 67.9, 54.0, and 60.2; causal relation extraction on CausalTB reaches 100.0, 50.0, and 66.7; and subevent extraction on HiEve reaches 21.4, 13.4, and 16.5 (Peng et al., 2023).

The evaluation paper adds a second type of result: recalibration under a consistent regime. On ACE-Full, “Original evaluation” versus “OmniEvent-consistent” evaluation produces absolute shifts of up to 8 F$C=\{c_1,\ldots,c_n\}$3 points in event detection and up to 12 points in argument extraction, and the relative ranking of models can change (Peng et al., 2023). The concrete ACE-Full comparison includes, for example, DMBERT ED moving from 73.2 to 74.5 F$C=\{c_1,\ldots,c_n\}$4, CLEVE ED from 80.6 to 78.3, BERT+CRF EAE from 43.0 to 48.6, EEQA EAE from 51.4 to 36.4, and Text2Event EAE from 54.5 to 50.0 (Peng et al., 2023). This is the empirical basis for the claim that “fairness” in OmniEvent is methodological rather than rhetorical.

The toolkit paper also reports scaling-law experiments in which FLAN-T5 and FLAN-UL2 with 20B parameters are fine-tuned. The summary states that larger models uniformly boost ED, EAE, and ERE, and that UL2-20B achieves a +3–5 point F$C=\{c_1,\ldots,c_n\}$5 gain over T5-XXL on ACE 2005 (Peng et al., 2023). The paper gives a relative improvement formula,

$C=\{c_1,\ldots,c_n\}$6

which frames large-model gains as a measurable extension of the same evaluation pipeline rather than as a separate experimental regime (Peng et al., 2023).

6. Demonstrations, applications, and homonymous usage in event-based vision

OmniEvent is publicly released together with a demonstration website and video, and its installation procedure is described through a standard git clone, Conda environment creation, requirement installation, and dataset preprocessing workflow (Peng et al., 2023). The toolkit supports raw-text inference for ED, full EE, and combined EE + ERE, and the online demo allows selection of English or Chinese, task type, and ontology such as ACE or MAVEN (Peng et al., 2023). The applications named in the toolkit description include hydro-meteorological event extraction for flood-prediction dashboards, adverse drug event mining from clinical narratives, structured event chains for financial-market forecasting, and intelligent legal-case analysis in Chinese via the LEVEN dataset (Peng et al., 2023).

Separately, a later paper reused the name “OmniEvent” for a task-agnostic event representation learner in event-based vision rather than NLP (Yan et al., 3 Aug 2025). In that work, event data are asynchronous tuples $C=\{c_1,\ldots,c_n\}$7 from event cameras, and the proposed architecture follows a decouple-enhance-fuse paradigm composed of Event Fusion and Sampling, Spatio-Temporal Decoupling, Space-Filling-Curve-based Aggregation, Spatio-Temporal Separated Attention, and Feature Tensorization (Yan et al., 3 Aug 2025). Its output is a grid-shaped tensor intended for standard vision backbones without architecture change.

The event-camera OmniEvent is evaluated on three tasks and ten datasets with the same set of hyper-parameters and no task-specific architectural changes. The paper reports top-1 accuracy of 85.2 on CIFAR10-DVS, 90.2 on N-Caltech101, 97.9 on N-CARS, 99.9 on N-MNIST, and 62.1 on N-ImageNet; optical-flow results on MVSEC of 0.70, 1.12, and 1.15 AEE on indoor_flying1, 2, and 3; and registration results of 1.146 rotation error and 0.327 translation error on MVSEC-E2P, with a reported 68.2% reduction in rotation error over E2PNet (Yan et al., 3 Aug 2025).

Because these two systems share a name but belong to different problem domains, “OmniEvent” is not a uniquely identifying term in the current literature. In NLP, it denotes a toolkit centered on reproducible event understanding and fair comparison across ED, EAE, and ERE. In event-based vision, it denotes a unified event representation learning framework for event cameras. The shared name does not imply a shared methodology, but it does reflect a common emphasis on unification across previously fragmented modeling settings.