ARIS: Agreement-based Reflective Event Extraction

• The paper integrates self-mixture agents, a discriminative sequence tagger, and an LLM reflective module to reconcile model disagreements in event extraction.
• ARIS employs consensus scoring and confidence-based filtering to enhance precision and recall in extracting triggers, event types, and arguments.
• The hybrid approach achieves state-of-the-art performance across varied domains, demonstrating robust extraction capabilities despite increased computational cost.

Searching arXiv for the cited paper and closely related identifiers. Agreement-based Reflective Inference System (ARIS) is a hybrid event extraction architecture that combines a Self Mixture of Agents with a discriminative sequence tagger, confidence-based filtering, and an LLM reflective inference module in order to improve robustness in extracting triggers, event types, and arguments from unstructured text (Haji et al., 26 Aug 2025). It is introduced in “Reflective Agreement: Combining Self-Mixture of Agents with a Sequence Tagger for Robust Event Extraction” and is motivated by a specific tradeoff in event extraction: traditional discriminative models demonstrate high precision but often exhibit limited recall, whereas generative approaches leveraging LLMs provide higher semantic flexibility and recall but suffer from hallucinations and inconsistent predictions (Haji et al., 26 Aug 2025).

1. Conceptual position and problem setting

ARIS is formulated for Event Extraction (EE), defined as automatically identifying and extracting structured information about events from unstructured text, including triggers, event types, and arguments (Haji et al., 26 Aug 2025). Its central premise is that discriminative tagging and generative inference expose complementary error profiles. The system therefore does not replace one paradigm with the other; instead, it organizes their interaction through explicit agreement detection, threshold-based decision rules, and a final reflective resolution stage (Haji et al., 26 Aug 2025).

The architecture integrates four main components: Self Mixture of Agents (Self-MoA), Discriminative Sequence Tagger, Confidence-Based Filtering, and LLM Reflective Inference Module (Haji et al., 26 Aug 2025). For each input document $x$, Self-MoA agents and the sequence tagger produce candidate extractions; span-level agreement is detected; low-confidence disagreements are filtered out; remaining ambiguous cases are resolved by an LLM “reflection” step; and the final event set $E_{\rm fin}(x)$ merges high-confidence agreements and reflected outputs (Haji et al., 26 Aug 2025).

A plausible implication is that ARIS should be understood less as a single model than as an inference system for reconciling heterogeneous predictors. This interpretation follows from the fact that its final decision boundary is distributed across consensus scoring, thresholding, and reflective adjudication rather than residing in one predictor alone.

2. Architectural components and data flow

The Self-MoA component is defined over $n$ parallel instances of the same base LLM, denoted $A=\{A_1,\dots,A_n\}$, with each agent sampled using temperature $T_i$ (Haji et al., 26 Aug 2025). For a document $x$, agent $A_i$ produces a set of event predictions

$E^{(i)}(x)\subseteq \mathcal{G}_x,$

where $\mathcal{G}_x$ is the universe of valid trigger–argument structures in $x$ (Haji et al., 26 Aug 2025). The raw Self-MoA output is

$E_{\rm fin}(x)$0

To align predictions with the source text, ARIS applies span validation, retaining only predictions whose textual spans occur in $E_{\rm fin}(x)$1, and positional sorting, ordering predictions by character offsets (Haji et al., 26 Aug 2025). The resulting cleaned set is denoted $E_{\rm fin}(x)$2 (Haji et al., 26 Aug 2025).

The discriminative component is a token-level sequence tagging model $E_{\rm fin}(x)$3, specifically a RoBERTa-based TagPrime model (Haji et al., 26 Aug 2025). Its input representation concatenates tokens with special markers indicating triggers and arguments, and its output schema extends BIO tagging to encode both trigger labels, $E_{\rm fin}(x)$4 and $E_{\rm fin}(x)$5, and argument labels, $E_{\rm fin}(x)$6 and $E_{\rm fin}(x)$7 (Haji et al., 26 Aug 2025). Each token receives one tag, and training uses a standard cross-entropy loss (Haji et al., 26 Aug 2025).

The reflective component is invoked only after confidence-based filtering has narrowed the disagreement space. Ambiguous predictions $E_{\rm fin}(x)$8 are passed to a specialized LLM $E_{\rm fin}(x)$9, fine-tuned with decomposed instructions (Haji et al., 26 Aug 2025). For each ambiguous candidate, the prompt contains a role specification, task description, generation rules requiring strict JSON output with no extra text, full passage context, and the ambiguous candidates (Haji et al., 26 Aug 2025). The LLM returns a binary decision or corrected span/type, aggregated as $n$0 (Haji et al., 26 Aug 2025).

This modularization matters because ARIS does not treat model disagreement as undifferentiated error. Instead, it stratifies disagreement into cases that can be retained, discarded, or reflected upon, thereby turning conflict between predictors into an explicit computational object.

3. Formalization of agreement, confidence, and final inference

ARIS defines a consensus score for each event candidate $n$1 as

$n$2

By definition, $n$3 if every agent produced $n$4, and $n$5 if none did (Haji et al., 26 Aug 2025). This score also serves as the Self-MoA confidence:

$n$6

Let the tagger’s prediction set be $n$7. ARIS forms

$n$8

where $n$9 is the consensus set, and the disagreement set is

$A=\{A_1,\dots,A_n\}$0

For the tagger, confidence is defined as

$A=\{A_1,\dots,A_n\}$1

Two thresholds, $A=\{A_1,\dots,A_n\}$2 and $A=\{A_1,\dots,A_n\}$3, are chosen by grid search on validation data (Haji et al., 26 Aug 2025). Disagreements are then partitioned into a high-confidence retain set,

$A=\{A_1,\dots,A_n\}$4

a low-confidence discard set,

$A=\{A_1,\dots,A_n\}$5

and an ambiguous set,

$A=\{A_1,\dots,A_n\}$6

(Haji et al., 26 Aug 2025).

The final event set is

$A=\{A_1,\dots,A_n\}$7

Algorithmically, the paper summarizes the trigger inference pipeline as: obtain $A=\{A_1,\dots,A_n\}$8 from each agent; clean their union into $A=\{A_1,\dots,A_n\}$9; compute $T_i$0; form consensus and disagreement sets; score each disagreement using $T_i$1 and the maximum tag probability; retain high-confidence cases, discard low-confidence cases, reflect on ambiguous cases, and merge consensus, retained, and reflected outputs into $T_i$2 (Haji et al., 26 Aug 2025). Argument extraction follows an analogous procedure per trigger (Haji et al., 26 Aug 2025).

The report also presents a mixture-model view:

$T_i$3

In practice, $T_i$4 is approximated by the indicator $T_i$5 and normalized via $T_i$6 (Haji et al., 26 Aug 2025). This suggests that ARIS can be interpreted as a constrained Bayesian reconciliation procedure whose practical implementation is deliberately simplified into count-based agreement.

4. Decomposed instruction fine-tuning and reflective reasoning

ARIS further investigates decomposed instruction fine-tuning for enhanced LLM event extraction understanding (Haji et al., 26 Aug 2025). The decomposed instruction dataset $T_i$7 is constructed by transforming a base event extraction corpus $T_i$8 into 13 subtasks (Haji et al., 26 Aug 2025). These subtasks include holistic generation with full structure and role ablation; trigger-focused tasks covering detection, classification, discrimination, and joint prediction; and argument-focused tasks covering extraction single/multi, role assignment single/multi, and joint prediction (Haji et al., 26 Aug 2025).

Each example follows a canonical prompt schema consisting of six fields: Role, Task description, Generation rules, Output format, Example, and Query (Haji et al., 26 Aug 2025). For trigger discrimination, negative sampling selects hard negative n-grams within a 3-token window of real triggers while ensuring no substring overlap (Haji et al., 26 Aug 2025). The training corpus comprises approximately $T_i$9 instruction examples across CASIE, M2E2, and MLEE, and is used to train the LLM to perform each subtask before full event construction (Haji et al., 26 Aug 2025).

The joint loss for instruction fine-tuning is given as

$x$0

where each example comes from one of the thirteen subtasks (Haji et al., 26 Aug 2025).

The paper’s interpretation of this stage is explicit: training the reflection LLM on decomposed instructions equips it to reason systematically over event chains, rather than relying on ad hoc debate prompts (Haji et al., 26 Aug 2025). A plausible implication is that reflective inference in ARIS is not merely a post hoc verifier; it is a task-specialized adjudicator whose capabilities are shaped by the subtask decomposition used during instruction tuning.

5. Experimental evaluation and benchmark behavior

ARIS is evaluated on CASIE (cybersecurity), M2E2 (news), and MLEE (biomedical), following the TextEE splits (Haji et al., 26 Aug 2025). The exact-match metrics are Trigger Identification (Trg-I), Trigger Classification (Trg-C), Argument Identification (Arg-I), and Argument Classification (Arg-C) (Haji et al., 26 Aug 2025).

Dataset	Model	Trg-I	Trg-C	Arg-I	Arg-C
CASIE	TagPrime	72.0	71.6	47.5	45.6
CASIE	ARIS	70.8	70.3	48.8	46.8
M2E2	TagPrime	63.97	63.30	37.47	34.19
M2E2	ARIS	73.49	71.39	41.41	38.84
MLEE	TagPrime	74.61	72.38	48.30	46.74
MLEE	ARIS	74.78	72.45	59.19	56.98

The reported result is that ARIS outperforms all strong baselines, particularly on argument tasks (Haji et al., 26 Aug 2025). The dataset-level breakdown shows, however, that the improvement pattern is not uniform. On CASIE, TagPrime exceeds ARIS on both trigger metrics, while ARIS improves both argument metrics; on M2E2, ARIS improves all four metrics; and on MLEE, ARIS produces small gains on trigger metrics and large gains on argument metrics (Haji et al., 26 Aug 2025). This profile is consistent with the paper’s claim that complementary strengths are being combined, with sequence taggers supplying high precision and exact spans and LLM agents supplying semantic coverage and high recall (Haji et al., 26 Aug 2025).

Ablation results indicate that removing the sequence tagger (“w/o TagPrime”) lowers precision sharply, while removing reflection harms recall on ambiguous cases; the full system yields the best F1 across all four subtasks (Haji et al., 26 Aug 2025). In temperature robustness experiments, varying MoA temperature $x$1 affects standalone Self-MoA precision and recall dramatically, but ARIS’s final F1 remains within 2 points across all temperatures, which the paper attributes to the normalization effect of consensus and filtering (Haji et al., 26 Aug 2025).

One common misconception would be to read ARIS as a purely generative event extractor with a discriminative auxiliary module. The reported ablations do not support that interpretation: removing TagPrime lowers precision sharply, indicating that the sequence tagger is structurally central rather than merely supplemental (Haji et al., 26 Aug 2025).

6. Strengths, limitations, and prospective extensions

The paper identifies three principal strengths. First, ARIS exploits complementary strengths: sequence taggers supply high precision and exact spans, whereas LLM agents supply semantic coverage and high recall; consensus filtering picks the overlap, confidence thresholds discard hallucinations, and reflection resolves the remainder (Haji et al., 26 Aug 2025). Second, reflective inference is specialized through decomposed instruction tuning rather than ad hoc debate prompting (Haji et al., 26 Aug 2025). Third, robustness is achieved by explicitly structuring agreement and confidence so that the stochasticity of generative outputs is controlled and performance remains stable across sampling settings (Haji et al., 26 Aug 2025).

The reported limitations are equally explicit. Running $x$2 LLM agents in Self-MoA plus the reflection module adds inference latency and GPU cost (Haji et al., 26 Aug 2025). The reflection stage still fails on extremely ambiguous or rare constructions (Haji et al., 26 Aug 2025). Proposed extensions include adaptive agent weighting $x$3, dynamic threshold tuning, retrieval-augmented reflection, and more efficient agent ensembling to reduce cost and further improve accuracy (Haji et al., 26 Aug 2025).

These limitations clarify the scope of the method. ARIS does not claim to eliminate ambiguity in event extraction; rather, it reallocates ambiguity into a specialized resolution stage and accepts the computational cost of doing so. This suggests that the framework is particularly relevant where extraction quality justifies multi-stage inference overhead, especially for argument-heavy settings in which the reported gains are largest (Haji et al., 26 Aug 2025).

7. Significance within hybrid event extraction

ARIS is presented as a general framework for robust, hybrid NLP systems that systematically integrates discriminative tagging and generative reasoning through structured agreement, confidence filtering, and reflection (Haji et al., 26 Aug 2025). Within event extraction specifically, its empirical significance lies in demonstrating that a consensus-oriented reconciliation procedure can convert disagreement between model classes into a source of robustness rather than instability (Haji et al., 26 Aug 2025).

Its broader methodological significance follows from how it decomposes the inference problem. Agreement is quantified rather than assumed; confidence is operationalized separately for generative and discriminative outputs; and unresolved cases are escalated to a dedicated reflective model trained on decomposed subtasks (Haji et al., 26 Aug 2025). This suggests a reusable pattern for hybrid systems in which heterogeneous predictors are not simply ensembled, but are arranged into a hierarchy of consensus, filtering, and adjudication.

In summary, ARIS defines a reflective agreement paradigm for event extraction: Self-MoA provides diverse generative hypotheses, TagPrime provides token-level discriminative structure, confidence thresholds separate reliable from unreliable disagreement, and a decomposed-instruction LLM resolves the residual ambiguous cases (Haji et al., 26 Aug 2025). The resulting system is reported to set new state-of-the-art on multiple benchmarks while also making explicit the computational and ambiguity-related constraints that remain (Haji et al., 26 Aug 2025).