EventGPT: LLMs for Event Understanding

• EventGPT is a family of LLM frameworks for structured event understanding across modalities including vision, NLP, and sports analytics.
• It integrates specialized encoders, spatio-temporal aggregators, and language adapters to convert multimodal event data into actionable insights.
• The frameworks achieve state-of-the-art performance in event extraction and simulation, with applications in robotics, surveillance, and sports analytics.

EventGPT is a family of LLM frameworks and architectures developed for structured event understanding across diverse domains, including event-based vision, natural language event extraction, and sequential decision modeling in spatiotemporal domains such as sports analytics. These systems unify multimodal or contextual event information with language modeling and reasoning, extending the capabilities of traditional LLMs to handle event streams or extract event schemas in new modalities.

1. Architectures and Core Methodologies

EventGPT denotes several model architectures, each tailored to event-centric data:

(a) Multimodal Event Stream Understanding:

EventGPT for event-based vision integrates asynchronous event streams—represented as pixel-level $(x, y, t, p)$ tuples from event cameras—into LLMs via a hierarchical architecture, comprising:

• Event Encoder: Pretrained OpenCLIP ViT-L/14-336px, applied to quantized event “bins.”
• Spatio-temporal Aggregator: Pools across temporal and spatial bins, concatenating max/average pooled features to produce a compact embedding.
• Linear Projector & Event–Language Adapter: Two-stage mapping aligns vision and event features to the LLM’s language space.
• LLM: Vicuna-v1.5 (7B/13B), with frozen weights during early optimization, fully fine-tuned in the final stage (Liu et al., 1 Dec 2024).

(b) NLP Event Extraction:

Event extraction is recast as a sequence-to-structure or sequence-to-sequence problem:

• Discrete Prompt-Based (ChatGPT): Zero-shot prompting with explicit event schemas, in-context demonstrations, and output format constraints (JSON) (Gao et al., 2023).
• Generative Template-based (GTEE-DynPref): Input templates encode type instructions and argument slots, modular “prefix tuning” injects event-type and context information at each attention layer of T5/BART. Contextual prefix vectors are computed on-the-fly from a BERT-encoded context using multihead attention over static, type-specific prefix embeddings (Liu et al., 2022).

(c) Sequential Action Events (Sports):

EventGPT (also “ScoutGPT” in (Hong et al., 19 Dec 2025)) models football match play as a tokenized event sequence:

• Token Embedding: All attributes (player, action type, coordinates, timing, success state, action value) are discrete tokens, embedded and positionally encoded.
• Autoregressive Transformer: Standard decoder-only GPT architecture predicts next event attributes and estimated “residual On-Ball Value (rOBV)” given prior sequence and fixed player identity.
• Counterfactual Simulation: Player identity is injected at context block level, enabling substitution and forward simulation to quantify transfer fit.

2. Optimization Paradigms and Training Procedures

Multistage Curriculum (Vision-EventGPT, (Liu et al., 1 Dec 2024)):

1. Image–Language Warmup: Pretraining linear projector with 558K GPT-generated RGB-image/text pairs, leveraging LLaVA data to align natural images with LLM input space.
2. Synthetic Event–Language Alignment: Training event–language adapter on 1M synthetic event frames and paired captions from N-ImageNet-Chat.
3. Real Event Fine-Tuning: Full-model fine-tuning on ~120K instruction–response event streams (Event-Chat), mainly collected from DSEC/E2VID under challenging visual conditions.

All stages optimize next-token cross-entropy objectives: $$\mathcal L = -\sum_{i}^{L} \log P(a_i | a_{<i}, E, Q)$$

Prefix-Tuning for Event Extraction (Liu et al., 2022):

• Prefix vectors are either static (tied per event type) or dynamic (context-conditioned via multihead attention). Training proceeds in three stages: LM base pretraining, static prefix learning, dynamic prefix optimization, each regularized with generative log-likelihood.

Autoregressive Prediction for Sequential Events (Hong et al., 19 Dec 2025):

• All event attributes are predicted tokenwise with a cross-entropy loss: $$\mathcal{L}_{\mathrm{CE}} = -\sum_{t=1}^{T} \log P(v_{t+1}|v_{1:t})$$
• Optionally, rOBV can be regressed via MSE if not discretized.

3. Data Representations and Datasets

Event-Based Vision:

• N-ImageNet-Chat: 1M synthetic event frames with captions for pretraining.
• Event-Chat: 59K real event streams × instructions collected from DSEC/E2VID under high dynamic conditions.
• N-ImageNet-Instruction: 69K synthetic captioning/VQA/reasoning samples.

NLP Event Extraction:

• ACE 2005: 33 event types, gold annotations for event triggers.
• ERE-EN: 38 event types, 21 roles, enabling portability and few-shot adaptation benchmarks.

Sports Analytics:

• Premier League Event Data: 1900 matches, 173,951 episodes, 1221 unique players. Each episode encoded using SPADL events into up to 122 tokens per sample.

4. Evaluation and Empirical Results

Vision EventGPT (Liu et al., 1 Dec 2024):

Model	N-ImageNet-Chat (DC/CR/VQA)	Event-Chat (DC/CR/VQA)
LLaVA-7B	1.54/1.07/1.88	2.20/4.04/3.26
Qwen2-VL-7B	1.74/1.46/1.91	2.38/4.02/2.91
Intern2VL-8B	1.51/1.87/2.08	2.37/4.00/3.71
EventGPT-7B	2.39/2.57/2.23	3.52/4.09/4.29
EventGPT-13B	2.41/2.81/2.40	3.40/4.13/4.26

(EventGPT consistently outperforms prior MLLMs on zero-shot event description, reasoning, and VQA.)

NLP Event Extraction (Gao et al., 2023, Liu et al., 2022):

• ChatGPT Event Detection: $F_1 = 64.0$ (high-frequency) vs. $F_1 = 30.51$ (low-frequency), aggregate $F_1$ is ~51% of EEQA.
• GTEE-DynPref: ACE05-E $F_1$ = 72.6 (trigger), 55.8 (argument); new SOTA on ERE (trigger $F_1$ = 66.9).

Ablation (ChatGPT): Removing event definitions or positive demonstration degrades performance sharply; negative examples may confuse the model.

Sequential Event Generation (Hong et al., 19 Dec 2025):

Model	ht Acc	et Acc	MAE$_{x,y}$	MAE$_{\Delta t}$	o Acc	rOBV MAE
LEM	85.20%	74.07%	9.01	8.11	90.51%	0.014
LEM Transformer	96.05%	80.42%	7.15	7.08	86.92%	0.008
EventGPT	94.12%	82.91%	4.30	4.31	92.87%	0.009

(EventGPT achieves the highest action-type accuracy and substantially improved spatial/temporal precision.)

5. Analysis, Ablations, and Challenges

• Domain Bridging (Vision): The stepwise optimizer narrows the domain gap from RGB to event space, with the event–language adapter and spatio-temporal aggregator essential for optimal grounding (Liu et al., 1 Dec 2024).
• Prompt Sensitivity (ChatGPT): Prompt construction and demonstration are critical, with substantial brittleness and lack of stability across evaluations (Gao et al., 2023).
• Dynamic Prefixing (NLP): Blending static type and context encodings via dynamic prefix attention yields state-of-the-art transfer and argument extraction, facilitating adaptation to new schemas (Liu et al., 2022).
• Counterfactual Reasoning (Sports): Substituting player identity in the context block enables robust simulation of player fit under different tactical or team environments, with results strongly context- and system-dependent (Hong et al., 19 Dec 2025).

6. Applications and Impact

• Autonomous Driving and Robotics: Real-time event understanding enables high-speed detection, action anticipation, and captioning in visually challenging conditions (Liu et al., 1 Dec 2024).
• Surveillance and Scientific Imaging: EventGPT’s robust reasoning on high-frame-rate or low-light event streams supports surveillance tasks and analysis of scientific experiments.
• NLP Event Structure Extraction: Event-aware LLMs facilitate open-domain information extraction, automatic event schema induction, and interactive systems with structured event output (Gao et al., 2023, Liu et al., 2022).
• Sports Analytics and Scouting: Principled transfer analysis, action-value simulation, and embedding-driven retrieval afford novel metrics for assessing player transfer fit and tactical compatibility (Hong et al., 19 Dec 2025).

7. Limitations and Future Directions

• Event Stream Vision: Reliance on frame-based binning and pooling may underutilize the temporal dynamics; future models may adopt spiking encoders or learned attention mechanisms for streaming data. Current instruction datasets are orders of magnitude smaller than best-in-class LLM pretraining corpora (Liu et al., 1 Dec 2024).
• Event Extraction with LLMs: Prompt brittleness, lack of stability, and performance deficits on long-tail or complex events limit zero-shot applicability without augmentation or hybridization (Gao et al., 2023). Advanced continuous prompt methods such as dynamic prefixing offer better transfer and schema scalability (Liu et al., 2022).
• Interpretability and Modeling Capacity (Sequential): Embedding-based player representation is limited to context inputs; richer disentanglement of skill and tactical effect may require hierarchical or role-specific embedding strategies (Hong et al., 19 Dec 2025).

Collectively, EventGPT systems constitute a paradigm shift toward LLM-powered, event-centric understanding in multimodal, text, and structured sequential domains. They demonstrate improvements across challenging spatiotemporal inference tasks, while exposing new methodological frontiers for domain adaptation, structural extraction, and context-conditional simulation in large generative models.