Video-and-Language Event Prediction

Updated 19 December 2025

VLEP is a multifaceted computational paradigm that integrates visual inputs and language context to predict, localize, and generate video events.
It employs methods like symbolic structure induction, temporal modeling, and multimodal architectures to enhance causal reasoning and boundary detection.
Benchmark datasets and metrics such as MacroAcc and mIoU validate its effectiveness in applications ranging from future event prediction to video generation.

Video-and-Language Event Prediction (VLEP) refers to a suite of computational tasks that require predicting, localizing, or reasoning about events in videos using both visual and linguistic inputs. VLEP subsumes classic future event prediction, event relation modeling, causal reasoning, temporal grounding, and even video generation as the output modality. Unlike general video understanding, VLEP emphasizes detecting or forecasting event boundaries, roles, and their interrelations, guided by both video content and natural language queries or context. Modern VLEP research unites advances in temporal modeling, symbolic structure induction, and multimodal LLMs. This article systematically presents the evolution, methodologies, benchmarking strategies, and current architectural paradigms in VLEP.

1. Task Formulations and Benchmark Datasets

The canonical VLEP formulation asks: given a video segment $V$ and aligned text context (dialogue $D$ , event description, or instruction), select or generate the most plausible future event $e$ from a set of options or in free-form. Typical instantiations include:

Video-and-Language Future Event Prediction (VLEP): Binary or multi-choice selection among likely next events, leveraging video $V$ , dialogue $D$ , and two candidates $C = \{e_1, e_2\}$ , predicting $\hat e = \arg\max_{e\in C} P(e|V, D)$ (Lei et al., 2020).
Script Event Induction: Given a chain of structured events (triggers, arguments, relations), induce the next event or detect logical relations (causal, temporal, etc.) (Liang et al., 3 Jun 2025).
Temporal Grounding and Dense Captioning: Detect fine-grained event boundaries and produce timestamped captions for each event segment (Guo et al., 8 Oct 2024, Cheng et al., 2 May 2025).
Video Event Generation: Generate a plausible video sequence that visualizes the predicted next event, given video context and instruction (Cheng et al., 20 Nov 2025, Attarian et al., 2022).

Prominent datasets include VLEP (28K video-dialogue-future event pairs), VidEvent (23K events, 17.5K relations, narrative script induction), VER (500K videos, dense event annotations for segmented causal reasoning), E.T. Bench (7K videos, 12 tasks spanning referring, grounding, captioning, and QA), Event-Bench (2K long videos, 6 event reasoning tasks), Charades-STA, YouCookII, QVHighlights, and AVEP (action-centric event chains for open prediction). These datasets span diverse domains (sitcoms, vlogs, movies, instructional tasks) and annotation formats (textual, structured, timestamped).

2. Symbolic and Structural Modeling Approaches

Initial VLEP systems focus on extracting and manipulating symbolic event structures—in effect, converting videos and accompanying text into scene-graph representations encoding event types, argument roles, and entities. Notable mechanisms include:

Structural Symbolic Representation (SSR): Each event $x = \{v, a_1, e_1, \ldots, a_m, e_m\}$ , capturing verb, role names (Arg0, AScn, etc.), and surface-form entities. SSR inputs are linearized as token sequences and embedded for reasoning (Lu et al., 2023).
Event-Sequence Models: Instead of modeling isolated candidate events, context is expanded to sequences of prior/future events, allowing transformers to capture co-occurrence patterns and temporal cues. Feeding all five consecutive events enables pooled representation and accurate relation classification (Lu et al., 2023).
External Knowledge Injection: Visual commonsense (e.g., VisualCOMET) is reformulated to SSR format via semantic parsing (AMR), providing pretraining for event-relation prediction (Lu et al., 2023).
Node-Graph Hierarchical Transformers: Action-centric models encode events and argument nodes as multimodal embeddings within graph structures and employ hierarchical attention (node-level, graph-level, coreference encoding) for next-event prediction (Su et al., 19 Oct 2025).

Empirical analysis reveals that SSR-only models, when properly tuned, outperform multimodal video baselines; contextual sequence modeling and external knowledge further improve accuracy on benchmarks such as VidSitu (e.g., SSR+VisualCOMET pretraining achieves 59.2% MacroAcc, a 25-point gain over prior SOTA) (Lu et al., 2023).

3. Multimodal and Vision-Language Architectures

State-of-the-art VLEP systems integrate spatiotemporal video features with text encoders (transformers or LLMs), often employing complex fusion modules:

Vision Foundation Model (VFM): Deep ViT-based models extract global and object-centric features, merged via cross-attention (Q-Former style) into a compact language-aligned token set, subsequently fed into instruction-tuned LLMs for causal reasoning and future event prediction (Dubois et al., 8 Jul 2025).
Temporal Expert + Spatial Expert: Dual-branch frameworks (e.g., VideoExpert) split temporal modeling (high frame-rate compressed tokens, direct timestamp prediction) and fine-grained content analysis (spatial tokens, textual generation), coordinated by special tokens indicating event boundaries. This modular separation isolates temporal grounding from content generation and counteracts text-pattern bias in timestamp localization (Zhao et al., 10 Apr 2025).
Task-Interleaved LLMs: TRACE arranges sampled frames, timestamp tokens, salience scores, and captions in a unified token stream, instructs the backbone LLM to autoregressively decode each component per event, and achieves large zero-shot gains across VTG, highlight, and captioning tasks (Guo et al., 8 Oct 2024).
Frame Selection and Distillation: ViLA and SeViLA employ learnable or language-guided frame selection (Frame-Prompter, Localizer), text-guided student-teacher distillation (QFormer-Distiller), and self-refinement cycles to balance computational efficiency against event sensitivity (Wang et al., 2023, Yu et al., 2023).
Video-as-Answer Generation: The VANS model unifies a VLM (predicting captions) and a video diffusion model (generating video), co-optimized by Joint-GRPO, ensuring the textual reasoning is visualizable and the generated video is faithful to both instruction and context (Cheng et al., 20 Nov 2025).

The adoption of instruction-tuning, chain-of-thought prompting, and large-scale pretraining over diverse datasets strengthens generalization to unseen video event queries and open-ended temporal reasoning.

4. Evaluation Protocols and Metrics

The evaluation of VLEP systems utilizes a spectrum of metrics, including:

Multiple-choice accuracy: Predominant for identification tasks (“What is more likely to happen next?”), typically using binary (2-choice) or multi-choice on datasets like VLEP, Event-Bench, GVQ, RAR, RVQ (Lei et al., 2020, Du et al., 20 Jun 2024, Liu et al., 26 Sep 2024).
Macro-accuracy: Aggregated per relation type in event-relation prediction ( $\mathrm{MacroAcc} = (1/|C|)\sum_{c \in C} TP_c / (TP_c+FP_c+FN_c)$ ) (Lu et al., 2023).
Temporal localization F₁: Computed over IoU thresholds ( $\theta \in \{0.1, 0.3, 0.5, 0.7\}$ ) for event boundaries, action localization, event matching, and summarization (Liu et al., 26 Sep 2024).
Captioning metrics: CIDEr, METEOR, SODA_c, ROUGE-L, BERTScore for dense video captioning and open-ended generation (Guo et al., 8 Oct 2024, Zhao et al., 10 Apr 2025).
Video similarity and generation metrics: FVD, CLIP-V, CLIP-T for quality assessment in video-as-answer generation (Cheng et al., 20 Nov 2025, Attarian et al., 2022).
Human evaluation: Expert A/B comparison for semantic correctness, grounding, and insightfulness (Dubois et al., 8 Jul 2025).
Ablation studies: Frame selection, Q-Former distillation, knowledge injection, and module composition are systematically evaluated via component-wise accuracy deltas.

Benchmarking consistently shows substantial performance gaps between model and human performance, especially for tasks requiring multi-step reasoning, causal inference, and event localization in long or multi-event videos.

5. Insights, Limitations, and Recent Advances

Key empirical findings include:

Symbolic context and knowledge boost reasoning: Properly trained SSR-only models outperform video-only baselines in event-relation prediction, contradicting prior assumptions about the necessity of continuous video features for reasoning (Lu et al., 2023).
Video features often introduce noise: In complex scenes with simultaneous events (foreground vs. background), raw video features can degrade SSR performance; oracle SSR derived from human annotation provides much higher reliability (Lu et al., 2023).
Temporal modeling is indispensable: Feeding extended event sequences allows transformers to capture causality and co-occurrence; chain-of-thought prompting and masked infilling strategies (as in TEMPURA) yield measurable accuracy improvements (Cheng et al., 2 May 2025).
Frame selection and compression impact efficiency: Models employing adaptive frame selection (ViLA, SeViLA) achieve SOTA accuracy using only a subset of frames, reducing computation and highlighting critical moments (Wang et al., 2023, Yu et al., 2023).
World knowledge fusion is essential: LLM-driven reasoning enables zero-shot prediction and deeper causal chaining, especially when fused with vision models via compact Q-Former modules (Dubois et al., 8 Jul 2025).
Emergent failure modes: Current architectures are “deaf” to fine-grained event discrepancies, temporal swaps, and subtle attribute manipulations unless specifically trained with hard negative samples (cf. SPOT Prober) (Zhang et al., 2023). Video-LLaVA style models excel only at object recognition, not event induction.

6. Future Directions and Open Problems

Rapid progress in VLEP motivates several outstanding research challenges:

Robust long-range event induction: Modeling event evolution and long causal chains, possibly using hierarchical or recurrent architectures compatible with hour-long videos (Liang et al., 3 Jun 2025, Su et al., 19 Oct 2025).
Temporal grounding with low bias: Preventing shortcut bias and text-pattern leakage in timestamp prediction (e.g., VideoExpert's dual-expert separation) and developing embedding-based outputs for numeric prediction (Zhao et al., 10 Apr 2025, Liu et al., 26 Sep 2024).
Few-shot and zero-shot event reasoning: Exploiting in-context learning pipelines, mixed-modality prompts, and symbolic structures for prompt-efficient generalization (VidIL achieves 72% accuracy in 10-shot VLEP; humans reach 90.5%) (Wang et al., 2022).
Open-ended video generation as answer modality: Extending next-event prediction from text to video through RL-optimization of VLM and VDM pairs (VANS), with a focus on semantic alignment and fidelity (Cheng et al., 20 Nov 2025, Attarian et al., 2022).
Data curation and annotation: Enriching benchmarks with multi-event, time-sensitive, and hierarchical annotation; promoting hard negative injection and structured QA for fine-grained reasoning (Zhang et al., 2023, Liu et al., 26 Sep 2024).
Integration of audio and multi-modal evidence: Multimodal grounding, incorporating audio cues for improved narrative comprehension in sports, instructional, and conversational domains (Guo et al., 8 Oct 2024, Zhao et al., 10 Apr 2025).

Recent models such as TEMPURA, TRACE, and VideoExpert demonstrate state-of-the-art advances by combining structured, causal reasoning objectives, precise temporal supervision, and modular architecture designs (Cheng et al., 2 May 2025, Guo et al., 8 Oct 2024, Zhao et al., 10 Apr 2025). However, full cognitive event prediction—replicating human understanding in unconstrained videos—remains an unsolved problem.

7. Summary Table: Representative VLEP Models, Datasets, and Evaluation Results

Model/Paper	Dataset	Task Type	Metric/Result
SSR Event Sequence (Lu et al., 2023)	VidSitu	Event Relation Prediction	MacroAcc 59.2% (SSR+COMET, SOTA)
TEMPURA (Cheng et al., 2 May 2025)	VER, Charades-STA, QVHighlights	Masked Event Reasoning, Segmentation	mIoU 39.2, HIT@1 51.7 (+6–11 pp SOTA)
TRACE (Guo et al., 8 Oct 2024)	YouCookII, Charades-STA, QVHighlights	Event sequence generation	CIDEr 8.1, R@[email protected] 40.3% (0-shot)
VideoExpert (Zhao et al., 10 Apr 2025)	Charades-STA, QVHighlights	Temporal Grounding, Captioning	mIoU 41.1, R@[email protected] 40.3%, SOTA dense captioning
SeViLA (Yu et al., 2023)	VLEP	Future Event Prediction	69.0% accuracy (fine-tuned, SOTA)
ViLA (Wang et al., 2023)	VLEP	Future Event Prediction	69.6% accuracy (4 frames, 1.45× faster)
VidIL (Wang et al., 2022)	VLEP	Few-shot Event Prediction	72.0% accuracy (10-shot, no pretrain)
VANS (Cheng et al., 20 Nov 2025)	VANS-Data-100K	Video-as-Answer Generation	BLEU@1 ↑, FVD ↓, Human overall 4.8/5
E.T. Chat (Liu et al., 26 Sep 2024)	E.T. Bench	Event-level multi-task	TVG F₁=38.6%, DVC F₁=38.4% (SOTA OS)

This selection reflects the methodological diversity and benchmarking progress across symbolic, multimodal, causal modeling, dense captioning, and generative paradigms in VLEP.

References: All factual claims and metrics are drawn directly from (Lu et al., 2023, Cheng et al., 2 May 2025, Guo et al., 8 Oct 2024, Dubois et al., 8 Jul 2025, Liang et al., 3 Jun 2025, Du et al., 20 Jun 2024, Wang et al., 2023, Cheng et al., 20 Nov 2025, Wang et al., 2022, Su et al., 19 Oct 2025, Barbu et al., 2012, Lei et al., 2020, Attarian et al., 2022, Zhang et al., 2023, Zhao et al., 10 Apr 2025, Liu et al., 26 Sep 2024, Yu et al., 2023).