Video-CoE: Reinforcing Video Event Prediction via Chain of Events

Abstract: Despite advances in the application of MLLMs for various video tasks, video event prediction (VEP) remains relatively underexplored. VEP requires the model to perform fine-grained temporal modeling of videos and establish logical relationships between videos and future events, which current MLLMs still struggle with. In this work, we first present a comprehensive evaluation of current leading MLLMs on the VEP task, revealing the reasons behind their inaccurate predictions, including lack of logical reasoning ability for future events prediction and insufficient utilization of visual information. To address these challenges, we propose \textbf{C}hain \textbf{o}f \textbf{E}vents (\textbf{CoE}) paradigm, which constructs temporal event chains to implicitly enforce MLLM focusing on the visual content and the logical connections between videos and future events, incentivizing model's reasoning capability with multiple training protocols. Experimental results on public benchmarks demonstrate that our method outperforms both leading open-source and commercial MLLMs, establishing a new state-of-the-art on the VEP task. Codes and models will be released soon.

• The paper introduces a chain-of-events framework as atomic reasoning units to capture fine-grained temporal progressions in video event prediction.
• It leverages a reinforcement learning objective with a specialized reward structure that uses embedding-based similarity signals to align event predictions.
• Experimental results demonstrate a 41.67% improvement with significant gains in AVEP Verb and Action-F1 scores, validating robust prediction for overlapping or nested events.

Reinforcing Video Event Prediction via Chain of Events: An Analysis of Video-CoE

Motivation and Problem Definition

The proliferation of multimodal LLMs (MLLMs) has accentuated their application in video event prediction (VEP)—the task of modeling temporal progressions and forecasting imminent occurrences in video data. However, extant approaches, particularly those grounded in frame-level or local region perception, exhibit bottlenecks in capturing temporally fine-grained, event-level evolution necessary for robust reasoning in complex scenarios. Video-CoE addresses this deficit by advocating for an explicit chain-of-events (CoE) structure as atomic reasoning units, diverging from action-centric temporal representations. The paradigm is precisely tailored to MLLM-based VEP, facilitating end-to-end event-level reasoning and open-set generation, rather than conventional closed-set classification.

Chain-of-Events Paradigm and Model Design

Video-CoE introduces the CoE framework, which encapsulates the logical sequence of discrete events as the principal granularity for temporal reasoning. The model constructs training datasets that encode causal and inclusion relations between past and future events, advancing from thematic representation toward fine-grained temporal granularity. This decoupled design allows hierarchical event structures—including overlap and nesting—to be processed during reasoning rather than imposed at the representation stage, enhancing flexibility and expressiveness.

The reinforcement learning (RL) objective is instantiated via CoE-GRPO, distinct from methods like T-GRPO or GRPO-GR, through targeted rewards emphasizing global event evolution over local perceptual cues. The reward structure is augmented by embedding-based similarity signals to encourage temporal alignment, calibrated to avoid reward hacking and maintain attribute-level correctness in thematic benchmarks.

Experimental Results

Empirical validation on VEP benchmarks demonstrates pronounced performance gains for Video-CoE. Notably:

• The model achieves an AVEP Verb score of 18.75 and an Action-F1 of 9.88, outpacing prior MLLM-based baselines by significant margins (e.g., VideoChat-R1: Verb 8.31, Action-F1 3.95).
• FutureBench average is reported as 75.00, validating superior predictive performance relative to alternatives (Video-R1: 67.47).
• Ablation studies confirm no detrimental side effects from hyperparameter selection such as event chain length $L$.
• CoE-specific reward structure ensures adherence to the reasoning paradigm (WR: 93\%, IR: 15.11\%), substantiating the model's capacity to utilize visual information in reasoning.

These results highlight a 41.67% relative improvement over previous baselines, supporting the claim that Video-CoE substantially mitigates prior bottlenecks in event-level prediction.

Practical and Theoretical Implications

Video-CoE's event-centric, chain-based reasoning paradigm establishes a template for future VEP research, suggesting the necessity of explicit logical structuring to realize more accurate temporal predictions in multimodal contexts. Its flexibility in handling hierarchical temporal relations enables practical deployment in scenarios involving overlapping or nested events, such as surveillance, sports analytics, and behavioral prediction. The open-set generation setting provides foundational support for scalable, unconstrained prediction tasks, extending applicability beyond pre-defined action vocabularies.

From a theoretical perspective, the decoupling of event representation and hierarchical reasoning offers a blueprint for modular model designs. The reward calibration strategy demonstrates feasible avenues for leveraging embedding-based similarity metrics without incurring attribute-level misalignment or reward exploitation, which is critical for robust RL applications in VEP.

Future Directions

Several avenues for expansion emerge:

• Extending chain-of-events reasoning to cross-modal tasks involving text, audio, or sensor data.
• Integrating stricter attribute grounding in similarity rewards via MLLM-derived signals for more demanding benchmarks.
• Scaling the CoE framework to longer videos with increased event density, potentially involving advanced hierarchical reasoning modules.

Conclusion

Video-CoE introduces a principled event-centric structure for video event prediction, reinforced by targeted RL objectives and carefully engineered reward signals. The approach yields non-trivial performance improvements on established VEP benchmarks, demonstrating its effectiveness in capturing temporally fine-grained event evolution at the video level. The model's design and results motivate continued research into logically structured temporal reasoning for multimodal prediction tasks (2603.14935).