Seahorse: A Unified Benchmarking Framework for Spatiotemporal Event Modeling

Abstract: Spatiotemporal point processes (STPPs) model event data in continuous time and space, with applications in mobility, epidemiology, and public safety. Recent neural STPPs span expressive intensity models, conditional density models, continuous-time latent dynamics, normalizing-flow spatial decoders, and score-based generative mechanisms. Yet comparison remains fragile because implementations differ in preprocessing, coordinate normalization, splits, likelihood conventions, and evaluation protocols. We present SEAHORSE, a unified framework for reproducible STPP experimentation. SEAHORSE formalizes neural STPPs through a common encode-evolve-decode interface and trains, tunes, and evaluates every model family under a single executable benchmark protocol with raw-coordinate likelihood reporting. This enables fair comparisons but, more importantly, controlled diagnostic studies. We pair SEAHORSE with HawkesNest, a synthetic stress-test suite, and show that increasing event-pattern complexity exposes each family's inductive bias, degrading some models sharply and leaving others stable. Code: https://github.com/YahyaAalaila/seahorse.

• The paper introduces a unified framework that standardizes data handling, model interfaces, and evaluation protocols for spatiotemporal point processes.
• It employs an encode–evolve–decode architecture to enable fair comparisons across diverse model families and diagnostic setups.
• Experimental results reveal domain-dependent performance variations and highlight challenges in stability, calibration, and computational efficiency.

Seahorse: A Unified Framework for Benchmarking Spatiotemporal Point Processes

Motivation and Problem Statement

Neural modeling of spatiotemporal point processes (STPPs) is challenged by the lack of standardized comparison due to variation in data preprocessing, coordinate normalization, model architectures, likelihood conventions, and evaluation protocols. This heterogeneity undermines the reproducibility and interpretability of empirical results, making it impossible to conduct controlled diagnostic studies or fair performance comparisons. Existing benchmarking efforts focus primarily on temporal point processes, leaving the complex space-time regime underserved.

Framework Overview

Seahorse provides a unified, executable contract for benchmarking STPP models, encapsulating data handling, model construction, configuration resolution, training, evaluation, and artifact management in a standardized and reproducible software architecture. All models, whether classical or neural, intensity-, density-, or generative-based, are cast under a four-stage encode--evolve--decode interface, enabling systematic cross-family evaluation while preserving expressive flexibility.

Figure 1: The Seahorse system: fixed event datasets, model presets, and benchmark configs flow through a unified contract, producing comparable metrics and reproducible artifacts.

Key architectural elements include:

• Data Handling: Fixed train/validation/test splits in raw coordinates; model-specific normalizations and transforms are learned only from the training partition and stored as run metadata.
• Model Interface: All models expose history encoders, state evolution mechanisms, and decoders, unified under schema-validated configuration objects.
• Execution Contract: Training, tuning, benchmarking, and evaluation are executed in modes that fix the dataset, configuration, and reporting conventions, preventing leakage and confounding.
• Artifact Management: Each run records the objective, optimization path, reporting space, checkpoint, and metric exposure for full auditability.

  Figure 3: The Seahorse execution contract separates benchmark configuration and reporting from model-specific training, ensuring comparability and reproducibility across heterogenous model families.

Unified Model Abstraction

Seahorse’s abstraction separates the model into three orthogonal components:

• History Encoder: Maps variable-length event sequences into conditioning representations; can be RNNs, transformers, amortized encoders, or parametric statistics.
• State Evolution: Governs the dynamics of the latent state between events; can be static (piecewise constant), ODE-based, or jump-diffusive.
• Decoder: Implements the conditional law for the next event (in time and space) given the latent state; supports factorized, joint, or generative parameterizations.

This separation renders implementation and architectural choices explicit, allowing additive benchmarking of inductive biases and calibration of fitness under varying complexity regimes.

Families and Model Inventory

The benchmark suite covers a broad range of STPP parameterizations, including:

• Factorized Classical Baselines: Poisson, Hawkes, self-correcting with parametric or neural conditional spatial decoders.
• Neural Temporal Baselines: RMTPP, THP with GMM spatial heads.
• Neural STPP Likelihood Models: DeepSTPP, AutoSTPP (exact integration), NSMPP (spectral, joint intensity).
• Continuous-Time Neural Variants: NJSDE, Neural Jump-CNF, Neural Attn-CNF (ODE/CNF decoders).
• Generative Models: SMASH (score-matching), DSTPP (diffusion generative).

Extensibility is programmed into the Seahorse interface: researchers can introduce new STPP methods via either native implementation of the encode–evolve–decode contract or wrappers around existing PyTorch modules, with explicit declaration of evaluation capability.

Figure 2: CLI- and Python-exposed Seahorse software architecture: separates configuration, data loading, model construction, runner/evaluation, and artifact layers.

Benchmark Protocol and Experimental Results

Data Coverage

The empirical suite spans 13 real and synthetic datasets, with three (COVID, Earthquakes, Citibike) serving as primary real-data benchmarks. Synthetic data is provided by HawkesNest, a controlled generator enabling structural stress tests, e.g., synthetic spatiotemporal entanglement axes.

Metrics and Evaluation Semantics

Seahorse reports test NLL (corrected for coordinate transforms), CRPS, ground-truth intensity recovery metrics, and autoregressive rollout diagnostics. Models with numerical/variational/surrogate objectives have their evaluation paths recorded, distinguishing exact, approximate, and sample-based likelihoods.

Numerical Findings

• On real datasets, likelihood performance is highly domain-dependent and unstable across architectures: e.g., Attn-CNF is strongest on COVID and Citibike, DeepSTPP on Earthquakes. Classical Hawkes with neural spatial heads remains competitive in some regimes, while simple Poisson/self-correcting processes perform poorly.
• On synthetic entanglement suites, NSMPP yields the best NLLs consistently under Hawkes-like generative mechanisms, aligning with its additive bias. However, DeepSTPP and AutoSTPP yield better correlation with ground-truth intensity, demonstrating that conditional likelihood fitness does not always equate to latent structure recovery or calibration.
• Autoregressive rollout diagnostics and wall-clock training times expose significant variance in generation stability and computational cost; continuous-time/flow-based models are orders of magnitude more expensive than factorized or window-based baselines.

Practical and Theoretical Implications

Seahorse exposes that state-of-the-art claims in STPP modeling are fragile without standardized execution contracts: leaderboard performance obscures failure modes, and conditioning on dataset-specific preprocessing, coordinate system choice, and likelihood reporting can yield misleading conclusions. The unified framework, coupled with diagnostic synthetic test suites, enables targeted investigation of model inductive biases, robustness to spatiotemporal entanglement, and relationships between likelihood fit and downstream metrics.

For practitioners, Seahorse offers immediate entry points for applied event modeling with preconfigured datasets and models, as well as infrastructure for large-batch benchmarking and diagnostic evaluation. For researchers, the framework enables evaluation of new STPP architectures, sampling methods, and generative mechanisms under fully-auditable, artifact-preserving protocols.

Prospects for Future Research

Continued expansion of curated datasets, integration of new models (including causal, covariate-driven, or operational-task-oriented STPPs), and deeper diagnostic metrics (calibration, uncertainty quantification, fairness) are natural extensions. Richer experimental protocols—e.g., adversarial robustness, simulation-based calibration, and domain-adaptive evaluation—will further our understanding of model limitations and operational constraints.

Conclusion

Seahorse delivers a much-needed experimental backbone for empirical STPP modeling, grounding research in reproducibility, auditability, and controlled diagnostics. Its contributions shift empirical evaluation from isolated leaderboard tables to systematic benchmarking, exposing both the strengths and liabilities of classical, neural, and generative STPP models—promoting scientifically meaningful progress in spatiotemporal event prediction and probabilistic modeling.