EBES: Easy Benchmarking for Event Sequences

Abstract: Event Sequences (EvS) refer to sequential data characterized by irregular sampling intervals and a mix of categorical and numerical features. Accurate classification of these sequences is crucial for various real-life applications, including healthcare, finance, and user interaction. Despite the popularity of the EvS classification task, there is currently no standardized benchmark or rigorous evaluation protocol. This lack of standardization makes it difficult to compare results across studies, which can result in unreliable conclusions and hinder progress in the field. To address this gap, we present EBES, a comprehensive benchmark for EvS classification with sequence-level targets. EBES features standardized evaluation scenarios and protocols, along with an open-source PyTorch library that implements 9 modern models. Additionally, it includes the largest collection of EvS datasets, featuring 10 curated datasets, including a novel synthetic dataset and real-world data with the largest publicly available banking dataset. The library offers user-friendly interfaces for integrating new methods and datasets. Our benchmarking results highlight the unique properties of EvS compared to other sequential data types, provide a performance ranking of modern models with GRU-based models achieving the best results and reveal the challenges associated with robust EvS learning. The goal of EBES is to facilitate reproducible research, expedite progress in the field, and increase the real-world impact of EvS classification techniques.

• The paper introduces EBES, a unified benchmarking framework for event sequences that standardizes model evaluation across diverse datasets.
• It employs multi-phase evaluation with statistical significance testing to fairly compare models such as GRU, CoLES, and others.
• The study highlights dataset quality analysis and offers performance insights to guide future research in hyperparameter optimization and model design.

EBES: Standardized Benchmarking for Event Sequence Models

The lack of standardized benchmarks impedes progress in modeling event sequences, which are prevalent in domains like healthcare, finance, and e-commerce. The paper "EBES: EASY BENCHMARKING FOR EVENT SEQUENCES" addresses this challenge by introducing EBES, a comprehensive benchmarking tool for event sequences (EvS). The proposed framework focuses on regression and classification tasks, providing standardized evaluation protocols and facilitating reproducible research.

Key Contributions

1. Benchmarking Framework: EBES provides a unified interface for datasets, models, and experimental protocols, making it easier for researchers to add datasets and integrate various models. The benchmark includes a variety of datasets, comprising both real-world and synthetic data, like the largest publicly available banking dataset and a novel synthetic pendulum dataset.
2. Evaluation Protocols: The benchmark encompasses various scenarios specific to event sequences, ensuring a fair and consistent comparison across models. The approach includes multi-phase evaluations with statistical significance testing, which enhances the robustness of model assessments.
3. Dataset Analysis: EBES highlights the issues related to dataset quality, proposing synthetic datasets and thorough dataset analysis to ensure reliability. The benchmark includes datasets of various sizes from different domains to examine model scalability and performance under diverse conditions.
4. Methodological Insights: The paper evaluates popular models, such as MLP, GRU, Mamba, Transformer, and some specifically designed for EvS like mTAND, PrimeNet, and CoLES. This evaluation elucidates the relative performance of models under different settings and stress tests, including sequence permutation and temporal component analysis.

Numerical Results

The results underscore that GRU-based models generally outperform others in EvS tasks. CoLES, GRU, and MLEM are consistently among the top performers across multiple datasets. In contrast, the Transformer and Mamba models show comparatively lower efficacy in capturing sequential dependencies inherent to EvS data.

Implications and Future Directions

Practically, EBES aims to streamline model comparison and accelerate research in event sequence modeling by providing a standardized evaluation environment. Theoretically, this work sheds light on the impact of hyperparameter optimization, dataset characteristics, and model architecture on EvS assessment, offering guidelines for future research in this domain.

The paper identifies critical points for future exploration, such as developing more efficient hyperparameter optimization strategies and enhancing model architectures to better account for temporal dependencies in real-world datasets. Additionally, the observations regarding the importance of time and sequence order highlight the need for models that inherently capture these aspects.

Conclusion

EBES stands as an important step toward standardized benchmarking for event sequences, addressing the inconsistencies that hinder the comparability of different modeling approaches. The framework's emphasis on model evaluation rigor, data diversity, and accessibility sets a foundation for more structured and reproducible research in the field. By encouraging contributions from the research community and maintaining benchmark relevance, EBES aims to significantly advance the study of event sequences, fostering innovations that resonate across various applied domains.