EasyTPP: Towards Open Benchmarking Temporal Point Processes (2307.08097v3)

Abstract: Continuous-time event sequences play a vital role in real-world domains such as healthcare, finance, online shopping, social networks, and so on. To model such data, temporal point processes (TPPs) have emerged as the most natural and competitive models, making a significant impact in both academic and application communities. Despite the emergence of many powerful models in recent years, there hasn't been a central benchmark for these models and future research endeavors. This lack of standardization impedes researchers and practitioners from comparing methods and reproducing results, potentially slowing down progress in this field. In this paper, we present EasyTPP, the first central repository of research assets (e.g., data, models, evaluation programs, documentations) in the area of event sequence modeling. Our EasyTPP makes several unique contributions to this area: a unified interface of using existing datasets and adding new datasets; a wide range of evaluation programs that are easy to use and extend as well as facilitate reproducible research; implementations of popular neural TPPs, together with a rich library of modules by composing which one could quickly build complex models. All the data and implementation can be found at https://github.com/ant-research/EasyTemporalPointProcess. We will actively maintain this benchmark and welcome contributions from other researchers and practitioners. Our benchmark will help promote reproducible research in this field, thus accelerating research progress as well as making more significant real-world impacts.

• The paper introduces a centralized, extensible benchmarking framework that standardizes datasets and model evaluation for temporal point processes.
• It employs a comprehensive suite of metrics—including likelihood, RMSE, and error rates—to compare neural and traditional models effectively.
• Its design supports cross-framework compatibility and future enhancements such as foundation models and integrated auxiliary data for improved predictions.

Insights into EasyTPP: A Benchmarking Framework for Temporal Point Processes

The paper "EasyTPP: Towards Open Benchmarking for Temporal Point Processes" introduces EasyTPP, a pioneering effort to establish a centralized repository and benchmarking framework for the evaluation and development of Temporal Point Processes (TPPs). TPPs are a class of probabilistic models designed to handle continuous-time event sequences, making them well-suited for fields such as healthcare, finance, and social networking. The work addresses a pressing need in the research community for standardized benchmarks to facilitate reproducible results and method comparisons.

Core Contribution and Implementation

The EasyTPP framework offers several key innovations:

1. Standardization: The authors have curated datasets spanning diverse domains and developed a unified data format. This ensures that researchers can avoid redundant preprocessing efforts, thus streamlining the exploration of tasks like transfer learning.
2. Comprehensive Evaluation: The project provides a suite of evaluation metrics - including likelihood-based assessments, RMSE for time prediction, and error rates for type prediction - enabling a thorough comparison of model performance. Additionally, significance tests such as permutation tests are included to assess statistical relevance.
3. Rich Library of Modules: EasyTPP provides a modular approach to model building, facilitating the rapid development of complex model architectures. It incorporates both neural architectures, traditional ODE-based models, and intensity-free frameworks, covering a wide spectrum of the current TPP landscape.
4. Cross-Framework Compatibility: Designed with flexibility in mind, EasyTPP supports both PyTorch and TensorFlow, the predominant deep learning frameworks, thus broadening its applicability across different research and industrial environments.
5. High Extensibility: The platform is structured for easy integration of new datasets, models, and evaluation tools. This extensibility encourages community contributions, thereby enhancing its utility and scope.

Experimental Observations and Insights

Among the nine models evaluated, which include traditional and cutting-edge neural approaches, an intensity-free model namely IFTPP showed superior performance in likelihood evaluation, indicative of advancements in non-standard TPP methodologies. Conversely, the classical multivariate Hawkes Process lagged in predictive tasks, reaffirming the efficacy of neural adaptations.

In the specific setting of next-event prediction, attention-based models generally outperformed RNN-based models. This finding underscores the importance of capturing long-range dependencies in temporal sequences—a clear strength of attention mechanisms.

Furthermore, the paper introduces a novel task, long-horizon prediction, which critiques models based on optimal transport distance (OTD) over extended event sequences. Although models such as THP showed promise, results indicated susceptibility to error propagation, suggesting a need for alternative modeling techniques that can effectively manage these challenges.

Future Directions and Implications

This paper delineates several avenues for future improvement and exploration:

• Foundation Models for Event Sequences: The potential for transfer learning and the deployment of large pre-trained models akin to NLP’s GPTs suggests a path forward in building generalized TPP models. EasyTPP’s infrastructure is well-equipped to support such interdisciplinary research initiatives.
• Integration with External Data Sources: Utilizing auxiliary information beyond the immediate event data, such as integrating knowledge bases or leveraging sensor data, could enhance model predictions.
• Embedding TPPs in Causal Frameworks and Real-World Interventions: Engaging TPPs in interactive settings where they can refine predictions based on feedback from actual interventions promises substantial practical gains, opening a new frontier for research.

By presenting a platform like EasyTPP, this work not only fills the current void for standardized TPP benchmarking but also propels the field towards innovative, cross-disciplinary research practices that promise to elevate the precision and applicability of temporal models.