Modelling and understanding count processes through a Markov-modulated non-homogeneous Poisson process framework

Abstract: The Markov-modulated Poisson process is utilised for count modelling in a variety of areas such as queueing, reliability, network and insurance claims analysis. In this paper, we extend the Markov-modulated Poisson process framework through the introduction of a flexible frequency perturbation measure. This contribution enables known information of observed event arrivals to be naturally incorporated in a tractable manner, while the hidden Markov chain captures the effect of unobservable drivers of the data. In addition to increases in accuracy and interpretability, this method supplements analysis of the latent factors. Further, this procedure naturally incorporates data features such as over-dispersion and autocorrelation. Additional insights can be generated to assist analysis, including a procedure for iterative model improvement. Implementation difficulties are also addressed with a focus on dealing with large data sets, where latent models are especially advantageous due the large number of observations facilitating identification of hidden factors. Namely, computational issues such as numerical underflow and high processing cost arise in this context and in this paper, we produce procedures to overcome these problems. This modelling framework is demonstrated using a large insurance data set to illustrate theoretical, practical and computational contributions and an empirical comparison to other count models highlight the advantages of the proposed approach.