Forecasting duration in high-frequency financial data using a self-exciting flexible residual point process

Abstract: This paper presents a method for forecasting limit order book durations using a self-exciting flexible residual point process. High-frequency events in modern exchanges exhibit heavy-tailed interarrival times, posing a significant challenge for accurate prediction. The proposed approach incorporates the empirical distributional features of interarrival times while preserving the self-exciting and decay structure. This work also examines the stochastic stability of the process, which can be interpreted as a general state-space Markov chain. Under suitable conditions, the process is irreducible, aperiodic, positive Harris recurrent, and has a stationary distribution. An empirical study demonstrates that the model achieves strong predictive performance compared with several alternative approaches when forecasting durations in ultra-high-frequency trading data.

• The paper introduces a unified framework that combines autoregressive conditional duration models with Hawkes-type self-exciting processes using flexible, heavy-tailed residual distributions.
• It employs a state-space Markovian structure to capture clustering and non-exponential behavior in ultra-high-frequency limit order book data.
• Empirical evaluations demonstrate that incorporating gamma and Burr residuals significantly improves forecasting accuracy over traditional exponential models.

Self-Exciting Flexible Residual Point Processes for Duration Forecasting in High-Frequency Financial Data

Introduction and Motivation

Accurately forecasting the timing of events in high-frequency limit order book (LOB) data is fundamental for characterizing market microstructure, price discovery, and for building robust trading algorithms. The task is challenging due to the strongly heterogeneous and heavy-tailed nature of interarrival durations, which can vary over several orders of magnitude on sub-second timescales. Traditional stochastic models, such as exponential and gamma processes, are often too restrictive, failing to capture temporal clustering and the non-exponential empirical distribution observed in ultra-high-frequency environments.

The paper introduces a unifying framework that encompasses classical autoregressive conditional duration (ACD) and Hawkes-type self-exciting processes, extending them via the explicit inclusion of flexible, empirically-motivated residual distributions. The result is a point process model parameterized by self-exciting and decay components, with a general state-space Markovian structure. Notably, the framework allows rigorous characterization of probabilistic properties, supports Markov chain stability and stationarity proofs, and demonstrates clear empirical superiority in LOB duration forecasting.

Empirical Features of High-Frequency Durations

A central motivation for the proposed model arises from empirical distributions of LOB durations. For instance, durations between mid-price one-tick moves for AAPL in December 2022 exhibit pronounced overdispersion, severe right-skewness, and extremal kurtosis, indicating the inadequacy of standard exponential modeling.

Figure 2: Histogram of AAPL durations with fitted exponential density, showing extreme prevalence of very short durations and an extended right tail.

The histogram in Figure 1 highlights a sharp modal peak near zero, corresponding to intense clusters of rapid events, and a heavy tail, encapsulating rare but extremely long idle periods. Such empirical features are systematically confirmed by descriptive statistics and cannot be accommodated by exponential or thin-tailed models.

Unified Point Process Modeling Framework

The core theoretical construct is a general point process on $\mathbb{R}_+$ defined via two deterministic mappings:

• $\Phi(t, x)$: strictly increasing in $t$, relating model innovation $\varepsilon_n$ to duration $\tau_n$ and latent state $X_{n-1}$.
• $\Psi(\tau, x)$: specifying the event-time update of the latent state.

This design unifies classical renewal, ACD, and self-exciting models by suitable choices of $\Phi$, $\Psi$, and residual distribution $f_\varepsilon$. The broad framework ensures the sequence of event times $\Phi(t, x)$0 can accommodate history dependence, empirical distribution shapes, and feedback effects.

A key innovation is the introduction of flexible, heavy-tailed residual distributions for $\Phi(t, x)$1, specifically gamma, generalized gamma, and Burr type XII parametrizations, enforcing only that $\Phi(t, x)$2 for identifiability.

Self-Exciting Flexible Residual Point Process

Specializing to the proposed self-exciting process, the latent intensity state $\Phi(t, x)$3 evolves as: $\Phi(t, x)$4 where $\Phi(t, x)$5 is baseline intensity, $\Phi(t, x)$6 controls self-excitation, and $\Phi(t, x)$7 is the rate of exponential decay. The process is driven by a flexible residual $\Phi(t, x)$8, so that the next duration is obtained by inverting the mapping: $\Phi(t, x)$9 This construction retains the positive feedback and decay structure of Hawkes processes but crucially allows the conditional distribution of durations to match the heavy-tailed empirical law observed in data.

From a probabilistic perspective, the process is a general state-space Markov chain. The paper proves irreducibility, aperiodicity, and positive Harris recurrence under the condition $t$0, ensuring the existence and uniqueness of a stationary distribution and robustness for inference and forecasting.

Flexible Distributions for Duration Residuals

Rather than relying on exponential residuals (which are recovered as special cases), the model accommodates:

• Gamma innovations: Parameter $t$1 modulates tail decay, allowing interpolation between exponential ($t$2) and overdispersed cases.
• Generalized Gamma: Provides further control over skew and kurtosis via dual shape parameters.
• Burr Type XII: Capable of modeling severe heavy-tailedness, as often demanded by real-world high-frequency data.

This flexibility yields substantial improvement in fitting both central and tail features of the empirical duration law, as tabulated in the empirical study.

Empirical Evaluation and Dynamic Forecasting

The model is empirically calibrated for ultra-high-frequency AAPL midpoint data. Rolling-window estimations (5,000-event windows) are used to adapt to diurnal and nonstationary trading dynamics, avoiding parametric seasonality corrections. Out-of-sample forecasts of the next 100 interarrival durations are generated via recursive state updates and expected durations are computed numerically—in most cases lacking closed-form solutions except for simple ACD variants.

Key empirical findings:

• The exponential Hawkes model (SE-Exp) fails to capture extreme duration heterogeneity, leading to inflated intensity parameters and poor distributional fit.
• Incorporating gamma and Burr residuals in both self-exciting and ACD classes results in substantial improvements in both predictive accuracy (in terms of relative RMSE and $t$3) and goodness-of-fit (KS and Wasserstein metrics).
• The SE-Burr model yields the best overall balance, improving both predictive accuracy and representational fidelity.

The predictive superiority of flexible residuals is particularly pronounced in goodness-of-fit diagnostics. Probability–probability plots and histograms of model-implied exponential residuals unequivocally demonstrate that heavy-tailed flexible distributions produce residuals much closer to the ideal standard exponential distribution.

Theoretical Properties and Implications

From a theoretical perspective, the framework provides a template for unification and extension of classical intensity-based and duration-based event models. The paper’s rigorous proof of positive Harris recurrence, using Foster–Lyapunov techniques, justifies consistent likelihood-based inference, ergodic estimation, and supports principled application to financial high-frequency settings.

The result has several implications:

• Practical: Enables robust model-based forecasting and simulation of high-frequency event data in modern exchanges, supporting both microstructural research and real-time trading infrastructure.
• Theoretical: Opens new directions for the analysis of nonlinear and history-dependent point processes by combining flexible residuals with self-exciting feedback, potentially accommodating even more complex dependency structures or nonlinear feedback in future work.

Conclusion

The study establishes that intensity-based self-exciting point processes with empirically flexible residuals provide a statistically well-posed and empirically potent approach to modeling LOB durations. Rigorously linking probabilistic theory with real-world data, the approach delivers improvement over classical Hawkes and ACD models, particularly for phenomena featuring clustering and heavy-tailed interarrival distributions. By providing a robust Markovian and likelihood-based framework, the methodology opens avenues for nonlinear and nonparametric extensions, as well as for enhanced application to volatility forecasting, market monitoring, and high-frequency trading.

Future developments may include the explicit modeling of nonstationarity, the incorporation of feedback from price and volume state variables, and nonlinear generalizations of the state evolution functions, all sustained by the flexible residual paradigm introduced in this work.