Continuous-Time Reinforcement Learning for Controlled Hawkes Jump-Diffusions

Abstract: We study stochastic control of multivariate Hawkes-driven stochastic differential equations with machine learning algorithms in a non-Markovian setting. Due to the path dependence of the memory of the Hawkes intensity, this problem does not fall within classical stochastic control theory outside particular Markovian kernels. We first develop a finite-dimensional Markovianization procedure and algorithm to approximate multivariate Hawkes processes with mixtures of exponential kernels. We prove the convergence of the Markovianized approximation of the Hawkes process, its intensity, and the value of the problem to the original non-Markovian processes and the value of the primal problem. We then formulate continuous-time deterministic policy gradient learning on the Markovianized approximation of the problem, called Hawkes-CT DDPG. We propose a model-free algorithm to solve the non-Markovian Hawkes-driven optimization by observing only the event times of the process, the realization of the solution to the SDE, and a chosen set of decay filters, while the Hawkes kernel coefficients remain unknown. We compare our continuous time reinforcement learning Hawkes-CT DDPG method with discrete time reinforcement learning techniques under three different types of kernels: simple exponential, Erlang, and power-law kernels.

• The paper develops Hawkes CT-DDPG, a model-free continuous-time actor–critic method that learns from observed event times and diffusion paths while using online exponential memory filters.
• Signed exponential Markovianization converts general Hawkes kernels into finite-dimensional Markov states, with proven convergence of processes, objectives, and optimal values under stated moment and subcriticality conditions.
• Experiments on exponential, Erlang, and power-law kernels show filtered CT-DDPG consistently outperforms discrete-time RL baselines, achieving costs within roughly 1.1% of the benchmark in the power-law case.

This paper develops a continuous-time reinforcement learning framework for stochastic control of multivariate Hawkes-driven jump-diffusions in a genuinely non-Markovian setting, where the self-excitation kernel is general and its coefficients are unknown to the learner (2608.19151). The contribution combines three elements: a finite-dimensional Markovianization of controlled Hawkes kernels via signed mixtures of exponentials with proven convergence of processes and value functions; a deterministic policy gradient method in continuous time, called Hawkes CT-DDPG, that is model-free apart from observed event times, diffusion realizations, and a user-chosen bank of exponential decay filters; and numerical evidence on three kernel families showing that the method outperforms discrete-time RL baselines.

Controlled Hawkes jump-diffusion model

The state process $X$ solves a jump-diffusion driven by an $m$-dimensional counting process $N$ whose intensity admits the multivariate Hawkes structure

$$\lambda_t^a = \mu(t,X_{t-}^a,a_t) + \int_{(0,t)} \Phi(t-s,a_t)\,dN_s^a,$$

with a matrix-valued kernel $\Phi$ encoding mutual excitation. A notable modeling choice is that the control enters the kernel through the current action $a_t$: past jumps are read through $\Phi(\cdot,a_t)$ at time $t$, so the control governs how accumulated memory is interpreted rather than how it was deposited. The strict upper integration limit $(0,t)$ keeps the intensity predictable and avoids algebraic loops. Admissible controls are predictable $A$-valued processes with compact $m$0, fixed independently of the approximation index $m$1, which allows comparing original and approximating systems under identical controls.

Standing assumptions require Lipschitz coefficients with linear growth for $m$2, entrywise nonnegative integrable kernel envelopes, and uniform (in control) strong well-posedness and moment bounds. The authors explicitly isolate these properties as assumptions rather than proving them, noting they can be verified under Lyapunov-type conditions from prior work; this is a deliberate scoping decision, not a hidden gap.

Signed exponential Markovianization

The core analytic device approximates any integrable kernel by $m$3, using density of signed exponential sums in $m$4. Because signed weights can produce negative raw memory, the approximating intensity applies a componentwise positive part to the memory term only, preserving both the density argument and nonnegativity of intensities. Each memory factor $m$5 solves a linear jump-ODE decaying at rate $m$6 and jumping at events, so the lifted state $m$7 is Markov — and, crucially, constructible online from observed event times alone, without knowledge of $m$8, $m$9, or $N$0. This makes the representation compatible with model-free learning; only the decay scale $N$1 and filter count $N$2 are design choices.

The main convergence theorem establishes, uniformly over admissible controls,

$N$3

together with convergence of counting processes in $N$4, of objectives $N$5, and hence of values $N$6, provided moment exponents satisfy $N$7. The proof proceeds through stopped stability estimates coupling both systems to common Poisson random measures, a Volterra-resolvent comparison controlling the intensity discrepancy by the kernel-approximation error $N$8, localization removal via uniform moments, and Hölder interpolation for the cost error. A sufficient condition for the required uniform moments is given: bounded baseline and jump amplitudes plus uniform subcriticality $N$9, verified via Galton–Watson cluster domination and Borel-distributed total progeny. The implication is that solving the lifted Markovian problem yields policies whose cost converges to the true non-Markovian optimum — the learning algorithm can therefore target a finite-dimensional HJB equation without loss in the limit.

Hawkes CT-DDPG: Bellman structure, martingale losses, and policy gradient

On the lifted state space, the paper derives the fixed-policy Bellman equation for $$\lambda_t^a = \mu(t,X_{t-}^a,a_t) + \int_{(0,t)} \Phi(t-s,a_t)\,dN_s^a,$$0 under an extended generator containing diffusion, memory-decay, and jump terms, with a verification converse. Two identification results drive the learning procedure:

Martingale characterization: if a candidate pair $$\lambda_t^a = \mu(t,X_{t-}^a,a_t) + \int_{(0,t)} \Phi(t-s,a_t)\,dN_s^a,$$1 renders the discounted process $$\lambda_t^a = \mu(t,X_{t-}^a,a_t) + \int_{(0,t)} \Phi(t-s,a_t)\,dN_s^a,$$2 a martingale under locally exploratory controls, then $$\lambda_t^a = \mu(t,X_{t-}^a,a_t) + \int_{(0,t)} \Phi(t-s,a_t)\,dN_s^a,$$3 and $$\lambda_t^a = \mu(t,X_{t-}^a,a_t) + \int_{(0,t)} \Phi(t-s,a_t)\,dN_s^a,$$4 equals the advantage rate $$\lambda_t^a = \mu(t,X_{t-}^a,a_t) + \int_{(0,t)} \Phi(t-s,a_t)\,dN_s^a,$$5. Training therefore drives empirical multi-step increments of this martingale toward zero, yielding critic losses for a value network $$\lambda_t^a = \mu(t,X_{t-}^a,a_t) + \int_{(0,t)} \Phi(t-s,a_t)\,dN_s^a,$$6 and a normalized advantage-rate network $$\lambda_t^a = \mu(t,X_{t-}^a,a_t) + \int_{(0,t)} \Phi(t-s,a_t)\,dN_s^a,$$7 satisfying $$\lambda_t^a = \mu(t,X_{t-}^a,a_t) + \int_{(0,t)} \Phi(t-s,a_t)\,dN_s^a,$$8 by construction.

Deterministic policy gradient: under regularity of the actor parametrization and polynomial growth conditions with $$\lambda_t^a = \mu(t,X_{t-}^a,a_t) + \int_{(0,t)} \Phi(t-s,a_t)\,dN_s^a,$$9, the performance difference lemma gives

$\Phi$0

justified via Vitali's theorem for uniform integrability. The actor loss uses the raw network $\Phi$1 since differentiating the normalized critic would vanish identically. The full algorithm augments standard CT-DDPG machinery (replay buffer, target networks, Polyak averaging) with the online filter update; a binned-count fallback is provided when exact timestamps are unavailable, though the event-exact update is preferred.

Numerical results

Experiments use a one-dimensional generative model with Hill-type saturating attenuation $\Phi$2 of the excitation gain, quadratic running and terminal costs, and three kernels: single exponential (exact one-factor Markov), Erlang (exact two-phase lift), and power-law (no finite Markov representation). Benchmarks are known-parameter DGM/HJB oracles solved by the Deep Galerkin Method, validation-selected static policies, and discrete-time SAC and DDPG given identical observations.

Policy	Single exp.	Erlang	Power-law
DGM/HJB oracle	0.1658 ± 0.0070	0.1037 ± 0.0046	1.5696 ± 0.0617
CT-DDPG filtered	0.1744 ± 0.0073	0.1056 ± 0.0049	1.5870 ± 0.0607
Best discrete RL (filtered)	–	0.1065 (SAC)	1.7056 (SAC)
Static policy	0.3247 ± 0.0214	0.1314 ± 0.0060	1.9242 ± 0.0734

In the single-exponential sanity check, filtered CT-DDPG attains cost 0.1744 versus the oracle's 0.1658, a 46.3% reduction over the static policy, confirming the method exploits an exact finite-dimensional memory state. In the Erlang experiment, filtering reduces mean cost by 13.4% for CT-DDPG relative to current-state observation, and filtered CT-DDPG beats filtered SAC and DDPG by 0.89% and 4.31%, remaining closest to the oracle. For the power-law kernel, filtered CT-DDPG is 6.96% below filtered SAC and 8.85% below filtered DDPG, within roughly 1.1% of the mixture DGM/HJB benchmark. Across all scenarios, Hawkes CT-DDPG is the best learned policy, extending prior CT-DDPG findings to Hawkes-driven diffusions. All excitation masses in the test environments lie strictly below one (0.96, 0.89, 0.96), verifying the subcriticality sufficient condition with $\Phi$3.

Limitations and open questions

The paper is candid about several boundaries. Well-posedness and uniform moments are assumed rather than derived for the nonlinear closed-loop system, with only sufficient conditions supplied. Convergence of the Markovianization is established, but the error incurred by training the actor and critic networks themselves "remains poorly understood within current reinforcement learning theory" and is assessed only numerically against oracles. The power-law benchmark conflates kernel-approximation error with HJB solver error, so no exact oracle exists for that case. The theoretical guarantees also concern the population objective; sample-complexity and stability of the joint actor–critic dynamics are not analyzed. An open question the paper leaves explicit is whether the network-training error can be bounded theoretically, and whether the framework extends beyond the subcritical regime assumed for moment control.

Conclusion

The paper provides a rigorous bridge between non-Markovian Hawkes control and continuous-time actor–critic learning: signed exponential Markovianization with provable value convergence supplies a finite-dimensional HJB surrogate, while a martingale-based critic and advantage-rate policy gradient learn near-oracle policies from observable data alone. Numerically, the filtered Hawkes CT-DDPG consistently dominates discrete-time RL baselines across exponential, Erlang, and power-law kernels, with the largest gains appearing precisely where memory matters most.