PEC-MDP Formalism and Event-Driven Control

• PEC-MDP formalism is an event-indexed framework that models continuous stochastic dynamics using discrete-time decision processes.
• It reduces complex continuous control problems to tractable MDPs via PDMP techniques, ensuring optimal policy derivation through dynamic programming.
• The approach supports planning with probabilistic event calculus and occupation measure analysis, addressing constraints and partial observability in control.

Searching arXiv for the cited papers to ground the article in current records. In the cited literature, the label PEC-MDP is attached to constructions in which event-driven stochastic dynamics are represented through discrete-time Markov decision machinery by indexing decisions at jumps, boundary hits, or normalized narrative instants rather than at every calendar instant. In the piecewise deterministic Markov process (PDMP) control lineage, the formalism starts from deterministic flow between random jumps and reduces optimal control to a discrete-time MDP on post-jump states or belief states (Bäuerle et al., 2017). In constrained discounted PDMP control, the same event-indexed viewpoint yields operators $L$, $H$, and $G$, discounted occupation measures, and an infinite-dimensional linear program on the embedded jump chain (Costa et al., 2014). In a distinct logic-based lineage, PEC-MDP denotes an MDP-like tuple $(S,A,p_0,T,\mu)$ obtained by translating Probabilistic Event Calculus (PEC) domains into integer-encoded states and action-taking situations, thereby supporting both temporal projection and objective-driven planning with standard MDP tools (Xu et al., 17 Jul 2025).

1. Defining architecture

A common structural theme is the separation of deterministic evolution between events from stochastic updates at events. In the PDMP formulation, the uncontrolled continuous-time state process $(Y_t)_{t\ge 0}$ evolves through jump times $0=T_0<T_1<T_2<\dots$, inter-jump times $S_n=T_n-T_{n-1}$, and post-jump states $\hat Y_n=Y_{T_n}$. Between jumps,

$Y_t=\Phi(\hat Y_n,t-T_n), \qquad T_n\le t<T_{n+1},$

where the flow $\Phi$ satisfies the semigroup property $H$0; jumps are governed by an intensity $H$1 and a transition kernel $H$2 (Bäuerle et al., 2017).

The control-theoretic PEC-MDP perspective arises when actions are chosen at event times and then applied on the whole inter-event interval. In the partially observable PDMP setting, a relaxed control is a measurable map $H$3, and policies may be written either as piecewise open-loop continuous-time policies $H$4 or as jump-indexed policies $H$5, with Lemma 3.1 establishing their equivalence almost everywhere in time. This is the formal bridge that turns a continuous-time event-driven control problem into a discrete-time decision problem indexed by jump number (Bäuerle et al., 2017).

The logic-based PEC-MDP uses the same event-indexed principle in a different guise. Time instants $H$6 are normalized to $H$7 by a map $H$8, fluent states are encoded as vectors and then as integer states, and actions are not restricted to single primitives. Instead, an action at a step is an action-taking situation, namely a subset of the actions that may be performed at that instant, including the empty set. This preserves simultaneous actions and null actions while placing the domain inside an MDP-style state-action-transition representation (Xu et al., 17 Jul 2025).

2. Event-driven continuous-time control models

In the controlled partially observable PDMP model, the action space $H$9 is assumed to be a compact metric space, and the admissible control space is

$G$0

equipped with the Young topology. Control enters the three local characteristics: the controlled flow $G$1, the controlled intensity $G$2, and the controlled jump kernel $G$3. A typical controlled flow is defined by

$G$4

and the cumulative controlled rate is

$G$5

The next inter-jump time and post-jump state are then distributed according to the joint law induced by $G$6, $G$7, and $G$8 (Bäuerle et al., 2017).

A broader constrained discounted PDMP formulation places the state in an open set $G$9 with boundary $(S,A,p_0,T,\mu)$0, deterministic flow $(S,A,p_0,T,\mu)$1, boundary hitting time

$(S,A,p_0,T,\mu)$2

and control pair $(S,A,p_0,T,\mu)$3. Here the interior control $(S,A,p_0,T,\mu)$4 drives the rate and transition measure along the flow through a coding map $(S,A,p_0,T,\mu)$5, while $(S,A,p_0,T,\mu)$6 governs the boundary kernel when the trajectory reaches $(S,A,p_0,T,\mu)$7. The resulting embedded discrete-time model has post-jump state $(S,A,p_0,T,\mu)$8, stage action $(S,A,p_0,T,\mu)$9, transition kernel $(Y_t)_{t\ge 0}$0, and per-stage costs formed from the operators

$(Y_t)_{t\ge 0}$1

$(Y_t)_{t\ge 0}$2

This is explicitly not the same framework as a standard discounted discrete-time MDP, because the discount factor is absorbed into $(Y_t)_{t\ge 0}$3, $(Y_t)_{t\ge 0}$4, and the substochastic kernel $(Y_t)_{t\ge 0}$5, and because PDMPs include boundary-forced jumps that are not captured by a pure exponential-jump continuous-time Markov chain picture (Costa et al., 2014).

3. Belief states, filtering, and dynamic programming

Partial observability in the PDMP lineage is concentrated at jump times. The state $(Y_t)_{t\ge 0}$6 is not directly observed; instead, after each jump the controller receives

$(Y_t)_{t\ge 0}$7

where $(Y_t)_{t\ge 0}$8 are i.i.d. with distribution $(Y_t)_{t\ge 0}$9. Observations are event-based, and there are no continuous measurements between jumps. To obtain a finite-dimensional filter, the post-jump state space is assumed finite,

$0=T_0<T_1<T_2<\dots$0

with $0=T_0<T_1<T_2<\dots$1 and the initial kernel $0=T_0<T_1<T_2<\dots$2 supported on $0=T_0<T_1<T_2<\dots$3. Beliefs therefore lie in the simplex $0=T_0<T_1<T_2<\dots$4 (Bäuerle et al., 2017).

The jump-indexed partially observable process has a substochastic transition density

$0=T_0<T_1<T_2<\dots$5

with

$0=T_0<T_1<T_2<\dots$6

The posterior belief update is the Bayes-type filter

$0=T_0<T_1<T_2<\dots$7

For continuity arguments, the paper introduces a regularized filter $0=T_0<T_1<T_2<\dots$8 based on mollification in the inter-jump time variable $0=T_0<T_1<T_2<\dots$9. Under assumptions (C1)–(C5), (B1), and (B2), the regularized filtered transition kernel is weakly continuous, the one-stage cost $S_n=T_n-T_{n-1}$0 is lower semicontinuous, and measurable selectors exist (Bäuerle et al., 2017).

The fully observed belief MDP has state space $S_n=T_n-T_{n-1}$1, action space $S_n=T_n-T_{n-1}$2, transition kernel $S_n=T_n-T_{n-1}$3, and one-stage cost

$S_n=T_n-T_{n-1}$4

Its Bellman operator is

$S_n=T_n-T_{n-1}$5

Theorem 5.2 states that the infinite-horizon value function satisfies the Bellman fixed-point equation $S_n=T_n-T_{n-1}$6, that $S_n=T_n-T_{n-1}$7 coincides with the true infinite-horizon value function, and that there exists a measurable selector $S_n=T_n-T_{n-1}$8 such that the stationary policy $S_n=T_n-T_{n-1}$9 is optimal in the belief MDP. Through Lemma 4.1 and Lemma 4.7, this yields an optimal policy for the original partially observable continuous-time PDMP. Remark 5.1 further notes that if $\hat Y_n=Y_{T_n}$0 and $\hat Y_n=Y_{T_n}$1 are uncontrolled, the original filter $\hat Y_n=Y_{T_n}$2 is already continuous in the Young topology, so the regularized filter is unnecessary (Bäuerle et al., 2017).

4. Occupation measures and constrained formulations

The constrained discounted PDMP framework augments the event-driven reduction with occupation-measure analysis. For nonnegative running costs $\hat Y_n=Y_{T_n}$3 and boundary costs $\hat Y_n=Y_{T_n}$4, the infinite-horizon discounted criterion is

$\hat Y_n=Y_{T_n}$5

where $\hat Y_n=Y_{T_n}$6 combines the continuous-time discounted running integral and the discounted boundary-hit costs. Proposition 4.2 rewrites these continuous-time criteria as a jump-indexed sum,

$\hat Y_n=Y_{T_n}$7

which is the key algebraic step in the PDMP-to-MDP reduction (Costa et al., 2014).

The central object is the discounted state-action occupation measure

$\hat Y_n=Y_{T_n}$8

defined on $\hat Y_n=Y_{T_n}$9. Its marginal $Y_t=\Phi(\hat Y_n,t-T_n), \qquad T_n\le t<T_{n+1},$0 satisfies the balance equation

$Y_t=\Phi(\hat Y_n,t-T_n), \qquad T_n\le t<T_{n+1},$1

and each discounted cost becomes the linear functional

$Y_t=\Phi(\hat Y_n,t-T_n), \qquad T_n\le t<T_{n+1},$2

This yields an infinite-dimensional linear program over $Y_t=\Phi(\hat Y_n,t-T_n), \qquad T_n\le t<T_{n+1},$3 with flow constraints, cost constraints, and weighted integrability conditions (Costa et al., 2014).

Theorem 6.1 establishes equivalence between the original constrained discounted PDMP control problem and the LP: every feasible policy induces a feasible occupation measure with matching objective value, and every feasible LP solution induces a randomized stationary policy satisfying the original constraints. The appendix extends a theorem of Dufour–Prieto-Rumeau to state-dependent action sets $Y_t=\Phi(\hat Y_n,t-T_n), \qquad T_n\le t<T_{n+1},$4, and Theorem 7.2 gives solvability of the LP under lower semicontinuity of $Y_t=\Phi(\hat Y_n,t-T_n), \qquad T_n\le t<T_{n+1},$5, compactness and upper semicontinuity of $Y_t=\Phi(\hat Y_n,t-T_n), \qquad T_n\le t<T_{n+1},$6, and weak continuity of $Y_t=\Phi(\hat Y_n,t-T_n), \qquad T_n\le t<T_{n+1},$7. The result is an optimal randomized stationary policy for the original constrained discounted PDMP problem (Costa et al., 2014).

5. Numerical approximation and representative applications

Numerical work on PDMPs isolates the embedded discrete-time chain

$Y_t=\Phi(\hat Y_n,t-T_n), \qquad T_n\le t<T_{n+1},$8

where $Y_t=\Phi(\hat Y_n,t-T_n), \qquad T_n\le t<T_{n+1},$9 is the post-jump state and $\Phi$0 the inter-jump time. For the functional

$\Phi$1

the discontinuous boundary indicator is regularized by the triangular approximation $\Phi$2, leading to $\Phi$3. The approximation is then computed by quantizing the marginals of $\Phi$4 onto grids $\Phi$5, constructing quantized operators $\Phi$6, and performing a backward recursion on the quantized chain. The main error estimate separates quantization error from regularization error and includes the term $\Phi$7. The same framework extends to time-dependent functionals and deterministic horizons by using the time-augmented PDMP $\Phi$8 (Brandejsky et al., 2011).

The practical interest of this embedded-chain viewpoint is illustrated by several examples. In the repair workshop model, the time-augmented scheme is used to maximize discounted profit over $\Phi$9 years, yielding an estimated optimal setting $H$00 and $H$01, while a Monte Carlo reference with $H$02 simulations gives $H$03. In the corrosion model, a log-log plot of error versus grid size shows approximate slope $H$04, consistent with the theoretical optimal quantization rate in dimension $H$05 (Brandejsky et al., 2011).

Within the partially observable control paper, a generic application considers a one-dimensional particle moving on a line, with post-jump states $H$06, $H$07, $H$08, discrete observation noise $H$09, and a target-zone cost that vanishes on $H$10. Because $H$11 and $H$12 are uncontrolled, the original filter is continuous and no regularization is needed. Numerical value iteration then reports effectively bang-bang optimal controls with values in $H$13, determined by the relative posterior masses on the left and right post-jump states (Bäuerle et al., 2017).

6. Probabilistic Event Calculus translation, planning, and interpretability

In the logic-based lineage, PEC is an action-language formalism built from fluents $H$14, actions $H$15, values $H$16, time instants $H$17, an initial distribution given by an i-proposition, probabilistic causal rules given by c-propositions, and action occurrence models given by p-propositions. The PEC-MDP translation constructs a reward-free MDP-like tuple

$H$18

where $H$19 is a finite set of integer-encoded fluent states, $H$20 is a finite set of integer-encoded action-taking situations, $H$21 is the initial state distribution, $H$22 is the transition function derived from c-propositions, and $H$23 is the possibly non-stationary policy induced by p-propositions (Xu et al., 17 Jul 2025).

State construction proceeds by fixing canonical orderings of fluents and values, mapping each fluent state $H$24 to a value-index vector

$H$25

ordering these vectors lexicographically, and assigning each one an integer index $H$26. Actions are encoded from action-taking situations. At each instant $H$27,

$H$28

and the possible situations are elements of the powerset $H$29, including singleton, concurrent, and null-action cases. Each situation receives an integer code $H$30. This preserves PEC’s flexible semantics, in which multiple actions may occur simultaneously and some instants may contain no action at all (Xu et al., 17 Jul 2025).

The transition kernel is generated by c-propositions through the update operator $H$31, which overwrites the coordinates of a state vector specified by a partial fluent state while leaving the remaining coordinates unchanged. If no c-proposition matches a given state-action pair, then persistence holds: $H$32 Otherwise, if a c-proposition body $H$33 is satisfied and the action situation matches, then

$H$34

The policy $H$35 is derived from p-propositions by first building per-action probabilities $H$36 and then using a product formula over performed and non-performed primitive actions. This implements an independence assumption at the action-occurrence level (Xu et al., 17 Jul 2025).

Temporal projection becomes matrix propagation. With policy-weighted transition matrices

$H$37

the state distribution evolves by

$H$38

and the probability that a partial fluent state $H$39 holds at time $H$40 is recovered from the $H$41-mass of $H$42 on the corresponding indicator vector. Once a reward function $H$43 is added, the PEC-MDP becomes a standard MDP and supports value iteration, policy iteration, Q-learning, SARSA, policy gradient variants, and related planning or reinforcement-learning methods. Deterministic stationary or non-stationary policies can then be translated back into human-readable PEC p-propositions, with further refinement by reachability pruning and minimal fluent conditions. The same source also records the principal limitations: finite domains, state-space size, the independence assumption in action occurrence, the absence of rewards in PEC itself, and the causal exclusivity requirement for c-propositions (Xu et al., 17 Jul 2025).

Taken together, these developments suggest a broad technical reading of PEC-MDP formalism: an event-indexed reduction in which rich temporal or hybrid dynamics are encoded so that continuous-time costs, partial observability, logical causation, and concurrent actions can be analyzed by discrete-time MDP, belief-MDP, or LP methods without discarding the original event structure.