---
title: PEC-MDP Formalism and Event-Driven Control
url: https://www.emergentmind.com/topics/pec-mdp-formalism
type: topic
---

# PEC-MDP Formalism and Event-Driven Control

Searching arXiv for the cited papers to ground the article in current records.
In the cited literature, the label PEC-MDP is attached to constructions in which event-driven stochastic dynamics are represented through discrete-time Markov decision machinery by indexing decisions at jumps, boundary hits, or normalized narrative instants rather than at every calendar instant. In the piecewise deterministic Markov process (PDMP) control lineage, the formalism starts from deterministic flow between random jumps and reduces optimal control to a discrete-time MDP on post-jump states or belief states [1706.09142]. In constrained discounted PDMP control, the same event-indexed viewpoint yields operators \(L\), \(H\), and \(G\), discounted occupation measures, and an infinite-dimensional linear program on the embedded jump chain [1402.6105]. In a distinct logic-based lineage, PEC-MDP denotes an MDP-like tuple \((S,A,p_0,T,\mu)\) obtained by translating Probabilistic Event Calculus (PEC) domains into integer-encoded states and action-taking situations, thereby supporting both temporal projection and objective-driven planning with standard MDP tools [2507.12989].

## 1. Defining architecture

A common structural theme is the separation of deterministic evolution between events from stochastic updates at events. In the PDMP formulation, the uncontrolled continuous-time state process \((Y_t)_{t\ge 0}\) evolves through jump times \(0=T_0<T_1<T_2<\dots\), inter-jump times \(S_n=T_n-T_{n-1}\), and post-jump states \(\hat Y_n=Y_{T_n}\). Between jumps,
\[
Y_t=\Phi(\hat Y_n,t-T_n), \qquad T_n\le t<T_{n+1},
\]
where the flow \(\Phi\) satisfies the semigroup property \(\Phi(y,t+s)=\Phi(\Phi(y,s),t)\); jumps are governed by an intensity \(\lambda\) and a transition kernel \(Q\) [1706.09142].

The control-theoretic PEC-MDP perspective arises when actions are chosen at event times and then applied on the whole inter-event interval. In the partially observable PDMP setting, a relaxed control is a measurable map \(r:[0,\infty)\to\mathcal P(A)\), and policies may be written either as piecewise open-loop continuous-time policies \(\pi^P\) or as jump-indexed policies \(\pi^D\), with Lemma 3.1 establishing their equivalence almost everywhere in time. This is the formal bridge that turns a continuous-time event-driven control problem into a discrete-time decision problem indexed by jump number [1706.09142].

The logic-based PEC-MDP uses the same event-indexed principle in a different guise. Time instants \(\mathcal I\) are normalized to \(T=\{0,\dots,n_I-1\}\) by a map \(\delta\), fluent states are encoded as vectors and then as integer states, and actions are not restricted to single primitives. Instead, an action at a step is an **action-taking situation**, namely a subset of the actions that may be performed at that instant, including the empty set. This preserves simultaneous actions and null actions while placing the domain inside an MDP-style state-action-transition representation [2507.12989].

## 2. Event-driven continuous-time control models

In the controlled partially observable PDMP model, the action space \(A\) is assumed to be a compact metric space, and the admissible control space is
\[
\mathcal R=\{r:[0,\infty)\to\mathcal P(A)\text{ measurable}\},
\]
equipped with the Young topology. Control enters the three local characteristics: the controlled flow \(\Phi^r\), the controlled intensity \(\lambda^A(x,a)\), and the controlled jump kernel \(Q^A(\cdot\mid x,a)\). A typical controlled flow is defined by
\[
\frac{d}{dt}\Phi^r(y,t)=\int_A b(\Phi^r(y,t),a)\,r_t(da), \qquad \Phi^r(y,0)=y,
\]
and the cumulative controlled rate is
\[
\Lambda^r(y,t)=\int_0^t\int_A \lambda^A(\Phi^r(y,s),a)\,r_s(da)\,ds.
\]
The next inter-jump time and post-jump state are then distributed according to the joint law induced by \(e^{-\Lambda^r}\), \(\lambda^A\), and \(Q^A\) [1706.09142].

A broader constrained discounted PDMP formulation places the state in an open set \(E\subset\mathbb R^n\) with boundary \(\partial E\), deterministic flow \(\phi\), boundary hitting time
\[
t_*(x)\doteq \inf\{t>0:\phi(x,t)\in\partial E\},
\]
and control pair \(\widehat a=(a,a_\partial)\in\mathbb S(x)=\mathbb U(x)\times\mathbb U(\phi(x,t_*(x)))\). Here the interior control \(a\) drives the rate and transition measure along the flow through a coding map \(\ell(x,a,t)\), while \(a_\partial\) governs the boundary kernel when the trajectory reaches \(\partial E\). The resulting embedded discrete-time model has post-jump state \(Z_k\), stage action \(\Theta_k=(\theta_k,\theta_{k,\partial})\), transition kernel \(G\), and per-stage costs formed from the operators
\[
L v(x,\widehat a)=\int_0^{t_*(x)} e^{-\alpha s-\Lambda^a(x,s)}v(\phi(x,s),a)\,ds,
\]
\[
H w(x,\widehat a)=e^{-\alpha t_*(x)-\Lambda^a(x,t_*(x))}w(\phi(x,t_*(x)),a_\partial).
\]
This is explicitly not the same framework as a standard discounted discrete-time MDP, because the discount factor is absorbed into \(L\), \(H\), and the substochastic kernel \(G\), and because PDMPs include boundary-forced jumps that are not captured by a pure exponential-jump continuous-time Markov chain picture [1402.6105].

## 3. Belief states, filtering, and dynamic programming

Partial observability in the PDMP lineage is concentrated at jump times. The state \(Y_t\) is not directly observed; instead, after each jump the controller receives
\[
X_n=\hat Y_n+\epsilon_n,
\]
where \(\epsilon_n\) are i.i.d. with distribution \(Q_\epsilon\). Observations are event-based, and there are no continuous measurements between jumps. To obtain a finite-dimensional filter, the post-jump state space is assumed finite,
\[
E^0=\{y^1,\dots,y^d\}\subset\mathbb R^d,
\]
with \(Q^A(E^0\mid y,a)=1\) and the initial kernel \(Q_0\) supported on \(E^0\). Beliefs therefore lie in the simplex \(\mathcal P(E^0)\) [1706.09142].

The jump-indexed partially observable process has a substochastic transition density
\[
\tilde q(s,y',x\mid y,r)=e^{-\Gamma^r(y,s)} f_\epsilon(x-y') \int_A \lambda^A(\Phi^r(y,s),a)\,Q^A(y'\mid \Phi^r(y,s),a)\,r_s(da),
\]
with
\[
\Gamma^r(y,t)=\beta t+\int_0^t\int_A \lambda^A(\Phi^r(y,u),a)\,r_u(da)\,du.
\]
The posterior belief update is the Bayes-type filter
\[
\Psi(\rho,r,s,x)(y')=
\frac{\sum_{y\in E^0}\tilde q(s,y',x\mid y,r)\rho(y)}
{\sum_{\hat y\in E^0}\sum_{y\in E^0}\tilde q(s,\hat y,x\mid y,r)\rho(y)}.
\]
For continuity arguments, the paper introduces a regularized filter \(\hat\Psi\) based on mollification in the inter-jump time variable \(s\). Under assumptions (C1)–(C5), (B1), and (B2), the regularized filtered transition kernel is weakly continuous, the one-stage cost \(\hat g(\rho,r)\) is lower semicontinuous, and measurable selectors exist [1706.09142].

The fully observed belief MDP has state space \(\mathcal P(E^0)\), action space \(\mathcal R\), transition kernel \(\hat Q\), and one-stage cost
\[
\hat g(\rho,r)=\sum_{y\in E^0} g(y,r)\rho(y).
\]
Its Bellman operator is
\[
(Tv)(\rho)=\inf_{r\in\mathcal R}\Big\{\hat g(\rho,r)+\int_{\mathcal P(E^0)} v(\rho')\,\hat Q(d\rho'\mid \rho,r)\Big\}.
\]
Theorem 5.2 states that the infinite-horizon value function satisfies the Bellman fixed-point equation \(TV=V\), that \(V\) coincides with the true infinite-horizon value function, and that there exists a measurable selector \(f^*\) such that the stationary policy \((f^*,f^*,\dots)\) is optimal in the belief MDP. Through Lemma 4.1 and Lemma 4.7, this yields an optimal policy for the original partially observable continuous-time PDMP. Remark 5.1 further notes that if \(\lambda^A\) and \(Q^A\) are uncontrolled, the original filter \(\Psi\) is already continuous in the Young topology, so the regularized filter is unnecessary [1706.09142].

## 4. Occupation measures and constrained formulations

The constrained discounted PDMP framework augments the event-driven reduction with occupation-measure analysis. For nonnegative running costs \(f_i\) and boundary costs \(r_i\), the infinite-horizon discounted criterion is
\[
\mathcal D_i^\alpha(U,\nu_0)=E_{\nu_0}^U[\mathbf J_i^\alpha(U,T_\infty)],
\]
where \(\mathbf J_i^\alpha(U,t)\) combines the continuous-time discounted running integral and the discounted boundary-hit costs. Proposition 4.2 rewrites these continuous-time criteria as a jump-indexed sum,
\[
\mathcal D_i^\alpha(U,\nu_0)=\sum_{k=0}^\infty E_{\nu_0}^U\Big[e^{-\alpha T_k}\big(Lf_i(Z_k,\Theta_k)+Hr_i(Z_k,\Theta_k)\big)\Big],
\]
which is the key algebraic step in the PDMP-to-MDP reduction [1402.6105].

The central object is the discounted state-action occupation measure
\[
\mu_{\nu_0}^U(\Gamma)=\sum_{k=0}^\infty E_{\nu_0}^U[e^{-\alpha T_k} I_\Gamma(Z_k,\Theta_k)],
\]
defined on \(\mathcal K\subset E\times\mathbb U\times\mathbb U\). Its marginal \(\widetilde\mu_{\nu_0}^U\) satisfies the balance equation
\[
\widetilde\mu_{\nu_0}^U(B)=\nu_0(B)+\int_{E\times\mathbb S} G(z,\widehat a;B)\,\mu_{\nu_0}^U(dz,d\widehat a),
\]
and each discounted cost becomes the linear functional
\[
\mathcal D_i^\alpha(U,\nu_0)=\int_{\mathcal K} (Lf_i+Hr_i)\,d\mu_{\nu_0}^U.
\]
This yields an infinite-dimensional linear program over \(\mathcal M_w(\mathcal K)_+\) with flow constraints, cost constraints, and weighted integrability conditions [1402.6105].

Theorem 6.1 establishes equivalence between the original constrained discounted PDMP control problem and the LP: every feasible policy induces a feasible occupation measure with matching objective value, and every feasible LP solution induces a randomized stationary policy satisfying the original constraints. The appendix extends a theorem of Dufour–Prieto-Rumeau to state-dependent action sets \(A(x)\), and Theorem 7.2 gives solvability of the LP under lower semicontinuity of \(Lf_i+Hr_i\), compactness and upper semicontinuity of \(\mathbb S(x)\), and weak continuity of \(G\). The result is an optimal randomized stationary policy for the original constrained discounted PDMP problem [1402.6105].

## 5. Numerical approximation and representative applications

Numerical work on PDMPs isolates the embedded discrete-time chain
\[
\Theta_n=(Z_n,S_n),
\]
where \(Z_n=X_{T_n}\) is the post-jump state and \(S_n=T_n-T_{n-1}\) the inter-jump time. For the functional
\[
J_N(l,c)(x)=\mathbb E_x\Big[\int_0^{T_N} l(X_t)\,dt+\sum_{j=1}^N c(X_{T_j^-})\mathbf 1_{\{X_{T_j^-}\in\partial E\}}\Big],
\]
the discontinuous boundary indicator is regularized by the triangular approximation \(\delta^A\), leading to \(J_N^A(l,c)\). The approximation is then computed by quantizing the marginals of \(\Theta_k\) onto grids \(\Gamma_k\), constructing quantized operators \(\widehat G_k\), and performing a backward recursion on the quantized chain. The main error estimate separates quantization error from regularization error and includes the term \(\frac{N C_c C_\lambda}{A}\). The same framework extends to time-dependent functionals and deterministic horizons by using the time-augmented PDMP \(\widetilde X_t=(X_t,t)\) [1105.0839].

The practical interest of this embedded-chain viewpoint is illustrated by several examples. In the repair workshop model, the time-augmented scheme is used to maximize discounted profit over \(t_f=5\) years, yielding an estimated optimal setting \(x^*\approx 0.78\) and \(B(x^*)=537.84\), while a Monte Carlo reference with \(10^8\) simulations gives \(B_{\text{MC}}=537.69\). In the corrosion model, a log-log plot of error versus grid size shows approximate slope \(-0.35\), consistent with the theoretical optimal quantization rate in dimension \(3\) [1105.0839].

Within the partially observable control paper, a generic application considers a one-dimensional particle moving on a line, with post-jump states \(E^0=\{-2,0,2\}\), \(\lambda^A\equiv 1\), \(\beta=1\), discrete observation noise \(f_\epsilon(-1)=f_\epsilon(0)=f_\epsilon(1)=1/3\), and a target-zone cost that vanishes on \([-1.5,1.5]\). Because \(\lambda^A\) and \(Q^A\) are uncontrolled, the original filter is continuous and no regularization is needed. Numerical value iteration then reports effectively bang-bang optimal controls with values in \(\{-1,0,1\}\), determined by the relative posterior masses on the left and right post-jump states [1706.09142].

## 6. Probabilistic Event Calculus translation, planning, and interpretability

In the logic-based lineage, PEC is an action-language formalism built from fluents \(\mathcal F\), actions \(U\), values \(\mathcal V\), time instants \(\mathcal I\), an initial distribution given by an i-proposition, probabilistic causal rules given by c-propositions, and action occurrence models given by p-propositions. The PEC-MDP translation constructs a reward-free MDP-like tuple
\[
(S,A,p_0,T,\mu),
\]
where \(S\) is a finite set of integer-encoded fluent states, \(A\) is a finite set of integer-encoded action-taking situations, \(p_0\) is the initial state distribution, \(T\) is the transition function derived from c-propositions, and \(\mu\) is the possibly non-stationary policy induced by p-propositions [2507.12989].

State construction proceeds by fixing canonical orderings of fluents and values, mapping each fluent state \(\tilde S\) to a value-index vector
\[
x(\tilde S)=(\nu(V_0),\dots,\nu(V_{n_F-1})),
\]
ordering these vectors lexicographically, and assigning each one an integer index \(\sigma(x)\). Actions are encoded from **action-taking situations**. At each instant \(I\),
\[
U_I=\{U\in U:(U\ \textit{performed-at}\ I)\in D\},
\]
and the possible situations are elements of the powerset \(\mathcal P(U_I)\), including singleton, concurrent, and null-action cases. Each situation receives an integer code \(a=\alpha(A_u)\). This preserves PEC’s flexible semantics, in which multiple actions may occur simultaneously and some instants may contain no action at all [2507.12989].

The transition kernel is generated by c-propositions through the update operator \(\Upsilon\), which overwrites the coordinates of a state vector specified by a partial fluent state while leaving the remaining coordinates unchanged. If no c-proposition matches a given state-action pair, then persistence holds:
\[
T(s,a,s')=
\begin{cases}
1 & \text{if } s'=s,\\
0 & \text{otherwise.}
\end{cases}
\]
Otherwise, if a c-proposition body \(\theta\) is satisfied and the action situation matches, then
\[
T(\sigma(x),a,\sigma(x'))=\sum_{X_j:\Upsilon(X_j,x)=x'} P_j^+.
\]
The policy \(\mu(a,s,t)\) is derived from p-propositions by first building per-action probabilities \(p_{j,k}^{(I)}\) and then using a product formula over performed and non-performed primitive actions. This implements an independence assumption at the action-occurrence level [2507.12989].

Temporal projection becomes matrix propagation. With policy-weighted transition matrices
\[
[\mathbf M_t]_{s,s'}=\sum_{a\in A}\mu(a,s,t)\,T(s,a,s'),
\]
the state distribution evolves by
\[
\mathbf p_t=\mathbf p_{t-1}^T \mathbf M_{t-1},
\]
and the probability that a partial fluent state \(X_Q\) holds at time \(I_Q\) is recovered from the \(L^1\)-mass of \(\mathbf p_{t_Q}\) on the corresponding indicator vector. Once a reward function \(R\) is added, the PEC-MDP becomes a standard MDP and supports value iteration, policy iteration, Q-learning, SARSA, policy gradient variants, and related planning or reinforcement-learning methods. Deterministic stationary or non-stationary policies can then be translated back into human-readable PEC p-propositions, with further refinement by reachability pruning and minimal fluent conditions. The same source also records the principal limitations: finite domains, state-space size, the independence assumption in action occurrence, the absence of rewards in PEC itself, and the causal exclusivity requirement for c-propositions [2507.12989].

Taken together, these developments suggest a broad technical reading of PEC-MDP formalism: an event-indexed reduction in which rich temporal or hybrid dynamics are encoded so that continuous-time costs, partial observability, logical causation, and concurrent actions can be analyzed by discrete-time MDP, belief-MDP, or LP methods without discarding the original event structure.

Source: https://www.emergentmind.com/topics/pec-mdp-formalism