Event-Driven MDPs: Models, Methods, and Challenges

• Event-Driven MDPs are sequential decision models that integrate external events into state evolution, action semantics, or decision timing.
• They enable diverse modeling approaches, including asynchronous macro-actions, exogenous dynamics, and event-calculus translations to enhance classical MDP frameworks.
• Algorithmic implementations like ExAVI, M-GAE, and DES-augmented RL demonstrate practical benefits in control, trading, and resilience applications.

An event-driven Markov decision process denotes a family of sequential decision models in which events play a constitutive role in state evolution, action semantics, policy conditioning, or decision timing. The literature does not use the term in a single canonical sense. Instead, it spans at least four technically distinct interpretations: discrete-time MDPs perturbed by external temporal processes; structured MDPs with exogenous state components; semi-Markov or asynchronous decision processes in which decisions occur at event times; and event-centric logical or supervisory models compiled into or coupled with MDP machinery (Ayyagari et al., 2023, Maran et al., 3 Mar 2026, Menda et al., 2017, Xu et al., 17 Jul 2025). Across these uses, the common theme is that transitions are not treated as arising solely from a stationary action-conditioned kernel on a uniformly ticking clock.

1. Terminological scope and historical uses

The phrase has been used for materially different model classes. In one line of work, it denotes a control problem in which an underlying MDP is perturbed by an external marked temporal process, so that the effective transition kernel becomes $Q_{H_t}(\cdot\mid s,a)$ and depends on event history $H_t$ (Ayyagari et al., 2023). In another, it refers to environments with exogenous state components whose transitions are independent of the agent’s actions, yielding the factorization

$$p_h(s_{h+1}\mid s_h,a_h)=p_h^\diamond(s_{h+1}^\diamond\mid s_h^\diamond,s_h^\bullet,a_h)\,p_h^\bullet(s_{h+1}^\bullet\mid s_h^\bullet)$$

in a Partially Controllable MDP (PCMDP) (Maran et al., 3 Mar 2026). A third usage is explicitly semi-Markov and asynchronous: high-level decisions are made at event times because macro-actions have stochastic durations and terminate when events occur (Menda et al., 2017). A fourth appears in symbolic reasoning, where event calculi are translated into finite-horizon MDPs by encoding concurrent or null action occurrences as “action-taking situations” and using a non-stationary policy over time-indexed instants (Xu et al., 17 Jul 2025).

The literature also contains looser or adjacent usages. “Finite-Horizon Markov Decision Processes with Sequentially-Observed Transitions” studies a discrete-time finite-horizon MDP in which candidate transition outcomes are sequentially revealed within a stage before commitment; the paper is explicitly characterized as not being event-timed in the semi-Markov sense, but only event-related through within-stage observations (Chamie et al., 2015). “Adaptive Reinforcement Learning for State Avoidance in Discrete Event Systems” presents a DES-augmented RL architecture rather than a formally defined event-driven MDP, combining a Discrete Event Supervisory model with RL under partial observability and belief-state updates (Dony, 28 Feb 2025). “Event-Driven Models” proposes a broader conceptual generalization in which action-driven models are a special case of event-driven models, with state changes occurring only when designated events occur (Dobrev, 2019).

This diversity suggests that the term is best treated as an umbrella label rather than a single textbook object. A plausible implication is that any precise use of the term should specify whether the emphasis is on event-triggered timing, exogenous disturbance structure, event-conditioned transition semantics, or symbolic event representation.

2. Core modeling patterns

A useful way to organize the literature is by asking what exactly is “event-driven” in the model. The answer differs by framework.

Interpretation	Representative formulation	Event role
External temporal process	$Q_{H_t}(\cdot\mid s,a)$ (Ayyagari et al., 2023)	Event history perturbs transitions
Exogenous dynamics	$p^\diamond p^\bullet$ factorization (Maran et al., 3 Mar 2026)	External state evolves independently of action
Asynchronous macro-action process	event-indexed trajectories $(o_{i,k},m_{i,k},\Delta t_{i,k},r_{i,k})$ (Menda et al., 2017)	Decisions occur at event times
Event-calculus compilation	action-taking situations and $\mu(a,s,t)$ (Xu et al., 17 Jul 2025)	Events define action semantics in discrete time
DES-augmented RL	DES supervisor plus RL agent (Dony, 28 Feb 2025)	Events constrain and inform learning
Sequentially observed transitions	acceptance probabilities $P_i(j,k,t)$ (Chamie et al., 2015)	Observed transition outcomes trigger within-stage accept/reject

In the external-temporal-process formulation, an ordinary MDP $M=(S,A,Q,r,\gamma)$ is exposed to an exogenous marked event process with history

$H_t=\{(t_1,x_1),(t_2,x_2),\dots,(t_j,x_j)\},$

and the transition law is replaced by $H_t$0 (Ayyagari et al., 2023). Since two visits to the same $H_t$1 pair may then yield different next-state laws if event histories differ, the original state process is non-stationary and generally non-Markov in $H_t$2 alone. Exact Markovianity is recovered by augmenting state with event history.

In the exogenous-dynamics formulation, the full state factors into endogenous and exogenous components, $H_t$3, and only $H_t$4 is action-sensitive (Maran et al., 3 Mar 2026). This is not asynchronous or semi-Markov, but it is event-driven in the sense that disturbances, arrivals, prices, congestion, weather, or other externally evolving signals are explicitly represented as action-independent dynamics.

In the asynchronous macro-action setting, the event-driven character lies in timing rather than only in transition content. Each agent executes a macro-action $H_t$5 for a random duration $H_t$6, and the next decision occurs when an event in the macro-action’s event set $H_t$7 occurs (Menda et al., 2017). This is the clearest event-time interpretation in the supplied literature.

By contrast, event-calculus translation remains discrete-time. Time instants are normalized to

$H_t$8

and a non-stationary policy $H_t$9 expresses the probability of an “action-taking situation” at each time and state (Xu et al., 17 Jul 2025). Here the model is event-centric in semantics, not event-triggered in clocking.

3. State, action, observation, and time

The state representation in event-driven formulations is often richer than in a stationary MDP because it must absorb timing, exogenous context, or event history.

For MDPs under external temporal processes, the exact Markov state is augmented to

$$p_h(s_{h+1}\mid s_h,a_h)=p_h^\diamond(s_{h+1}^\diamond\mid s_h^\diamond,s_h^\bullet,a_h)\,p_h^\bullet(s_{h+1}^\bullet\mid s_h^\bullet)$$0

yielding an infinite-dimensional augmented MDP $$p_h(s_{h+1}\mid s_h,a_h)=p_h^\diamond(s_{h+1}^\diamond\mid s_h^\diamond,s_h^\bullet,a_h)\,p_h^\bullet(s_{h+1}^\bullet\mid s_h^\bullet)$$1 (Ayyagari et al., 2023). The purpose of this construction is explicit: the original process under exogenous perturbations “does not remain an MDP,” but the augmented process does.

For PCMDPs, the state is explicitly factored as

$$p_h(s_{h+1}\mid s_h,a_h)=p_h^\diamond(s_{h+1}^\diamond\mid s_h^\diamond,s_h^\bullet,a_h)\,p_h^\bullet(s_{h+1}^\bullet\mid s_h^\bullet)$$2

with $$p_h(s_{h+1}\mid s_h,a_h)=p_h^\diamond(s_{h+1}^\diamond\mid s_h^\diamond,s_h^\bullet,a_h)\,p_h^\bullet(s_{h+1}^\bullet\mid s_h^\bullet)$$3 exogenous and $$p_h(s_{h+1}\mid s_h,a_h)=p_h^\diamond(s_{h+1}^\diamond\mid s_h^\diamond,s_h^\bullet,a_h)\,p_h^\bullet(s_{h+1}^\bullet\mid s_h^\bullet)$$4 controllable (Maran et al., 3 Mar 2026). Rewards may depend on both components and the action, so exogenous variables may dominate stochasticity without being controllable.

For event-driven multi-agent macro-action control, the underlying state remains that of a Dec-POMDP or MacDec-POMDP, but policy execution is indexed by event decisions. Agent $$p_h(s_{h+1}\mid s_h,a_h)=p_h^\diamond(s_{h+1}^\diamond\mid s_h^\diamond,s_h^\bullet,a_h)\,p_h^\bullet(s_{h+1}^\bullet\mid s_h^\bullet)$$5 observes histories $$p_h(s_{h+1}\mid s_h,a_h)=p_h^\diamond(s_{h+1}^\diamond\mid s_h^\diamond,s_h^\bullet,a_h)\,p_h^\bullet(s_{h+1}^\bullet\mid s_h^\bullet)$$6 and acts via

$$p_h(s_{h+1}\mid s_h,a_h)=p_h^\diamond(s_{h+1}^\diamond\mid s_h^\diamond,s_h^\bullet,a_h)\,p_h^\bullet(s_{h+1}^\bullet\mid s_h^\bullet)$$7

while trajectories record tuples of the form

$$p_h(s_{h+1}\mid s_h,a_h)=p_h^\diamond(s_{h+1}^\diamond\mid s_h^\diamond,s_h^\bullet,a_h)\,p_h^\bullet(s_{h+1}^\bullet\mid s_h^\bullet)$$8

(Menda et al., 2017). Reward accrual is continuous over macro-action duration: $$p_h(s_{h+1}\mid s_h,a_h)=p_h^\diamond(s_{h+1}^\diamond\mid s_h^\diamond,s_h^\bullet,a_h)\,p_h^\bullet(s_{h+1}^\bullet\mid s_h^\bullet)$$9

DES-augmented RL introduces a different state semantics. The nominal DES has $Q_{H_t}(\cdot\mid s,a)$0, with an undesirable avoid state $Q_{H_t}(\cdot\mid s,a)$1, actions $Q_{H_t}(\cdot\mid s,a)$2, partial observability, and belief states $Q_{H_t}(\cdot\mid s,a)$3, where each $Q_{H_t}(\cdot\mid s,a)$4 maps to a probability distribution over $Q_{H_t}(\cdot\mid s,a)$5 (Dony, 28 Feb 2025). The paper explicitly states that state transition probabilities are deterministic in its model, while supervisory choices are enabled event sets such as $Q_{H_t}(\cdot\mid s,a)$6, $Q_{H_t}(\cdot\mid s,a)$7, or $Q_{H_t}(\cdot\mid s,a)$8, which is closer to supervisory control than to standard primitive-action MDP semantics.

A different event-centric semantics appears in “Event-Driven Models,” where an event-driven model is an oriented graph $Q_{H_t}(\cdot\mid s,a)$9 with

$p^\diamond p^\bullet$0

and transition is given by an oracle

$p^\diamond p^\bullet$1

If the event set is empty, the state remains unchanged; actions are treated as a special case of events (Dobrev, 2019).

4. Bellman structures, value recursion, and policy classes

When events or exogenous processes alter the information pattern, the Bellman structure usually survives only after reformulation.

For MDPs under external temporal processes, standard discounted MDP theory applies on the augmented space. The main structural result is that the augmented MDP admits a deterministic stationary optimal policy and optimal value function under the paper’s assumptions (Ayyagari et al., 2023). The associated finite-history approximation is central: if the influence of old events decays summably through sequences $p^\diamond p^\bullet$2 and $p^\diamond p^\bullet$3, then for any $p^\diamond p^\bullet$4 there exists a horizon $p^\diamond p^\bullet$5 and a policy depending only on the current state and the past $p^\diamond p^\bullet$6 events that is $p^\diamond p^\bullet$7-optimal. The approximation error is controlled by the tail

$p^\diamond p^\bullet$8

In the sequentially observed transition model, the Bellman recursion remains finite-horizon but the control variable is an acceptance matrix $p^\diamond p^\bullet$9, not a standard action distribution (Chamie et al., 2015). A key closed-form structure is

$(o_{i,k},m_{i,k},\Delta t_{i,k},r_{i,k})$0

with statewise backward induction over within-stage accept/reject decisions. The nonlinearity in the acceptance variables is handled by a change of variables leading to an offline linear program at each state and time.

The delayed-execution model establishes a different structural departure from classical MDP results. With execution delay $(o_{i,k},m_{i,k},\Delta t_{i,k},r_{i,k})$1, stationary Markov policies are sub-optimal in general, but deterministic Markov policies in the original state space remain sufficient if they are allowed to be non-stationary (Derman et al., 2021). This is a timing-driven result rather than a purely event-semantic one, but it is closely related to asynchronous and pipeline-based event-driven systems. The paper proves that the brute-force augmented-state baseline $(o_{i,k},m_{i,k},\Delta t_{i,k},r_{i,k})$2 suffers exponential complexity in $(o_{i,k},m_{i,k},\Delta t_{i,k},r_{i,k})$3, while optimality can still be attained by non-stationary Markov policies on the original state space.

In PEC-MDP translation, the model is finite-horizon and time-indexed. The policy is explicitly non-stationary: $(o_{i,k},m_{i,k},\Delta t_{i,k},r_{i,k})$4 derived from p-propositions. Temporal projection then proceeds through the policy-weighted transition matrices

$(o_{i,k},m_{i,k},\Delta t_{i,k},r_{i,k})$5

followed by a query projection onto states satisfying the target fluent condition (Xu et al., 17 Jul 2025).

These variants clarify a frequent misconception: event-driven formulations do not necessarily replace dynamic programming or Markov control theory. More commonly, they alter the state, action, or policy class so that a Bellman principle can be restored.

5. Learning and algorithmic methods

The algorithmic literature reflects the same heterogeneity as the modeling literature.

In PCMDPs, exploiting exogenous structure yields sharp learning gains. Exogenous-Aware Value Iteration (ExAVI) estimates only the unknown exogenous kernel $(o_{i,k},m_{i,k},\Delta t_{i,k},r_{i,k})$6, combines it with known $(o_{i,k},m_{i,k},\Delta t_{i,k},r_{i,k})$7, and performs planning in the induced model (Maran et al., 3 Mar 2026). Its main regret guarantee is

$(o_{i,k},m_{i,k},\Delta t_{i,k},r_{i,k})$8

Exogenous-Aware Q-Learning (ExAQ) integrates over known endogenous dynamics and reuses each observed exogenous transition across all controllable state-action pairs sharing the same exogenous context. Its main regret guarantee is

$(o_{i,k},m_{i,k},\Delta t_{i,k},r_{i,k})$9

and the lower bound

$\mu(a,s,t)$0

shows the dependence on $\mu(a,s,t)$1 is unavoidable (Maran et al., 3 Mar 2026).

For asynchronous multi-agent macro-action control, the main algorithmic contribution is Modified Generalized Advantage Estimation (M-GAE) combined with PS-TRPO (Menda et al., 2017). The TD residual is changed from step-based discounting to elapsed-time discounting: $\mu(a,s,t)$2 and the advantage estimator becomes

$\mu(a,s,t)$3

This directly reflects stochastic macro-action duration and event-indexed trajectories.

In delayed environments, Delayed-Q combines future-state prediction with execution-time-aligned Q-learning rather than relying on state augmentation (Derman et al., 2021). The current action is chosen by predicting the state $\mu(a,s,t)$4 steps ahead under the queue of pending actions and applying a Q-policy there. Training shifts replay tuples so that each committed action is paired with the state in which it actually executes.

DES-augmented RL uses a hybrid architecture in which a Discrete Event Supervisory model operates alongside an RL agent, with event observations appended to a history string and belief states updated from $\mu(a,s,t)$5 (Dony, 28 Feb 2025). The paper frames the approach as integrating event-based supervisory insights with RL adaptability, not as defining a formal semi-Markov or event-triggered MDP.

Approximate dynamic programming also appears in event-response infrastructure models. Distribution-system resilience under unfolding extreme events is modeled as a finite-horizon MDP with post-decision states and iterative ADP (Wang et al., 2019), while wildfire resilience under decision-dependent uncertainty is formulated as a distributionally robust MDP using post-decision states and a linear value approximation $\mu(a,s,t)$6 (Zhao et al., 8 Apr 2026).

6. Applications, controversies, and open questions

The application domains represented in the literature are broad. Exogenous-dynamics MDPs are validated in modified Taxi with Traffic, optimal execution and trading, and an elevator environment (Maran et al., 3 Mar 2026). Asynchronous event-driven multi-agent control is demonstrated in real-time bus holding control and wildfire fighting with unmanned aircraft (Menda et al., 2017). DES-augmented RL targets industrial automation and intelligent traffic systems, with a motivating state-avoidance example involving a small discrete event system (Dony, 28 Feb 2025). Event-responsive MDPs are used for distribution-system resilience during typhoons and other unfolding extreme events (Wang et al., 2019), and for wildfire-aware grid reconfiguration under decision-dependent uncertainty in 54-bus and 138-bus systems (Zhao et al., 8 Apr 2026). Event-calculus translation is motivated by interpretable narrative reasoning and planning, including logistics-style demonstrations (Xu et al., 17 Jul 2025).

Several controversies and common confusions recur across the sources. One is the conflation of event-driven timing with event-conditioned transitions. A model may be event-centric without being event-triggered in the semi-Markov sense. This is stated explicitly for sequentially observed transitions (Chamie et al., 2015), external-temporal-process MDPs (Ayyagari et al., 2023), exogenous-dynamics PCMDPs (Maran et al., 3 Mar 2026), and PEC-MDPs (Xu et al., 17 Jul 2025), all of which remain discrete-time and stage-based. Another is the assumption that any event-driven formulation requires partial observability. Some models do involve belief states or partial observations, notably DES-augmented RL (Dony, 28 Feb 2025) and Dec-POMDP-style macro-action control (Menda et al., 2017), but others assume full observability of both endogenous and exogenous components (Maran et al., 3 Mar 2026).

A third issue concerns what events are allowed to be. In “Event-Driven Models,” an event is broadly a Boolean function that may be visible, semi-visible, or invisible, and actions are themselves events (Dobrev, 2019). In supervisory-control hybrids, events may be observable or uncontrollable (Dony, 28 Feb 2025). In temporal-process MDPs, events are marked exogenous perturbations with history-dependent effects (Ayyagari et al., 2023). In PEC, events are encoded through action occurrence and probabilistic effects in narrative logic (Xu et al., 17 Jul 2025). The notion is therefore semantically broader than “external shock.”

Open questions follow naturally from these divergences. The supplied literature repeatedly identifies gaps in handling approximate factorization and unknown endogenous dynamics in PCMDPs (Maran et al., 3 Mar 2026), stochastic or varying delays beyond fixed-delay theory (Derman et al., 2021), stronger formal equivalence and scalability analyses for event-calculus translation (Xu et al., 17 Jul 2025), and richer continuous-time or asynchronous event models beyond discrete-time approximations (Ayyagari et al., 2023, Zhao et al., 8 Apr 2026). A plausible implication is that future work will continue to split along two directions: one toward more faithful event-time models with asynchronous decisions, and another toward structured discrete-time MDPs that absorb event processes into state, policy, or transition factorizations.

In current usage, the most defensible encyclopedic characterization is therefore conditional rather than singular: an event-driven Markov decision process is a Markov decision formulation in which event structure is not peripheral but explicitly enters the model through timing, state augmentation, exogenous dynamics, transition semantics, supervisory constraints, or narrative action representation. The precise mathematics depends on which of those roles events are required to play.