Hybrid Markov Decision Process (HMDP)

• Hybrid Markov Decision Processes are mathematical models combining discrete modes, continuous state dynamics, and probabilistic events to model complex systems.
• They use computational frameworks like HALP, hybrid-DDP, and abstraction techniques to synthesize safe, cost-optimal strategies and enable scalable approximations.
• HMDPs have practical applications in embedded controllers, autonomous vehicles, robotics, and multimodal routing, with ongoing research addressing scalability and optimality challenges.

A Hybrid Markov Decision Process (HMDP) is a mathematical model that generalizes classical Markov Decision Processes by integrating both discrete modes and continuous state variables, often in conjunction with stochastic or dynamic behaviors of external environments. HMDPs provide a rigorous framework for sequential decision-making in domains such as embedded controllers, autonomous vehicles, robotics, and multi-agent systems where system evolution depends simultaneously on discrete choices, continuous dynamics, and probabilistic events. Key research contributions have focused on formal model definition, safety and optimality constraints, computational methodology for strategy synthesis, scalable approximations, and compact policy representation (Ashok et al., 2019, Guestrin et al., 2012, Guestrin et al., 2011, Wang et al., 2024, Choudhury et al., 2019, Pajarinen et al., 2017).

1. Formal Definition and Structure

An HMDP is characterized by a state space that is the Cartesian product of discrete controllable modes, uncontrollable (environment) modes, and a vector of continuous real-valued state variables:

$\mathcal{H} = (C,\,U,\,X,\,F,\,\delta)$

• $C = \{c_1, \dots, c_k\}$: Discrete controllable modes or states.
• $U = \{u_1, \dots, u_\ell\}$: Discrete environment or uncontrollable modes.
• $X = (x_1, \dots, x_n)$: Continuous state variables.
• For each pair $(c, u) \in C \times U$: A flow map

$$F_{(c, u)}: \R_{\ge 0} \times \R^X \longrightarrow \R^X$$

governs the evolution via $\dot{x} = f(c, u, x)$.

• The transition kernel $\delta_\gamma$ assigns a probability distribution on $U$ for each configuration $\gamma = (c, u, x)$.

A global transition with fixed period $P$ comprises: (i) continuous flow evolution, (ii) probabilistic jump in environment mode, and (iii) controller action selection. The overall state space is infinite-dimensional:

$S = C \times U \times \R^X$

Action sets can be discrete, continuous, or hybrid; controllers can select single actions or permissive sets in each configuration.

2. Safety and Cost-Optimality Objectives

Safety is formulated as the preservation of state trajectories within a designated safe set:

$S_{\text{safe}} \subseteq S$

A strategy $\sigma: S \to 2^C$ is sure-safe if all induced system runs remain entirely within $S_{\text{safe}}$. Temporal logic formulas (e.g., TCTL: $A\,\Box\,\mathit{in\_safe}$) are used for specification.

The cost objective is defined over a finite or infinite horizon $H$, with a stage cost function $c(\gamma, a)$ or an instantaneous cost $\ell(x(t))$:

$$D(\pi) = \sum_{i=0}^{H-1} c(\gamma_i, a_i) \quad \text{or} \quad D(\pi) = \int_{0}^{H P} \ell(x(t))\,dt$$

The expected cumulative cost under policy $\sigma$ measures performance:

$$E_{\sigma,H}^{\mathcal{H},\gamma}[D] = \mathbb{E}_{\pi \sim (\mathcal{H} \upharpoonright \sigma, \gamma)}[D(\pi)]$$

Optimization seeks a sub-strategy $\sigma^*$ that is safe and (deterministically) minimizes this expectation (Ashok et al., 2019, Wang et al., 2024).

3. Computational Frameworks and Solution Algorithms

Multiple frameworks have emerged addressing computational challenges:

• Abstraction and Synthesis: Discretize continuous space; abstract to timed games $TG$ for synthesis (UPPAAL TIGA) of sure-safe permissive strategies.
• Cost-Optimal Learning: Restrict to safe strategies, solve for optimal policy via value iteration or reinforcement learning methods.
• Hybrid Approximate Linear Programming (HALP): Approximate the value function by linear combinations of factored basis functions, optimize weights via LP constrained by Bellman-type inequalities. Integrations with hybrid dynamic Bayesian networks facilitate closed-form computation for transition expectations, leveraging problem structure to reduce complexity (Guestrin et al., 2012, Guestrin et al., 2011).
• Hybrid Stochastic Planning (HSP): Combines open-loop discrete-mode planning with local continuous-state MDP solvers and hierarchical interleaving for dynamic contexts (e.g., multimodal routing) (Choudhury et al., 2019).
• Hybrid-Differential Dynamic Programming (Hybrid-DDP): Convexifies the discrete-continuous control selection, anneals solutions toward pure discrete actions, supporting partial observability in hybrid POMDP regimes (Pajarinen et al., 2017).

4. Strategy Synthesis and Compact Representation

Traditional optimal safe strategies in HMDPs are massive lookup tables ($|S|$ explicit entries). "SOS" (Ashok et al., 2019) introduces CART-style decision tree compression techniques for multi-label strategies:

• Features: Discrete mode, environment mode, integer parts of clocks/continuous variables.
• Labels: Allowed action sets per configuration.
• Learning and pruning parameters ($k$: minimum split size, $p$: safe-pruning rounds) yield a Pareto frontier between policy size and optimality loss.
• Safety is preserved at every compression step; cost suboptimality bound is quantified by $\Delta(k, p)$:

$$E_{\sigma_{\mathcal{T}},H}[D] \leq E_{\sigma^*,H}[D] + \Delta(k,p)$$

with $\Delta(k,p) \to 0$ as pruning is reduced. In practical cases, millions of explicit strategy entries are reduced to thousands of tree nodes with negligible loss.

5. Scalable Approximation and Theoretical Guarantees

Factored discretization of continuous variables enables tractable approximate LP formulations of HMDPs:

• Constraints and reward factors typically depend on small variable subsets ("scope"), permitting grid spacings that ensure $\epsilon$-infeasibility and manageable computational effort.
• Error bounds on HALP approximations are proportional to the best possible linear value function approximation plus discretization error.

$$\|V^*-V_w\|_{1,\psi} \leq 2\psi(L)\frac{1}{1-\kappa} \min_w \|V^*-V_w\|_{\infty,1/L} + \frac{2\epsilon}{1-\gamma}$$

For MPC-embedded HMDPs, recursive feasibility and closed-loop stability are proven: truncated finite-horizon policies with a baseline terminal cost sustain feasibility and guarantee monotonic decrease of cost-to-go until goal attainment (Wang et al., 2024).

6. Applied Domains and Illustrative Examples

HMDPs have been adopted in various high-impact domains:

• Embedded controllers and autonomous driving: Adaptive cruise control and lane-change systems, combining discrete maneuver logic with continuous motion dynamics (Ashok et al., 2019, Wang et al., 2024).
• Multimodal stochastic path planning: DREAMR problem for autonomous routing; hybrid planning yields superior cost and energy-efficiency in dynamic, context-rich environments (Choudhury et al., 2019).
• Robotics and manipulation: Hybrid-DDP applied to gear selection in driving, box pushing under uncertainty; efficiently navigates the exponential action-sequence search space (Pajarinen et al., 2017).
• Irrigation network control: HALP frameworks scale to 17–28 dimensions in continuous state, showcasing the impact of factored representations (Guestrin et al., 2012, Guestrin et al., 2011).

7. Current Challenges, Limitations, and Research Directions

Hybrid mode/continuous-state MDP planning remains fundamentally hard (generally NP-hard in the presence of stochastic dynamic contexts, as in DMSSP formulation (Choudhury et al., 2019)). No global optimality guarantees are proven except in constrained structures. Key technical limitations include:

• Exponential state-action space growth ("curse of dimensionality"), mitigated by factored representations and constraint generation.
• Practical scalability is limited to problems with modest treewidth or when local independence structures exist.
• Decision-tree compression, while retaining safety guarantees, admits a quantifiable but nonzero loss in cost-optimality.
• Ongoing work targets integration with MPC frameworks, extension to partially observable hybrid systems, and improved automated abstraction/refinement methodologies.

Recent research continues to pursue new compact representations, scalable approximations, and robust control integration for HMDPs, reflecting their centrality in formal decision-making under uncertainty and hybrid dynamics (Ashok et al., 2019, Wang et al., 2024, Guestrin et al., 2012, Guestrin et al., 2011, Choudhury et al., 2019, Pajarinen et al., 2017).