Process Supervision with MDPs

• Process supervision with MDPs is defined as integrating supervisory feedback into standard MDPs, enabling enhanced observability, constraint enforcement, and decision making under uncertainty.
• Sequentially-Observed MDPs and temporal-logic-based supervisor synthesis expand policy spaces, offering higher performance and safety benefits compared to traditional MDP approaches.
• The article details practical implementations including counterexample-guided learning, linear programming for state constraints, and rolling-horizon control for real-time policy improvement in complex systems.

Process supervision with Markov Decision Processes (MDP) refers to the systematic deployment of MDP frameworks and their extensions to monitor, control, and optimize stochastic dynamical systems, ensuring compliance with operational constraints, maximizing expected performance, or enforcing safety and correctness under uncertainty. This article surveys the principal MDP formalisms and algorithms that explicitly incorporate supervision—either via model extensions, algorithmic overlays, or through interaction with external supervisory processes—covering theory, synthesis, and applications in automated process control, verification, multi-agent systems, and reinforcement learning.

1. Supervised Markov Decision Process Fundamentals

An MDP is a tuple $(S,A,P,R,\gamma)$ where $S$ is a finite state space, $A$ is the finite action set, $P(s'|s,a)$ specifies transitions, $R(s,a)$ is the reward, and $\gamma\in[0,1)$ is the discount factor. Process supervision augments the standard MDP setting by one or more mechanisms:

• Extending the information structure accessible to the controller (e.g., sequential next-state peeking in SMDPs)
• Constraining policy or system behavior (e.g., via state/action constraints or temporal-logic specifications)
• Introducing a meta-process that intermittently supplies feedback, observations, or corrections (e.g., a human supervisor, monitoring component, or external policy oracle)
• Direct algorithmic integration of supervisor feedback during learning or policy updating

Each variant modifies the Markovian decision framework to encode richer operational semantics while preserving algorithmic tractability for analysis and synthesis.

2. Sequentially-Observed MDPs and Enhanced Policy Spaces

The SMDP (Sequentially-observed-horizon MDP) generalizes the standard MDP by allowing, at each step, sequential, irrevocable observation of the random transitions resulting from a structured sequence of candidate actions $a_1,\ldots,a_m$. The process proceeds in ordered phases: the controller observes the specific realization of each action's outcome (the next-state $j$ according to $G_s(j,k)$), and selectively accepts or rejects each until forced to accept at phase $m$.

The action-selection policy in SMDPs is therefore contingent on realization-specific information and is characterized by the set of acceptance probabilities $S$0 at epoch $S$1 in state $S$2 for realization $S$3 under action $S$4. The feasible policy set in SMDPs strictly contains that of the standard MDP, potentially achieving strictly higher value:

$S$5

This policy class expansion often provides substantial performance gains in process-supervision scenarios involving phased inspections, sequential resource allocation, or contingency-driven actuation—without further convexity or monotonicity assumptions (Chamie et al., 2015).

3. Supervisor Synthesis and Temporal-Logic-Constrained MDPs

In the supervisor synthesis paradigm, as formalized for multi-agent MDP systems, a supervisor is typically a deterministic finite automaton (DFA) that disables disallowed actions or paths. The permissive supervisor synthesis problem aims to compute, for each agent MDP $S$6, the most permissive local supervisor $S$7 such that the globally composed system $S$8 satisfies a given probabilistic temporal property, often in time-bounded PCTL form (e.g., $S$9).

The synthesis algorithm iteratively:

1. Uses assume–guarantee compositional model checking to avoid full state-space explosion.
2. Refines local supervisors via counterexample-guided learning, employing modified L* automata learning.
3. Terminates with correctness guarantees (no further counterexamples) and maximal permissiveness (local DFAs allow every trace not strictly necessary to prohibit for safety) (Wu et al., 2017).

The convergence is finite, with complexity polynomial in local subsystem sizes and temporal formula length—exploiting counterexample extraction, compositional verification, and language refinement.

4. State-Constrained MDPs for Safety-Bounded Supervision

In numerous process-supervision tasks, operational and safety requirements are formalized as state constraints. The finite-horizon CMDP framework encodes constraints as linear inequalities on the state-occupancy vectors $A$0 at all time steps:

$A$1

CMDPs seek policies (usually randomized, non-stationary) maximizing expected reward while strictly adhering to these constraints. The solution is derived via a sequence of linear programs (LPs) over occupancy variables $A$2, ensuring both dynamic consistency and safety feasibility for every $A$3. Dual variables have a shadow-price interpretation, quantifying the value of slackening/tightening particular bounds (Chamie et al., 2015).

Algorithms for CMDPs involve backward induction with per-timestep LPs, fast primal/dual solvers, and optional projection of unconstrained MDP solutions onto feasible convex sets. The practical efficacy is demonstrated, for instance, in chemical batch-reactor control, maintaining risk probabilities under prescribed levels and incurring minimal performance degradation relative to unconstrained solutions.

5. Monitored and Partially-Observable Reward Supervision

Monitored MDPs (Mon-MDPs) explicitly separate the reward-generation process from the observation channel: the agent interacts with a joint system comprising the environment MDP and a monitor MDP, which decides when—and at what cost or risk—the agent observes reward feedback. The observed reward at each step may be missing ($A$4), incurring distinctive challenges in policy evaluation, value estimation, and convergence.

Several algorithms have been proposed for Mon-MDPs:

• "Assign Constant Zero": treat missing rewards as zero (statistically unsound).
• "Ignore Updates": skip updates when no reward is observed.
• "Reward-Model Q-Learning": maintain per-state/action mean estimators for true rewards from observed samples, impute these for missing values, and learn on the joint state/action space.

Convergence to optimality with reward-model Q-learning is ensured under ergodicity and truthful monitor assumptions. Mon-MDPs naturally generalize settings in active RL, human-in-the-loop supervision, partial monitoring, and expensive or intermittent sensing, offering a unifying analysis framework for process-supervised RL (Parisi et al., 2024).

6. Rolling-Horizon Control and On-Line Supervised Policy Improvement

For infinite-horizon control, rolling-horizon (receding-horizon) algorithms with explicit supervisor feedback are deployed. At each step, an $A$5-step "forecast" policy is maintained and improved only at the currently visited state via policy iteration and policy-switching. External supervisor input is admitted as H-length advice or as immediate recommended actions.

Key properties include:

• Asynchronous updates (at the current state only)
• Monotonic improvement in the forecast-horizon value function
• Provable finite-time convergence to a globally optimal $A$6-step policy under communicative MDPs, or to locally optimal policies otherwise

Supervisor feedback is seamlessly integrated, influencing the candidate policy set for policy-switching without sacrificing the theoretical guarantees of monotonic improvement and termination (Chang, 2022).

7. Mapping Practical Process Supervision Problems to MDP Frameworks

Process supervision scenarios routinely mapped to these MDP-based frameworks include:

• Sequential Quality Control: Production lines where sequential testing (ordered by cost, informativeness, or risk) is naturally represented via SMDPs, exploiting the sequential observation structure for anticipated outcomes (Chamie et al., 2015).
• Safety-Critical Resource Management: Systems (chemical reactors, vehicle fleets) with strict probabilistic safety constraints (e.g., maximum allowed probability in dangerous states), formulated and solved via finite-horizon CMDPs (Chamie et al., 2015).
• Distributed Multi-Agent Networks: Communication, transportation, or robotic clusters with temporal-logic supervision needs, addressed through compositional, permissive supervisor synthesis (Wu et al., 2017).
• Human-in-the-Loop/Rare Feedback Environments: RL or automation contexts where rewards are available only through costly or sparse monitoring actions, directly modeled as Mon-MDPs (Parisi et al., 2024).
• Adaptive, On-the-Fly Optimization: High-dimensional or unmodeled systems benefit from rolling-horizon supervised online algorithms, integrating supervisory advisory policies for globally consistent and practical convergence (Chang, 2022).

These mappings and frameworks support rigorous, performance-guaranteed, and constraint-satisfying supervision across a spectrum of stochastic process control challenges.