Transition-Markov Decision Processes

• Transition-Markov Decision Processes (TMDPs) are a family of MDP extensions that treat transition laws as dynamic and uncertain objects rather than fixed kernels.
• They incorporate time-varying, exogenously influenced, and even adversarial transition structures to capture real-world complexities in decision-making.
• TMDPs are applied in domains like robot navigation, marine robotics, and robust control, driving the development of specialized algorithms to handle transition uncertainty.

Transition-Markov Decision Processes (TMDPs), in a transition-centric reading of the MDP literature, denote models in which transition laws are themselves the primary objects of specification, uncertainty, observation, or control. The label is not standardized. One paper uses TMDP to mean a topological Markov decision process, a multi-objective MDP with a directed acyclic graph over objectives and slack variables (Wray et al., 2022). Another uses TMDP for a Threatened Markov Decision Process, where an adversary acts through state transitions and rewards (Gallego et al., 2019). A broader transition-centered interpretation therefore treats TMDPs not as a single canonical formalism but as a family of MDP extensions in which the kernel $P(\cdot\mid s,a)$ is time-varying, uncertain, exogenously modulated, observed before commitment, or optimized under structural constraints.

1. Terminological scope and acronym ambiguity

The standard MDP recalled in the literature is the tuple $\langle S,A,T,R\rangle$ or its notational variants, with a fixed transition model and scalar reward. Transition-centered generalizations modify that baseline by enriching the transition component rather than merely replacing the reward with a different scalarization. In this sense, several distinct research programs fall under a plausible transition-centric umbrella, even when they do not explicitly use the phrase “Transition-Markov Decision Process.”

The acronym itself is overloaded. In “Multi-Objective Policy Gradients with Topological Constraints,” a TMDP is defined as $\langle S,A,T,\mathbf R,E,\boldsymbol\delta\rangle$, where $\mathbf R$ is vector-valued, $E$ is a directed acyclic graph over objectives, and $\boldsymbol\delta$ are slack variables controlling degradation of ancestor objectives (Wray et al., 2022). In “Opponent Aware Reinforcement Learning,” a TMDP is $$(\mathcal S,\mathcal A,\mathcal B,\mathcal T,R,p_A)$$, where $\mathcal B$ is an adversary action space, $$\mathcal T:\mathcal S\times\mathcal A\times\mathcal B\to\Delta(\mathcal S)$$, $$R:\mathcal S\times\mathcal A\times\mathcal B\to\Delta(\mathbb R)$$, and $\langle S,A,T,R\rangle$0 is the decision maker’s belief over threats (Gallego et al., 2019).

This ambiguity matters because “TMDP” is not a universally accepted technical name for one object. A common misconception is to treat the acronym as settled terminology. The literature instead supports a narrower statement: multiple non-equivalent models place transition structure at the center, and two of them explicitly reuse the same acronym for different purposes.

2. Transition laws as dynamic, exogenous, and adversarial objects

One major line of work studies time-varying transition kernels. “A Solution to Time-Varying Markov Decision Processes” defines $\langle S,A,T,R\rangle$1, with $\langle S,A,T,R\rangle$2 and $\langle S,A,T,R\rangle$3 depending on physical time rather than only on the decision epoch. The model augments Bellman propagation with transition-time estimation $\langle S,A,T,R\rangle$4, so future values are evaluated under the transition kernel appropriate to the estimated arrival time. The resulting update,

$\langle S,A,T,R\rangle$5

is explicitly designed for spatiotemporal environments such as ocean-current navigation (Liu et al., 2016).

A second line treats transitions as modulated by exogenous temporal processes. “Markov Decision Processes under External Temporal Processes” starts from $\langle S,A,T,R\rangle$6 and replaces the stationary kernel $\langle S,A,T,R\rangle$7 by a history-conditioned kernel $\langle S,A,T,R\rangle$8, where $\langle S,A,T,R\rangle$9 is a history of marked external events. The process on $\langle S,A,T,\mathbf R,E,\boldsymbol\delta\rangle$0 alone is not Markov, but the augmented state $\langle S,A,T,\mathbf R,E,\boldsymbol\delta\rangle$1 is Markov. Under summably decaying influence of old events, the paper proves that policies depending only on a finite event window can approximate the optimal augmented-state policy arbitrarily well (Ayyagari et al., 2023).

A third line makes transitions adversarially time-varying. “Online Learning in Markov Decision Processes with Adversarially Chosen Transition Probability Distributions” considers a finite-state, finite-action online game with time-indexed kernels $\langle S,A,T,\mathbf R,E,\boldsymbol\delta\rangle$2 and losses $\langle S,A,T,\mathbf R,E,\boldsymbol\delta\rangle$3. The core difficulty is that changing policies and changing kernels both perturb state occupancies. Under a uniform mixing condition, the paper proves $\langle S,A,T,\mathbf R,E,\boldsymbol\delta\rangle$4-type regret for the low-switching SD-MDP algorithm against a fixed stationary comparator policy (Abbasi-Yadkori et al., 2013).

These formulations share a common structural move: the transition law ceases to be a stationary background assumption and becomes an evolving object indexed by physical time, event history, or adversarial round.

3. Uncertain transitions, robustness, and verification

A distinct body of work treats TMDP-like models as MDPs with uncertain transition parameters. “Multi-Objective Approaches to Markov Decision Processes with Uncertain Transition Parameters” uses bounded-parameter MDPs (BMDPs) and stochastic bounded-parameter MDPs (SBMDPs), where transition rows and rewards lie in intervals. Instead of optimizing only the worst case, the paper evaluates each pure stationary policy by its worst-case, average-case, and best-case discounted value vectors, and computes the pure Pareto-optimal set in that value-vector space. This reframes transition uncertainty as a multi-scenario, multi-objective optimization problem rather than a purely pessimistic robust one (Scheftelowitsch et al., 2017).

A more general robust formulation appears in “Transition Uncertainties in Constrained Markov Decision Models: A Robust Optimization Approach.” There the true transition law is modeled as $\langle S,A,T,\mathbf R,E,\boldsymbol\delta\rangle$5, with deviations $\langle S,A,T,\mathbf R,E,\boldsymbol\delta\rangle$6 constrained by polyhedral and second-order cone conditions. The paper studies a discounted infinite-horizon robust CMDP, shows that each inner worst-case transition problem can be transformed into an SOCP, and then uses strong duality to obtain an equivalent finite reformulation with bilinear and SOC constraints. The result is exact but nonconvex, and the numerical study emphasizes how the shape of the uncertainty set—not only its size—changes optimal values and policies (Varagapriya, 15 Mar 2025).

Robust verification under time-varying transition uncertainty is developed in “Data-Driven Robust Safety Verification for Markov Decision Processes.” For a finite-horizon time-inhomogeneous MDP with fixed policy $\langle S,A,T,\mathbf R,E,\boldsymbol\delta\rangle$7, the paper constructs Wasserstein ambiguity sets around empirical transition distributions obtained from multiple executions, interprets the result as an interval-MDP representation, and defines the robust safety function

$\langle S,A,T,\mathbf R,E,\boldsymbol\delta\rangle$8

A high-confidence bound shows that the empirical robust safety value upper-bounds the true unknown safety value with probability at least $\langle S,A,T,\mathbf R,E,\boldsymbol\delta\rangle$9 (Mazumdar et al., 8 Dec 2025).

Taken together, these papers make transition uncertainty a first-class modeling primitive. The common mathematical pattern is a min–max or multi-scenario optimization over admissible kernels, often under rectangular or state-wise decomposable ambiguity.

4. Transition observability, belief dynamics, and inferability

Another transition-centered strand studies what happens when transitions are partially observed, fully observed before commitment, or deliberately obscured.

“Finite-Horizon Markov Decision Processes with Sequentially-Observed Transitions” extends a finite-horizon MDP by allowing the controller to inspect candidate transition outcomes action by action before choosing whether to commit. At phase $\mathbf R$0, the controller observes the realized next state that would result from action $\mathbf R$1, accepts or rejects it, and if all early options are rejected must take the last action unobserved. Policies are therefore functions of current state, time, phase, and observed candidate next state, encoded through acceptance probabilities $\mathbf R$2. The paper derives a Bellman recursion whose per-state update is transformed from a nonconvex program into a linear program (Chamie et al., 2015).

Belief-state constructions provide a different transition-centered mechanism. “Markov Decision Processes with Incomplete Information and Semi-Uniform Feller Transition Probabilities” studies an MDPII $\mathbf R$3, where $\mathbf R$4 is hidden and $\mathbf R$5 is observed, and shows that the reduction to a completely observable belief-state MDP on $\mathbf R$6 preserves semi-uniform Feller continuity under precise conditions. The induced belief transition kernel

$\mathbf R$7

inherits the regularity needed for optimality equations, value iteration, and existence of optimal policies (Feinberg et al., 2021).

“Least Inferable Policies for Markov Decision Processes” turns induced transitions into the protected object. For a stationary policy, the observer tries to infer the policy-induced transition probabilities

$\mathbf R$8

The paper defines a Fisher-information-based transition information metric

$\mathbf R$9

and minimizes expected total information under a reachability constraint via convex optimization over state-action residence times (Karabag et al., 2018).

These models emphasize that transition structure is not only something to optimize under uncertainty; it can also be something to observe, transform into a belief kernel, or hide from an observer.

5. Algorithmic paradigms for transition-centered models

The algorithmic landscape is correspondingly heterogeneous. In the topological-MDP line, “Multi-Objective Policy Gradients with Topological Constraints” derives a policy-gradient theorem in which the ordinary action-value is replaced by a Lagrangian action-value incorporating ancestor-preservation penalties. The resulting topological policy optimization (TPO) algorithm extends PPO by computing a generalized Lagrangian advantage and traversing the objective DAG in topological order (Wray et al., 2022).

Information-regularized transition models yield a different computational pattern. “Transfer-Entropy-Regularized Markov Decision Processes” minimizes

$E$0

proves that it is sufficient to consider policies of the form $E$1, and derives a coupled system of forward occupancy recursions and backward soft-value equations. The proposed algorithm is a forward-backward procedure analogous to Arimoto–Blahut, and every limit point is a stationary point of the TERMDP objective (Tanaka et al., 2017).

Transition-constrained Bayesian optimization introduces yet another planning template. “Transition Constrained Bayesian Optimization via Markov Decision Processes” encodes feasible successor queries by $E$2 and occupancy measures $E$3. The utility is a convex function of the visitation distribution, and Frank–Wolfe reduces each planning step to a linearized RL subproblem over the MDP occupancy polytope. Because posterior updates are folded back into the planning objective, the resulting policy is explicitly described as potentially history-dependent and non-Markovian (Folch et al., 2024).

A representation-oriented algorithmic perspective appears in “A Tensor Network Approach to Finite Markov Decision Processes,” which encodes the finite episodic kernel $E$4 as local transition tensors and expresses expected return as a tensor-network contraction. Policy optimization is then performed by DMRG-style local tensor updates, with a backward sweep playing a role analogous to finite-horizon dynamic programming (Gillman et al., 2020).

A plausible implication is that there is no single “TMDP algorithm.” Transition-centered models give rise to policy gradients, robust conic reformulations, low-switch online learning, occupancy-measure optimization, belief-state dynamic programming, and tensor-network contractions, depending on which aspect of the transition law is being emphasized.

6. Application domains, conceptual reach, and open issues

Transition-centered MDP formulations have been used across markedly different domains. Topological MDPs are evaluated on multi-objective robot navigation in simulation and on a real household robot (Wray et al., 2022). TVMDPs are developed for marine robotics under spatiotemporal ocean currents (Liu et al., 2016). External-process MDPs are demonstrated on a non-stationary Pendulum-v1 control problem (Ayyagari et al., 2023). Dynamic-transition MDPs model basketball possessions with shot-clock-dependent kernels and hierarchical Bayesian transition tensors (Sandholtz et al., 2018). Robust transition-uncertain CMDPs are studied on machine replacement (Varagapriya, 15 Mar 2025). Transition-constrained Bayesian optimization is applied to chemical reactor optimization, informative path planning, and machine calibration (Folch et al., 2024).

Several limitations recur. Some frameworks remain primarily theoretical, with convergence analysis or broad benchmarking left open, as in the topological policy-gradient work (Wray et al., 2022). In adversarially varying transitions, efficient low-regret learning beyond the uniform-mixing setting remains open (Abbasi-Yadkori et al., 2013). In external-process and TERMDP models, finite-memory approximations mitigate but do not remove the scaling burden of history augmentation (Ayyagari et al., 2023, Tanaka et al., 2017). Robust transition-uncertain CMDPs admit exact reformulations, but the final optimization problems are nonconvex and empirically harder as the state space grows (Varagapriya, 15 Mar 2025). Data-driven robust safety verification provides guarantees for a fixed policy, not full robust synthesis (Mazumdar et al., 8 Dec 2025).

The literature therefore supports a restrained conclusion. “Transition-Markov Decision Processes” is best treated as an umbrella description for transition-centered MDP extensions rather than a single universally accepted formalism. The unifying idea is that transition laws are no longer passive background objects: they become time-indexed, uncertain, exogenously driven, sequentially observed, adversarially perturbed, information-bearing, or structurally constrained, and the mathematics of planning must be reorganized around that fact.