--- title: Finite-Horizon Episodic MDPs url: https://www.emergentmind.com/topics/finite-horizon-episodic-markov-decision-processes-mdps type: topic --- # Finite-Horizon Episodic MDPs A finite-horizon episodic Markov Decision Process (MDP) is a mathematical framework for modeling sequential decision problems where an agent interacts with an environment over a fixed, finite number of time steps, or "episodes." Each episode consists of a sequence of decisions, transitions, and rewards, and the agent's objective is to optimize an expected sum of rewards or costs, which may be subject to constraints. The finite horizon necessitates time-dependent (nonstationary) policies and special sample/algorithmic considerations distinct from infinite-horizon or continuing MDPs. ## 1. Formal Structure and Policy Classes A finite-horizon episodic MDP is described by the tuple $(S, A, H, (P_t)_{t=0}^{H-1}, (r_t)_{t=0}^{H-1}, p_0)$, where: - $S$: finite state space. - $A$: finite action space. - $H$: integer-valued horizon, i.e., the episode ends after $H$ steps. - $P_t(s'|s,a)$: time-(stage-)dependent transition kernels. - $r_t(s,a)$ (or $r_t(s,a,s')$): possibly time-dependent reward or cost. - $p_0$: initial state distribution. A nonstationary policy $\pi = (\pi_0, \ldots, \pi_{H-1})$, with each $\pi_t(a|s)$ a distribution over $A$, governs the agent's choices at each time step, potentially allowing randomized (stochastic) actions. Trajectories consist of $(s_0, a_0, r_0, \ldots, s_H)$, where each $s_{t+1} \sim P_t(\cdot|s_t,a_t)$. The value function at stage $t$ under policy $\pi$ from state $s$ is defined as $V_t^\pi(s) = \mathbb{E}^\pi\left[\sum_{k=t}^{H-1} r_k(s_k, a_k, s_{k+1}) \mid s_t=s \right]$, with optimal value $V_t^*(s) = \max_\pi V_t^\pi(s)$ [2112.10599]. ## 2. Dynamic Programming and Bellman Recursion The backbone of computation in finite-horizon episodic MDPs is backward dynamic programming (DP) using the stage-indexed Bellman equations: \[ Q_t^*(s,a) = r_t(s,a) + \sum_{s'} P_t(s'|s,a) V_{t+1}^*(s'), \] \[ V_t^*(s) = \max_{a \in A} Q_t^*(s,a). \] The finite horizon necessitates distinct (nonstationary) policies for each stage $t$, in contrast to the stationary structure of infinite-horizon discounted or average-reward MDPs [2112.10599, 1510.08906]. The optimal policy at each stage chooses actions maximizing $Q_t^*(s,a)$. ## 3. Sample Complexity and Regret: PAC-Learning, Lower and Upper Bounds Finite-horizon episodic MDPs have been central to the theory of sample complexity and regret minimization. Key results include: - **PAC (Probably Approximately Correct) sample complexity** for learning $\epsilon$-optimal policies scales as $\tilde{O}(\frac{|S|^2|A|H^2}{\epsilon^2}\log \frac{1}{\delta})$ in the general (tabular) case, given known rewards and unknown transitions. Matching lower bounds are $\tilde{\Omega}(\frac{|S||A|H^2}{\epsilon^2}\log \frac{1}{\delta + c})$ [1510.08906]. This quadratic dependence on the horizon $H$ is optimal up to logarithmic factors. - A crucial technical advance was the Bellman-variance analysis, improving the horizon-exponent from $H^3$ (typical for reductions to discounted MDPs or Hoeffding-style bounds) to $H^2$ via refined Bernstein/variance-matching concentration [1510.08906]. - For constrained MDPs ("CMDPs"), where policies must satisfy constraints on secondary costs, occupancy measure LPs yield sample complexity of $\tilde{O}(|S||A|C^2 H^2 / \epsilon^2)$ episodes, with $C$ the maximal possible number of successor states [2009.11348]. - For standard reward-only problems, recent non-constructive analyses establish the horizon-free upper bound $N=O(1)$ (for fixed $|S|,|A|,\epsilon,\delta$), i.e., the number of episodes to PAC-optimality need not grow with $H$, under a bounded-sum reward normalization and with computational intractability [2111.00633]. ## 4. Algorithmic Methodologies: LPs, Bilinear Programs, and Reinforcement Learning Multiple algorithmic frameworks are available for finite-horizon MDPs and their constrained/generalizations: - **Dynamic programming/value iteration** is efficient for moderate problem sizes, exploiting the backward-recursive Bellman structure [2112.10599, 1510.08906]. - **Occupancy measure LPs:** The set of feasible state-(action) occupancy sequences over the $H$ steps forms a polytope described by flow-conservation constraints. The optimal policy is encoded as the solution to an LP minimizing expected cost/reward, subject to constraints on expected secondary costs [2009.11348, 1510.08906]. - **Bilinear programming in CMDPs:** For CMDPs with both additive and multiplicative utilities (e.g., risk-sensitive or multiplicative cost criteria), model transformation (via augmented state variables) reduction turns the problem into a purely additive CMDP on an expanded state space, solvable via a finite-dimensional bilinear program. The augmented state doubles for each multiplicative component, but practical scenarios often keep this manageable ($2^{K+1}$ for $K$ multiplicative indices) [2303.07834]. | Problem Type | Formulation | Notes/Comments | |----------------------|------------------------------|----------------------------------------------------| | Unconstrained MDP | LP over occupancy measures | Flows, no constraints; solution gives policy | | Additive + multiplicative CMDP | Bilinear program (augmented state) | Polynomial in horizon, exponential in # mult. objectives [2303.07834] | | CMDP (constraints) | LP with cost constraints | Scales as $|S||A|C^2H^2$ (worst-case) episodes [2009.11348] | - **Reinforcement learning algorithms:** Finite-horizon Q-learning algorithms have been developed for this setting, utilizing nonstationary Q-functions $Q_t(s,a)$, with sample-based updates and convergence established by ODE-based stochastic approximation [2110.15093]. ## 5. Strategy and Memory Complexity Finite-horizon MDPs generally require policies that keep explicit track of elapsed time (or steps-to-go). Characterizing the memory requirements: - Any $\epsilon$-optimal policy for a fixed-horizon MDP with $n$ states can be implemented by a counter-based strategy with $O(n + \log \log(1/\epsilon))$ bits of memory. This is tight: there exist MDPs where $\Omega(n + \log\log(1/\epsilon))$ bits are required [1209.3617]. - For exactly optimal strategies, the period of counter-based implementations can be as large as $2^{\Omega(\sqrt{n \log n})}$ for some $n$-state MDPs, due to periodic dependence on the horizon modulo various primes [1209.3617]. ## 6. Extensions: Resource Constraints, Risk Sensitivity, and Advanced Utility Criteria Recent research has generalized finite-horizon episodic MDPs to accommodate a range of advanced objectives: - **Mean-variance optimization:** Formulated as a bilevel MDP with an augmented state (tracking accumulated returns), solved by alternating optimization over the pseudo-mean and a dynamic program over the remainder. The optimal policy may be history-dependent and the value function is piecewise quadratic-concave. This paradigm extends to multi-period mean-variance portfolio optimization, queueing control, and inventory management [2507.22327]. - **Additive and multiplicative constraints/objectives:** The use of binary auxiliary state variables allows for the incorporation of multiplicative utility components (e.g., risk-sensitive criteria, survival probabilities), transforming into a pure additive augmented CMDP [2303.07834]. - **Resource allocation:** Online learning frameworks based on dual mirror descent solve episodic finite-horizon CMDPs with unknown kernels and stochastic reward/resource consumption, attaining tight regret bounds in both observe-then-decide and decide-then-observe regimes [2305.10744]. ## 7. Connections, Complexity, and Theoretical Implications Finite-horizon episodic MDPs form a universal formalism encompassing a wide range of sequential stochastic optimization problems, from basic stochastic control, resource allocation, and inventory to learning under bandit feedback [2602.02260]. - Computational complexity is governed by the underlying structure: standard MDPs are polynomially tractable (in $|S|,|A|,H$) via DP; constrained and risk-sensitive variants may induce exponential complexity in the number of constraints or multiplicative utilities, but linear in horizon for modest $K$ [2303.07834]. - Sample complexity is optimally quadratic in the horizon for tabular RL under classical regret/PAC guarantees, but logarithmic or constant for certain horizon-normalized or computationally-inefficient models [1510.08906, 2111.00633]. - Algorithmic frameworks offer tractable solutions for small/medium-scale problems; novel structures (low-rank tensors [2501.10598], tensor networks [2002.05185], quantum algorithms [2508.05712]) provide scalable or computational advantages in high dimensions or under special structural assumptions. These developments position finite-horizon episodic MDPs as a highly expressive, deeply analyzed model at the core of sequential decision theory, with ongoing advances driven by the intersection of learning, control, and optimization. Source: https://www.emergentmind.com/topics/finite-horizon-episodic-markov-decision-processes-mdps