---
title: Finite-Horizon Markov Decision Process
url: https://www.emergentmind.com/topics/finite-horizon-markov-decision-process-mdp
type: topic
---

# Finite-Horizon Markov Decision Process

A finite-horizon Markov Decision Process (MDP) is a discrete-time stochastic control framework in which the decision-maker operates over a fixed sequence of time steps, rather than an indefinite or infinite horizon. The objective is typically to optimize the probability of attaining designated goals, expected cumulative rewards, or related objectives within the specified horizon. Finite-horizon MDPs are ubiquitous in stochastic planning, operations research, reinforcement learning, and probabilistic verification.

## 1. Formal Definition and Model Construction

A finite-horizon MDP is specified as the tuple $(S, A, P, s_0, F, H)$, where:
- $S$ is a finite set of $n$ states
- $A$ is a finite set of actions, with $A_s\subseteq A$ denoting admissible actions in state $s$
- $P: S \times A \times S \to [0,1]$ is the transition probability function, so that $P(s,a,s')$ is the probability of transitioning from $s$ to $s'$ under action $a$
- $s_0 \in S$ is the initial state
- $F \subseteq S$ is a set of target (goal) states
- $H \in \mathbb{N}$ is the finite horizon

At each stage $t = 0, 1, \ldots, H-1$, if the system is in state $s_t\notin F$, the controller chooses $a_t \in A_{s_t}$, and the next state $s_{t+1}$ is drawn from $P(s_t, a_t, \cdot)$. If $s_t\in F$, the process is absorbed in $F$. The canonical reachability objective is to maximize $\Pr[\exists t \leq H : s_t \in F]$ [1209.3617]. More general reward-based objectives can also be defined, using cumulative rewards $r_t(s, a)$ and terminal rewards $r_H(s)$.

## 2. Value Functions, Bellman Recursion, and Optimal Policies

The optimal value function $V_t^*(s)$ is defined recursively by
\[
V_H^*(s) = \mathbf{1}\{s \in F\}
\]
\[
V_t^*(s) = \max_{a\in A_s} \left[ r_t(s, a) + \sum_{s'} P(s, a, s') V_{t+1}^*(s') \right]
\]
for $t=H-1,\dots,0$. The corresponding optimal policy $\pi^* = (\pi_0^*,\dots,\pi_{H-1}^*)$ satisfies $\pi_t^*(s) = \arg\max_{a\in A_s} Q_t^*(s, a)$, with $Q_t^*(s, a)=r_t(s, a) + \sum_{s'} P(s,a,s')V_{t+1}^*(s')$ [1209.3617].

For reachability, rewards are typically indicators for entering $F$, and for general total or average reward, $r_H(s)$ captures terminal payoff. The time dependence of the policy (nonstationarity) is intrinsic: optimal actions generally depend on the remaining steps $H-t$ [2511.17598, 1209.3617].

## 3. Strategy Complexity and Memory Requirements

Strategy complexity in finite-horizon MDPs is governed by two primary lower bounds [1209.3617]:
- For any $\epsilon > 0$, every $\epsilon$-optimal counter-based strategy requires at most $\log_2\log_2(1/\epsilon) + n + 1$ bits of memory, and memory of size $\Omega(\log\log(1/\epsilon) + n)$ is required in the worst case.
- This memory is necessary to distinguish both long paths (requiring $\Omega(n)$ bits) and precise timing to $\epsilon$-precision ($\Omega(\log\log(1/\epsilon))$ bits).

Counter-based strategies, which update their memory state as a deterministic function of stage count, suffice for $\epsilon$-optimality. For exact optimality, the minimal period of any finite-memory optimal strategy may be as large as $2^{\Omega(\sqrt{n \log n})}$ [1209.3617].

| Complexity Metric                    | Upper Bound                     | Lower Bound                   |
|:-------------------------------------|:-------------------------------|:-----------------------------|
| Bits to $\epsilon$-optimality        | $\log_2\log_2(1/\epsilon) + n + 1$ | $\Omega(\log\log(1/\epsilon)+n)$ |
| Period of exact-optimal strategy     | $-$                            | $2^{\Omega(\sqrt{n\,\log n})}$   |

This establishes tight memory and period bounds for finite-horizon reachability MDPs.

## 4. Structural Properties and Periodicity of Strategies

A counter-based strategy $\sigma$ is characterized by an eventually periodic memory sequence $(m_0, m_1, \ldots)$: after a finite pre-period, controls repeat with some period $P$. There are explicit constructions (combining clocked "gadgets" controlled by primes) that necessitate sub-exponential periods $P \geq 2^{\Omega(\sqrt{n\log n})}$ for optimality [1209.3617]. This demonstrates that even for moderate $n$, optimal control can entail highly complex periodic switching.

## 5. Algorithms and Computational Methods

Backward dynamic programming (value iteration) is the standard method for solving finite-horizon MDPs [1209.3617]. For $H$ steps, $n$ states, and $m$ actions, the computational complexity is $O(H n m)$ per iteration. In cases requiring explicit strategy extraction, one reconstructs the nonstationary optimal policy from stored $Q^*_t(s,a)$.

For more complex specifications (e.g., vector-valued objectives or constraints), additional linear programming or bilinear programming techniques are introduced. Notably, linear program formulations for occupancy measures cover both standard and multi-objective formulations [2502.13697]. In the context of strategy complexity, the focus is strictly on counter-based strategies, whose memory and periodicity requirements sharply govern implementation complexity [1209.3617].

## 6. Bounds, Limitations, and Reachability MDPs

The structural results detailed above reveal fundamental limits in memory and periodicity that cannot be bypassed by algorithmic improvements: $\epsilon$-optimal policies in finite-horizon MDPs can never be implemented with less than $\Omega(\log\log(1/\epsilon) + n)$ bits of memory, and the period of exact-optimal strategies has a provable subexponential lower bound [1209.3617]. These findings underscore intrinsic trade-offs in policy synthesis for high-precision planning or systems with large state spaces.

For reachability objectives, infinite-horizon memoryless strategies suffice in the unconstrained case (Condon '92). However, the presence of finite horizon drastically increases the required memory and periodicity, as demonstrated by the matching lower bounds.

## 7. Relevance and Implications in Modern Research

These results have direct consequences for probabilistic verification, controller synthesis in robotics, and stochastic games. The explicit dependence of memory requirements on both problem size and precision parameter $\epsilon$ serves as a practical and theoretical guide for scalable controller and policy synthesis in finite-horizon environments. The sub-exponential periodicity lower bound further implies that, for certain MDPs, any exact-optimal controller must implement a schedule with extremely long, non-repetitive preamble [1209.3617].

These bounds also set optimality benchmarks for current and future research into symbolic, memory-efficient, and computationally tractable planning in stochastic finite-horizon settings.

Source: https://www.emergentmind.com/topics/finite-horizon-markov-decision-process-mdp