Stochastic Shortest Path (SSP)

Updated 4 July 2026

SSP is an undiscounted Markov decision process with an absorbing goal state, focused on minimizing the accumulated cost before reaching the target.
It employs Bellman optimality equations and occupancy-measure methods to derive stationary deterministic proper policies under standard conditions.
Recent extensions and learning algorithms adapt SSP for risk-aware, constrained, and function-approximation settings, offering practical heuristics and sublinear regret guarantees.

Stochastic Shortest Path (SSP) is an undiscounted Markov decision process with an absorbing goal state, in which the objective is to minimize the total expected cost accumulated before reaching the goal. In common finite-state formulations, an SSP is specified by a state space, action space, transition kernel, cost function, initial state, and absorbing goal, and the central quantity is the cost-to-go until the hitting time of that goal. SSP strictly generalizes the finite-horizon model, and under standard SSP conditions it admits stationary deterministic proper optimal policies characterized by Bellman optimality equations (Jafarnia-Jahromi et al., 2021, Chen et al., 2022).

1. Formal model and foundational assumptions

A standard finite SSP model uses a finite state space $S$ , a finite action space $A$ , an initial state, and an absorbing goal state $g$ . In one common notation, the augmented state space is $S^+ := S \cup \{g\}$ , the per-step cost is $c : S \times A \to [0,1]$ , and the transition kernel is $P(s' \mid s,a)$ or $\theta(s' \mid s,a)$ over $S^+$ . A policy $\pi$ induces a random hitting time $T$ or $A$ 0, and the cost-to-go is

$A$ 1

The goal state is cost-free and absorbing, so $A$ 2 (Jafarnia-Jahromi et al., 2021, Yin et al., 2022).

A policy is proper if it reaches the goal with probability $A$ 3 from every relevant state. In the classical SSP setting, properness is the condition that replaces discounting: termination ensures that undiscounted total cost is well defined. Under standard SSP conditions, the optimal value function is the unique fixed point of the Bellman operator, and an optimal policy can be taken stationary, deterministic, and proper. In a single-goal formulation, the Bellman optimality equations are

$A$ 4

A commonly used complexity parameter is

$A$ 5

the maximum optimal expected cost-to-go over states (Jafarnia-Jahromi et al., 2021, Yin et al., 2022).

Several equivalent formulations coexist. Some planning papers use a set of absorbing goal states $A$ 6, while some polyhedral treatments allow real-valued one-step costs and row-substochastic transition matrices. These are alternative formalizations of the same undiscounted goal-reaching problem rather than different problem classes in substance (Pineda et al., 2017, Guillot et al., 2017).

2. Bellman equations, occupancy measures, and polyhedral structure

A useful alternative to the Bellman view is the occupancy-measure view. For an initial distribution $A$ 7, the occupancy measure $A$ 8 is the expected number of times action $A$ 9 is taken in state $g$ 0 before hitting the target. In matrix form, the flow constraints are

$g$ 1

and the corresponding linear objective is to minimize $g$ 2. This produces the standard occupancy-measure LP for SSP and directly connects SSP to network-flow structure (Guillot et al., 2017).

This perspective yields a precise generalization of negative-cost cycles. A transition cycle is any nonzero $g$ 3 with $g$ 4; a negative transition cycle is such a vector with $g$ 5. Under the pair of assumptions that the target is reachable from every state in the support graph and that there is no negative transition cycle, the SSP is well-defined, deterministic stationary optimal policies exist, the Bellman equations hold, and Value Iteration, Policy Iteration, and a primal–dual method extending Dijkstra’s algorithm all converge (Guillot et al., 2017).

The same framework clarifies policy structure. Extreme points of the occupancy polytope correspond to proper deterministic stationary policies, and complementary slackness links optimal occupancy vectors to Bellman-minimizing actions. This link between Bellman inequalities and active constraints has also motivated recent constraint-generation methods: instead of evaluating all actions at every backup, one can treat each action as a Bellman constraint and add only violated constraints during search (Schmalz et al., 2024).

3. Exact, heuristic, and hybrid planning algorithms

Classical exact methods for SSP include Value Iteration, Policy Iteration, and LP. In one Bellman–Ford–type construction, Value Iteration is implemented through an auxiliary finite-horizon SSP with a “big- $g$ 6 to target” action and converges to $g$ 7 when initialized above the optimal value. Howard’s Policy Iteration has a reduced-cost interpretation and terminates finitely under the standard structural assumptions, while the primal–dual method reduces to Dijkstra’s mechanism when transitions are deterministic and costs are nonnegative (Guillot et al., 2017).

In probabilistic planning, heuristic search replaces global sweeps by partial-graph computation. LAO* alternates graph expansion along the current best partial policy with Bellman backups on the current partial solution graph. iLAO* similarly maintains an explicit partial SSP, expands only frontier states reachable under the current greedy policy, and tests $g$ 8-consistency on the greedy-policy envelope. These methods focus computation on reachable regions rather than the full state space (Pineda et al., 2017, Schmalz et al., 2024).

A separate planning line uses determinization and reduced models. In reduced models, each action has a chosen primary outcome and an exception bound $g$ 9, producing an $S^+ := S \cup \{g\}$ 0 model that interpolates between pure determinization and limited probabilistic reasoning. FF-LAO* solves such reduced SSPs by combining LAO* on the probabilistic region with the deterministic planner FF once the exception counter reaches $S^+ := S \cup \{g\}$ 1, and online replanning maps the reduced-model policy back to the original SSP (Pineda et al., 2017).

Constraint generation sharpens heuristic search further. CG-iLAO* partially expands fringe states by adding only currently greedy actions and later adds other actions only when their Bellman inequalities are violated. Empirically, CG-iLAO* “ignores up to 57% of iLAO*’s actions” and “solves problems up to 8x and 3x faster than LRTDP and iLAO*,” while preserving $S^+ := S \cup \{g\}$ 2-consistency and optimality under standard SSP assumptions (Schmalz et al., 2024).

4. Learning in unknown SSPs

Online reinforcement learning in SSP is harder than finite-horizon learning because episode length is not fixed and is itself policy-dependent. Posterior-sampling, optimism-based, and reduction-based algorithms now all achieve sublinear regret in this setting. PSRL-SSP samples a model from the posterior at the beginning of each epoch, follows the optimal policy for the sampled model during that epoch, and achieves Bayesian regret $S^+ := S \cup \{g\}$ 3 (Jafarnia-Jahromi et al., 2021). EB-SSP constructs an optimistic SSP by skewing empirical transitions toward the goal and perturbing empirical costs; it is parameter-free and achieves the minimax rate $S^+ := S \cup \{g\}$ 4, with nearly horizon-free guarantees in several regimes (Tarbouriech et al., 2021). A separate reduction from SSP to finite-horizon MDPs yields $S^+ := S \cup \{g\}$ 5 regret with matching lower bounds up to logarithmic factors (Cohen et al., 2021), while another optimism-based analysis removes dependence on the minimum instantaneous cost and proves $S^+ := S \cup \{g\}$ 6 regret together with the lower bound $S^+ := S \cup \{g\}$ 7 (Cohen et al., 2020).

The learning picture also includes nonstationary costs and policy optimization. For adversarially changing costs with fixed unknown transitions, occupancy-measure online mirror descent yields the first sublinear regret guarantees: $S^+ := S \cup \{g\}$ 8 under strictly positive costs and $S^+ := S \cup \{g\}$ 9 in the general case (Rosenberg et al., 2020). Policy-optimization methods extend this to stochastic and adversarial SSP environments under full-information or bandit feedback. Their key tool is stacked discounted approximation, which learns a near-stationary policy with only logarithmic changes during an episode and can yield an exponential improvement in space complexity relative to finite-horizon approximation (Chen et al., 2022).

Function approximation and offline learning introduce separate statistical issues. For linear mixture SSPs, LEVIS and LEVIS+ attain $c : S \times A \to [0,1]$ 0 and $c : S \times A \to [0,1]$ 1 regret, together with an $c : S \times A \to [0,1]$ 2 lower bound (Min et al., 2021). A complementary line studies optimistic approximate fixed points and gives computationally efficient stationary-policy algorithms with sublinear regret under linear realizability (Vial et al., 2021). In the offline setting, simple value-iteration procedures for off-policy evaluation and off-policy learning obtain instance-dependent guarantees that imply near-minimax worst-case bounds (Yin et al., 2022). More recently, actor–critic methods for tabular and function-approximation SSP have been shown to converge asymptotically almost surely under standard SSP assumptions (Guin et al., 19 Aug 2025).

5. Extensions beyond expectation minimization

Several active extensions retain the absorbing-goal structure of SSP while changing the optimization criterion or the model class.

Variant	Objective or model change	Representative result
Two-player SSP games	Zero-sum stochastic game with cost-free absorbing terminal state	Bellman–Isaacs equation has a unique solution; stationary saddle-point policies exist; value iteration, policy iteration, and Q-learning converge under the paper’s SSP game conditions (Yu, 2014)
Risk-aware SSP	Replace expectation by $c : S \times A \to [0,1]$ 3	Exact LP and exact value-iteration algorithms are given for CVaR-SSP on Markov chains and MDPs (Meggendorfer, 2022)
Risk-sensitive sampling-based planning	Optimize cumulative segment-wise CVaR in incremental sampling	RA-RRT* replaces expectation by CVaR, keeps asymptotic optimality under the paper’s assumptions, and has similar query time and space complexity to RRT* (Enwerem et al., 2024)
Variance-penalized SSP	Optimize $c : S \times A \to [0,1]$ 4	An optimal deterministic finite-memory scheduler exists; the threshold problem is EXPTIME-hard and in NEXPTIME (Piribauer et al., 2022)
Goal-oriented MDPs with dead ends	Relax classical SSP to admit dead ends	SSPADE, fSSPUDE, and iSSPUDE are introduced; $c : S \times A \to [0,1]$ 5, while iSSPUDE first maximizes goal-reach probability and then minimizes conditional expected cost (Kolobov et al., 2012)
Constrained SSP	Minimize a primary cost subject to secondary-cost constraints	CARL solves a series of unconstrained SSPs built from scalarisations and “solves 50% more problems than the state-of-the-art” on the reported benchmarks (Schmalz et al., 24 Aug 2025)

These variants show that expectation-optimal SSP structure does not automatically transfer to nonlinear criteria. In particular, CVaR and variance-penalized objectives are sensitive to tail behavior, and the optimal controller may require additional state information or finite memory beyond the stationary deterministic policies that suffice in classical SSP. This suggests a broader distinction between Bellman-optimal cost-to-go control and risk-sensitive path-distribution control (Meggendorfer, 2022, Piribauer et al., 2022).

6. Complexity, representations, and open directions

Under reachability of the target and absence of negative transition cycles, SSP is weakly polynomial via linear programming, and detecting negative transition cycles is also weakly polynomial. At the same time, Value Iteration and Policy Iteration may still require exponentially many iterations in the worst case, and a strongly polynomial algorithm remains open. The complexity of the MAXPROB subproblem and of the extended Dijkstra method is also left open in the polyhedral framework (Guillot et al., 2017).

Beyond explicit finite-state models, SSP has also been studied on succinct MDPs. In that setting, the MDP is represented by variables and linear update rules inside a while-loop program, and upper bounds can be synthesized by linear upper potential functions via LP, while lower bounds can be synthesized by linear lower potential functions via a reduction to quadratic programming. These computations are polynomial in the size of the implicit description and apply even to infinite-state MDPs (Chatterjee et al., 2018). In explicit probabilistic planning, learned determinization and exception-bounded reduced models remain important because they allow classical planners to be reused inside SSP search, but empirical performance depends strongly on the domain and on how much stochastic reasoning is retained (Pineda et al., 2017).

Several open directions recur across the literature. Offline SSP work identifies sharper horizon-free higher-order terms, extension to function approximation, and tighter lower bounds for off-policy evaluation as open problems (Yin et al., 2022). Risk-aware planning papers emphasize the dependence of current guarantees on assumptions such as independent additive Gaussian noise, static geometries, or dynamically consistent risk decompositions (Enwerem et al., 2024, Meggendorfer, 2022). Across exact and learning-based SSP, the persistent themes are the same: preserving properness without discounting, exploiting occupancy structure without enumerating irrelevant actions or states, and controlling the interaction between random episode length and statistical uncertainty.