Linear Contextual Stochastic Shortest Path

• Linear CSSP is a framework where both transition probabilities and costs depend linearly on exogenous context variables, extending traditional SSP models.
• Key algorithmic advances include the use of optimistic planning, ridge regression-based confidence sets, and surrogate optimization to ensure proper policy termination.
• Theoretical results provide minimax regret bounds and practical insights into balancing model uncertainty with contextual variability in online learning.

A Linear Contextual Stochastic Shortest Path (CSSP) problem generalizes classical stochastic shortest path (SSP) settings by allowing both transitions and costs to depend on exogenous context variables via an unknown linear mapping. The learner’s goal is to minimize cumulative expected loss to an absorbing goal state across sequential episodes, under model uncertainty and online feedback. Recent research rigorously formulates CSSP and provides minimax regret guarantees, efficient algorithms, and theoretical insights into the dynamics between contextual variability, policy learning, and termination reliability (Polikar et al., 16 Nov 2025, Vial et al., 2021, Chen et al., 2021, Min et al., 2021, Hu et al., 2024).

1. Problem Formulation and Model Structure

A linear CSSP instance consists of a finite state space $S$, action space $A$, and a distinguished absorbing goal state $g \notin S$. At episode $k$, the learner observes a context $c_k \in \mathcal{C} \subseteq \mathbb{R}^d$ (typically the simplex) which adversarially or stochastically parameterizes the underlying MDP:

• Cost function: $\ell^{c}(s,a) = \langle c, L^*(s,a) \rangle$, where $L^*(s,a) \in \mathbb{R}^d$ are unknown cost embeddings.
• Transition kernel: $P^{c}(s'|s,a) = P^*(s,a)c$, $P^*(s,a) \in \mathbb{R}^{|S| \times d}$, with $P^c(g|s,a) = 1-\sum_{s' \in S} P^c(s'|s,a)$.
• Each episode proceeds until the process hits $g$, incurring cumulative loss.

The objective is to minimize the expected cumulative loss for starting state $s_0$ and episode-specific context $c_k$, without prior knowledge of $L^*$ or $P^*$, and under adversarial or stochastic selection of context (Polikar et al., 16 Nov 2025, Hu et al., 2024).

A policy $\pi: S \rightarrow A$ is proper if it reaches $g$ with probability one under any feasible context.

2. Algorithmic Frameworks for CSSP

The principal approaches for online learning in linear CSSP involve optimistic planning, ridge regression-based confidence sets, and episodic replanning.

LR-CSSP Algorithm (Polikar et al., 16 Nov 2025):

• Maintains per-$(s,a)$ ridge regression estimates for cost and transition parameters using all historical data across episodes.
• Uses confidence ellipsoids in parameter space, combined with current context $c_k$, to derive context-specific confidence sets for costs/transitions.
• Performs optimistic extended value iteration (EVI) over the confidence sets to compute a proper policy at each interval.
• Declares $(s,a)$ "known" for context $c$ when the associated regression uncertainty is below a threshold, else replans upon visiting an "unknown" pair.

Induced Empirical Risk Minimization (IERM) Under Bandit Feedback (Hu et al., 2024):

• Observes only realized loss for each played path/decision, not the full contextual cost vector.
• Forms cross-fitted nuisance estimators (context-cost functions, propensity scores) and optimizes convex surrogate losses (SPO+) for the policy function class.
• Scores can be constructed using direct method, inverse spectral weighting, or doubly-robust estimation for theoretical fast rates.

Linear Mixture SSP Algorithms (LEVIS/LEVIS+) (Min et al., 2021):

• Utilize Hoeffding or Bernstein-type confidence sets for the unknown transition model, updated at cost-doubling intervals.
• Execute damped extended value iteration (DEVI) to ensure contraction and optimism.
• Bernstein refinement uses transition variance to speed up confidence contraction.

3. Regret Guarantees and Theoretical Results

General Regret Bounds:

• LR-CSSP achieves horizon-free regret $$R(K) = \widetilde{O}(K^{2/3} d^{2/3} |S| |A|^{1/3} B_\star^2 T_\star)$$ for arbitrary, possibly tiny costs. If $\ell_{\min} > 0$, $$R(K) = \widetilde{O}(\sqrt{K d^2 |S|^3 |A| B_\star^3 / \ell_{\min}})$$, where $B_\star$ is a bound on the optimal cumulative cost and $T_\star$ bounds policy hitting time (Polikar et al., 16 Nov 2025).
• For linear mixture SSPs, Bernstein-type LEVIS+ achieves minimax-optimal $\tilde O(d B_\star \sqrt{K/c_{\min}})$ regret, up to poly-logarithmic and $1/\sqrt{c_{\min}}$ factors (Min et al., 2021).
• In bandit feedback contexts, IERM achieves fast rates proportional to the critical radius of the policy function class and nuisance estimators. If misspecification is negligible and nuisance is well estimated, regret scales as $\tilde O(r^{(2\alpha+2)/(\alpha+2)})$ for margin parameter $\alpha$ (Hu et al., 2024).

Comparison Table of Regret Guarantees

Algorithm/Class	Regret Bound	Assumptions/Notes
LR-CSSP (Polikar et al., 16 Nov 2025)	$$\widetilde{O}(K^{2/3} d^{2/3} |S| |A|^{1/3} B_\star^2 T_\star)$$	Arbitrary small cost, adversarial context
LR-CSSP ($\ell_{\min}>0$)	$$\widetilde{O}(\sqrt{K d^2 |S|^3 |A| B_\star^3 / \ell_{\min}})$$	Cost bounded below
LEVIS (Min et al., 2021)	$\tilde O(d B_\star^{1.5} \sqrt{K/c_{\min}})$	Linear mixture, Hoeffding confidence
LEVIS+ (Bernstein) (Min et al., 2021)	$\tilde O(d B_\star \sqrt{K/c_{\min}})$	Minimax up to slack/log factors
IERM (Hu et al., 2024)	Fast rate in critical radius	Offline, bandit feedback, misspec.

These bounds indicate statistical efficiency in $d$ (context/feature dim.) and problem size, with termination ensured unless improper policies are selected.

4. Proper Policy Class and Termination Guarantees

Unlike in finite-horizon contextual MDPs, in CSSP the challenge extends beyond parameter estimation:

• The episode length is a random variable dependent on transition uncertainty and policy quality.
• Insufficient knowledge may lead to recurrent cycles or even infinite-horizon episodes, violating problem well-posedness.
• Algorithms guarantee properness by replanning upon any $(s,a)$ with insufficient confidence, and constructing policies only over known pairs (Polikar et al., 16 Nov 2025, Min et al., 2021).
• The key model-dependent control parameters for termination are $B_\star$ and $T_\star$.

5. Methodological Advances and Practical Insights

Key advances include:

• Construction of context-dependent confidence sets via ridge regression embedding, enabling generalization across arbitrary continuous context spaces (Polikar et al., 16 Nov 2025).
• Surrogate optimization (SPO+) and doubly-robust scores, which provide robust learning even under model misspecification and bandit feedback (Hu et al., 2024). Empirically, convex surrogates and regularized DR estimators outperform plug-in methods, especially in high-dimensional and misspecified settings.
• Exploitation of problem linearity: regret bounds, computational cost, and confidence set updates all scale in $d$ but not directly in $|S|, |A|$ (with the exception of count-based modeling for unknown pairs).
• Variance-aware confidence sets (Bernstein-type) remove extraneous factors in regret bounds, matching minimax limits (Min et al., 2021).

Example empirical results (Hu et al., 2024):

• Relative regret for SPO+ DM with $n=1600$ samples: $0.16\%$ (well-specified), $0.54\%$ (misspecified); naive methods without linear structure incur $>10\%$ regret.

6. Connections to Related Paradigms

• The CSSP framework generalizes tabular SSP, linear MDPs, and discounted/episodic contextual MDPs (Chen et al., 2021).
• Importantly, the policy termination issue is unique to CSSP: finite-horizon models always terminate after $H$ steps, but CSSP must guarantee properness under continuous context variability.
• Regret-optimal algorithms for CSSP make explicit use of optimism, confidence-driven replanning, and robust estimators, resolving longstanding open questions on termination assurance and statistical rates under adversarial context (Polikar et al., 16 Nov 2025, Vial et al., 2021).
• Current methods are robust to adversarial context selection and arbitrary model misspecification provided underlying linearity is maintained.

7. Open Problems and Future Directions

• Removing the $1/\sqrt{c_{\min}}$ slack from Bernstein-type algorithms to match minimax rates for arbitrarily small or zero costs (Min et al., 2021).
• Extending confidence-based approaches to nonlinear function approximation regimes—potentially under low Bellman rank or other generalized features.
• Closing remaining gaps (e.g., $d^{2.5}$ factors) in horizon-free regret bounds with computationally efficient, scalable algorithms (Chen et al., 2021).
• Improving practical integration of variance-aware confidence and surrogate losses for scalable deployment of CSSP learning in large, dynamic, and continuous-context domains.

Linear Contextual Stochastic Shortest Path thus represents an active research area at the interface of reinforcement learning, robust online optimization, and contextual decision processes, with recent work establishing statistical, computational, and termination guarantees under rigorous linearity-based modeling assumptions.