Safe Accelerated Value Iteration (S-AVI)

• S-AVI is a set of techniques combining acceleration strategies with safety guarantees to efficiently solve Markov decision processes and compute reachability probabilities.
• It integrates Nesterov-style acceleration for optimizing discounted MDPs and probabilistic scalar bounds for sound reachability analysis in model checking.
• Empirical studies show S-AVI can reduce iterations by factors of 2–10 with minimal fallback to classical methods while preserving strong theoretical guarantees.

Safe Accelerated Value Iteration (S-AVI) refers to a class of techniques for solving Markov decision processes (MDPs) and Markov chains (MCs) that blend acceleration strategies with safety mechanisms to obtain fast, robust, and sound convergence to optimal value functions or reachability probabilities. Notably, two unrelated algorithms share the acronym S-AVI: one developed by Sidford et al. for optimizing discounted MDPs using Nesterov-style acceleration within value iteration (Goyal et al., 2019), and another by Quatmann & Katoen for model checking, providing sound bounds for reachability analysis (Quatmann et al., 2018). Both aim to exploit acceleration yet enforce tight safety guarantees, albeit via distinct mathematical and algorithmic designs.

1. Markov Decision Process Formulations

S-AVI methods are grounded in the fixed-point structures of discrete-time stochastic processes. In the discounted infinite-horizon MDP framework, one considers a tuple $(S, A, P, r, \lambda)$, where $S$ is a finite state set ($|S|=n$), $A$ is the action set, $P_{i,a,j}$ is the probability of transitioning from $i$ to $j$ via action $a$, $r_{i,a}$ is the immediate reward, and $\lambda \in (0,1)$ is the discount factor. A (stationary) policy $S$0 prescribes for each state $S$1 a distribution $S$2 over $S$3. The value function for policy $S$4 is the unique solution to the Bellman linear system:

$S$5

where $S$6 and $S$7 aggregate rewards and transitions under $S$8. The optimal value function $S$9 is the unique fixed point of the Bellman optimality operator $|S|=n$0:

$|S|=n$1

with an optimal policy recoverable from maximizing actions.

In reachability analysis, as addressed by Quatmann & Katoen, the task is to compute, for an MC or MDP, the maximal probability of reaching a set of absorbing goal states $|S|=n$2. The objective is to solve the Bellman-type equation:

$|S|=n$3

2. Motivation for Acceleration and Safety

Classical value iteration (VI) repeatedly applies the Bellman operator: $|S|=n$4. This is a $|S|=n$5-contraction in $|S|=n$6, so for reaching $|S|=n$7-accuracy, VI requires $|S|=n$8 iterations. As $|S|=n$9, the number of steps and hence the computational effort grows rapidly, which is prohibitive for long-horizon planning.

Sidford et al. established a formal analogy between VI and gradient descent, observing that $A$0 is Lipschitz with constant $A$1 and "strongly monotone" with $A$2. This mirrors smooth, strongly convex minimization, where Nesterov acceleration reduces the condition number dependence from $A$3 to $A$4. For reversible MDPs, a corresponding iteration bound of $A$5 is achieved in policy evaluation.

Classical VI lacks reliable stopping criteria in some settings (notably for reachability in model checking), as difference-based stopping rules can yield unsafe results. In response, the S-AVI of Quatmann & Katoen provides provably sound bounds—trading full vector intervals for cheap-to-compute scalar intervals—ensuring the accuracy of the returned solution throughout the iterative process.

3. Algorithmic Descriptions

3.1 Safe Accelerated Value Iteration (Sidford et al.)

S-AVI alternates between an aggressive accelerated (Nesterov-style) step and a fallback safe (classical VI) step, using a test to determine step safety. Let $A$6, fix target contraction $A$7 with $A$8, and select Nesterov-like constants $A$9 (e.g., $P_{i,a,j}$0):

Initialization:

• $P_{i,a,j}$1 arbitrarily,
• $P_{i,a,j}$2,
• $P_{i,a,j}$3.

For $P_{i,a,j}$4:

1. Accelerated trial:
   • $P_{i,a,j}$5,
   • $P_{i,a,j}$6.
2. If $P_{i,a,j}$7 accept accelerated step: $P_{i,a,j}$8; else fallback to VI: $P_{i,a,j}$9.

3.2 Sound Value Iteration (Quatmann & Katoen)

S-AVI for reachability replaces the standard double-sided VI (upper/lower vectors) by tracking two scalars $i$0. At each step $i$1, probabilities to reach $i$2 in $i$3 steps ($i$4) and staying outside $i$5 for $i$6 steps ($i$7) are recursively updated, and the interval $i$8 is computed using

$i$9

Termination occurs when $j$0, after which the solution is approximated by $j$1.

4. Theoretical Guarantees

For Sidford et al.'s S-AVI, the following theorem holds: for any $j$2, $j$3. If $j$4, asymptotic convergence matches that of VI, yielding $j$5 iterations for $j$6-accuracy. Increasing $j$7 permits more aggressive steps, yet the algorithm never exceeds the worst-case iteration count of VI. A lower bound establishes that no first-order method, whose iterates lie in the span of past residues and estimates, can outperform VI’s rate uniformly. Thus, S-AVI is worst-case optimal among such algorithms (Goyal et al., 2019).

Quatmann & Katoen's S-AVI obtains guaranteed soundness with $j$8-precision for all absorbing goal sets and tolerances. Soundness and convergence stem from monotonicity of $j$9, $a$0, the path-splitting lemma, and the contraction property ($a$1), ensuring the interval collapses suitably fast and that the returned estimate is within $a$2 of the true value (Quatmann et al., 2018).

5. Empirical Performance and Comparative Analysis

Extensive evaluations show that:

• Sidford et al.'s S-AVI typically accelerates convergence by an order of magnitude or more versus VI when $a$3. Empirical speedups of up to tenfold are observed on both structured (forest management, healthcare) and dense random Garnet MDPs. Policy Iteration (PI) is competitive for smaller $a$4, but is hampered by costly per-iteration linear solves. Gauss–Seidel VI offers modest gains on dense problems. Anderson acceleration can help for moderate $a$5 but is often unstable for high $a$6 (Goyal et al., 2019).
• In Quatmann & Katoen's analysis, S-AVI reduced the number of iterations by factors of 2–10 (occasionally an order of magnitude) compared to double-sided interval VI across PRISM benchmarks. Wall-clock speed-ups average 20% over interval VI, with particular topological variants achieving the fastest sound runtimes, nearly matching "plain" VI for many large MCs and MDPs (Quatmann et al., 2018).

Both algorithms exhibit empirical acceleration while preserving provable guarantees. In Sidford et al.'s S-AVI, fallback to safe (classical) steps is rare (typically $a$7), allowing nearly all benefits of acceleration without risk of divergence.

6. Connections to Convex Optimization and Algorithmic Context

The S-AVI framework of Sidford et al. exploits a close analogy between value iteration and first-order optimization of strongly convex, smooth functions. The constants $a$8 (Lipschitz) and $a$9 ("strong monotonicity") play roles analogous to the smoothness and strong convexity parameters, with the Bellman residual acting as a surrogate gradient. This supports the introduction of Nesterov-like acceleration and an associated safe guard.

Quatmann & Katoen's S-AVI does not leverage Nesterov-style dynamics but arises from a probabilistic argument—path splitting and scalar bounding—which enables soundness without reliance on explicit initial intervals or contraction rates.

A plausible implication is that, while both algorithms are termed "safe" and "accelerated," the techniques are fundamentally distinct—Sidford et al. embedding optimization-theoretic acceleration within value iteration under contraction and residual control; Quatmann & Katoen enforcing soundness for verification via pathwise probabilistic analysis.

7. Summary Table: S-AVI Algorithmic Properties

S-AVI Variant	Core Application	Safety Mechanism	Acceleration Source
Sidford et al. (Goyal et al., 2019)	Discounted MDP solving	Fallback to VI if residual too large	Nesterov-style (momentum)
Quatmann & Katoen (Quatmann et al., 2018)	MC/MDP reachability	Online scalar bounds via path-splitting	Tight scalar interval bounds

Both S-AVI variants represent significant advances for their respective problem domains, demonstrating that acceleration and safety can be synergistically realized within iterative fixed-point and model-checking algorithms.