Sound Value Iteration (SVI)

• Sound Value Iteration (SVI) is a fixed-point method that computes certified lower and upper bounds for reachability probabilities and expected rewards.
• It adapts to different models such as Markov chains, MDPs, stochastic games, and POMDPs using techniques like Bellman backups, ratio bounds, and trial-based search.
• SVI converges faster than classical value iteration by providing a sound stopping criterion based on a prescribed precision bound.

Sound Value Iteration (SVI) denotes value-iteration procedures that compute sound lower and upper bounds for quantitative verification and planning objectives, most prominently unbounded reachability probabilities and expected rewards. Unlike classical value iteration, which approximates a fixed point from below and does not provide a sound stopping criterion, SVI maintains certified brackets and terminates once the gap satisfies a prescribed precision bound. The term appears in several closely related settings: Markov chains and MDPs (Quatmann et al., 2018), simple stochastic games and MDPs with end components (Azeem et al., 2024, Azeem et al., 17 Sep 2025), and trial-based heuristic search value iteration for the indefinite-horizon maximal reachability probability problem in POMDPs (Ho et al., 2024).

1. Formal setting and fixed-point semantics

In the original probabilistic model-checking setting, a finite Markov chain is a tuple $(S,P,s_I)$ with finite state set $S$, initial state $s_I\in S$, transition matrix $P\colon S\times S\to[0,1]$, and a set of target states $G\subseteq S$. For $k\in\mathbb N\cup\{\infty\}$, the step-bounded and unbounded reachability probabilities are

$$\mathsf{Reach}^{\le k}_s(G) =\Pr_{\,s}\bigl(\text{reach }G\text{ in at most }k\text{ steps}\bigr), \qquad \mathsf{Reach}_s(G)=\lim_{k\to\infty}\mathsf{Reach}^{\le k}_s(G).$$

For MDPs, maximum reachability is

$$\mathsf{Reach}^{\max}_s(G)=\sup_{\sigma}\Pr^\sigma_s(\diamond G).$$

Standard value iteration approximates the least fixed-point of

$$f(x)_s=\sum_{t}P(s,t)x_t,\quad x|_{G}=1,\;x|_{S\setminus G}=0$$

from below; for MDPs the matrix-vector product is replaced by a Bellman backup $\max_{\alpha}\sum_tP(s,\alpha,t)\cdot x_t$ (Quatmann et al., 2018).

The same fixed-point perspective persists in later SVI variants. In simple stochastic games, the reachability value is

$S$0

and classical VI computes lower approximations by alternating $S$1 and $S$2 Bellman updates depending on state ownership (Azeem et al., 2024). In the POMDP MRPP setting, the belief-space Bellman operator is

$S$3

with the unique fixed point $S$4 satisfying $S$5 (Ho et al., 2024).

This common fixed-point structure explains why SVI is best understood as a soundification of value iteration rather than as a single model-specific algorithm. The core design question is how to preserve monotone lower approximations while also deriving safe upper approximations that support a valid stopping rule.

2. Original SVI for Markov chains and MDPs

The 2018 formulation of SVI was presented as an alternative to interval iteration that “does not require the a priori computation of starting vectors” (Quatmann et al., 2018). Its key decomposition for Markov chains is

$S$6

Here $S$7 is obtained by $S$8 iterations of $S$9 from the zero vector, while $s_I\in S$0 is obtained by $s_I\in S$1 iterations of

$s_I\in S$2

from the all-ones vector (Quatmann et al., 2018).

As soon as $s_I\in S$3 for all $s_I\in S$4, SVI derives local ratio bounds

$s_I\in S$5

and therefore defines

$s_I\in S$6

This yields the sound bracket

$s_I\in S$7

The stopping criterion

$s_I\in S$8

guarantees final error at most $s_I\in S$9 (Quatmann et al., 2018).

The algorithmic presentation in the same work precomputes

$P\colon S\times S\to[0,1]$0

initializes $P\colon S\times S\to[0,1]$1, $P\colon S\times S\to[0,1]$2, and iteratively updates $P\colon S\times S\to[0,1]$3 and $P\colon S\times S\to[0,1]$4 by

$P\colon S\times S\to[0,1]$5

once $P\colon S\times S\to[0,1]$6 (Quatmann et al., 2018).

For MDPs, the same paper states that one replaces the matrix-vector product by a Bellman backup and maintains “a scheduler” choice that maximizes the combined local bound at each iteration, plus “a small linear program to tighten the upper bound without violating optimality of that choice” (Quatmann et al., 2018). A common misconception is to equate SVI with interval iteration. The original construction is explicitly different: its upper and lower bounds are derived on the fly from the current iterates rather than supplied a priori.

3. Soundness, convergence, and expected rewards

The central invariant of SVI is the maintenance of a sound bracket. In the Markov-chain formulation, once $P\colon S\times S\to[0,1]$7 for all $P\colon S\times S\to[0,1]$8, defining

$P\colon S\times S\to[0,1]$9

implies

$G\subseteq S$0

As $G\subseteq S$1, $G\subseteq S$2 on the set of states from which $G\subseteq S$3 is (almost) surely reached, and the bracket width $G\subseteq S$4 (Quatmann et al., 2018).

The computational profile of the original algorithm is correspondingly simple. Each iteration costs one sparse-matrix-vector multiply for $G\subseteq S$5 and one for $G\subseteq S$6, for total $G\subseteq S$7, plus an $G\subseteq S$8 scan to recompute $G\subseteq S$9 and $k\in\mathbb N\cup\{\infty\}$0. The same source reports that SVI “often needs far fewer iterations than interval iteration because its local bounds $k\in\mathbb N\cup\{\infty\}$1 typically converge quickly to the actual $k\in\mathbb N\cup\{\infty\}$2, whereas interval iteration uses global worst–best initial bounds (0 and 1) and thus wastes many iterations shrinking a large gap” (Quatmann et al., 2018).

The extension to expected rewards follows the same decomposition principle. If each MC state $k\in\mathbb N\cup\{\infty\}$3 carries reward $k\in\mathbb N\cup\{\infty\}$4 and the objective is expected total reward until hitting $k\in\mathbb N\cup\{\infty\}$5, then

$k\in\mathbb N\cup\{\infty\}$6

and the remainder is bounded by

$k\in\mathbb N\cup\{\infty\}$7

The abstract of the original paper states that “lifting this technique to expected rewards is trivial for both Markov chains and MDPs” (Quatmann et al., 2018).

These results situate SVI within fixed-point verification rather than purely numerical approximation. The lower sequence remains monotone, the upper sequence remains sound, and the termination test is tied directly to a certified residual quantity instead of to a heuristic difference between successive iterates.

4. End components, stochastic games, and geometric-series acceleration

Later work extended SVI beyond the original Markov-chain and MDP setting. The 2024 and 2025 papers titled “Sound Value Iteration for Simple Stochastic Games” state that SVI had not been applicable to simple stochastic games, nor to MDPs with end components, and that the main technical challenge is the proper treatment of end components (Azeem et al., 2024, Azeem et al., 17 Sep 2025).

In the SG formulation, SVI maintains step-bounded reach and stay probabilities. For a fixed strategy pair $k\in\mathbb N\cup\{\infty\}$8,

$k\in\mathbb N\cup\{\infty\}$9

yielding bounds of the form

$$\mathsf{Reach}^{\le k}_s(G) =\Pr_{\,s}\bigl(\text{reach }G\text{ in at most }k\text{ steps}\bigr), \qquad \mathsf{Reach}_s(G)=\lim_{k\to\infty}\mathsf{Reach}^{\le k}_s(G).$$0

The associated pseudocode in the 2025 account uses arrays $$\mathsf{Reach}^{\le k}_s(G) =\Pr_{\,s}\bigl(\text{reach }G\text{ in at most }k\text{ steps}\bigr), \qquad \mathsf{Reach}_s(G)=\lim_{k\to\infty}\mathsf{Reach}^{\le k}_s(G).$$1 and $$\mathsf{Reach}^{\le k}_s(G) =\Pr_{\,s}\bigl(\text{reach }G\text{ in at most }k\text{ steps}\bigr), \qquad \mathsf{Reach}_s(G)=\lim_{k\to\infty}\mathsf{Reach}^{\le k}_s(G).$$2, updates strategies to optimize $$\mathsf{Reach}^{\le k}_s(G) =\Pr_{\,s}\bigl(\text{reach }G\text{ in at most }k\text{ steps}\bigr), \qquad \mathsf{Reach}_s(G)=\lim_{k\to\infty}\mathsf{Reach}^{\le k}_s(G).$$3 for the Maximizer and $$\mathsf{Reach}^{\le k}_s(G) =\Pr_{\,s}\bigl(\text{reach }G\text{ in at most }k\text{ steps}\bigr), \qquad \mathsf{Reach}_s(G)=\lim_{k\to\infty}\mathsf{Reach}^{\le k}_s(G).$$4 for the Minimizer, and then tightens

$$\mathsf{Reach}^{\le k}_s(G) =\Pr_{\,s}\bigl(\text{reach }G\text{ in at most }k\text{ steps}\bigr), \qquad \mathsf{Reach}_s(G)=\lim_{k\to\infty}\mathsf{Reach}^{\le k}_s(G).$$5

once all $$\mathsf{Reach}^{\le k}_s(G) =\Pr_{\,s}\bigl(\text{reach }G\text{ in at most }k\text{ steps}\bigr), \qquad \mathsf{Reach}_s(G)=\lim_{k\to\infty}\mathsf{Reach}^{\le k}_s(G).$$6 (Azeem et al., 17 Sep 2025).

The distinctive claim of this extension is that SVI “exploits a geometric-series argument to handle probabilistic loops in a sound and often faster way” (Azeem et al., 17 Sep 2025). The complication is that end components create sure cycles, so basic upper-bound updates may stall. The 2024 presentation states that, because of non-determinism, one cannot simply set

$$\mathsf{Reach}^{\le k}_s(G) =\Pr_{\,s}\bigl(\text{reach }G\text{ in at most }k\text{ steps}\bigr), \qquad \mathsf{Reach}_s(G)=\lim_{k\to\infty}\mathsf{Reach}^{\le k}_s(G).$$7

and therefore introduces “decision values” to preserve monotonicity of the chosen action (Azeem et al., 2024).

The mechanism for end components combines two ideas. First, a recursive $$\mathsf{Reach}^{\le k}_s(G) =\Pr_{\,s}\bigl(\text{reach }G\text{ in at most }k\text{ steps}\bigr), \qquad \mathsf{Reach}_s(G)=\lim_{k\to\infty}\mathsf{Reach}^{\le k}_s(G).$$8 procedure collects best exits of an end component $$\mathsf{Reach}^{\le k}_s(G) =\Pr_{\,s}\bigl(\text{reach }G\text{ in at most }k\text{ steps}\bigr), \qquad \mathsf{Reach}_s(G)=\lim_{k\to\infty}\mathsf{Reach}^{\le k}_s(G).$$9 with respect to a current over-approximation $$\mathsf{Reach}^{\max}_s(G)=\sup_{\sigma}\Pr^\sigma_s(\diamond G).$$0; trap end components with no Maximizer exit are absorbed into the sink set $$\mathsf{Reach}^{\max}_s(G)=\sup_{\sigma}\Pr^\sigma_s(\diamond G).$$1 (Azeem et al., 2024). Second, a phantom “delay” action $$\mathsf{Reach}^{\max}_s(G)=\sup_{\sigma}\Pr^\sigma_s(\diamond G).$$2 preserves $$\mathsf{Reach}^{\max}_s(G)=\sup_{\sigma}\Pr^\sigma_s(\diamond G).$$3 whenever every real action would worsen the local bound $$\mathsf{Reach}^{\max}_s(G)=\sup_{\sigma}\Pr^\sigma_s(\diamond G).$$4, so that over-approximations become monotone non-increasing (Azeem et al., 17 Sep 2025). Both papers also describe topological or SCC-based variants that maintain local bounds per strongly-connected component and freeze an SCC once its lower and upper bounds coincide (Azeem et al., 2024, Azeem et al., 17 Sep 2025).

A plausible implication is that the SG/EC extensions shift SVI from a purely global bracketing scheme to one that must explicitly control local strategy choices inside cyclic structures. The formal motivation given in the sources is exactly that end components admit multiple fixpoints for the upper iteration and therefore require additional structure beyond the original ratio bound.

5. Trial-based heuristic-search SVI for undiscounted POMDPs

The 2024 paper “Sound Heuristic Search Value Iteration for Undiscounted POMDPs with Reachability Objectives” studies the indefinite-horizon maximal reachability probability problem in POMDPs (Ho et al., 2024). A POMDP is given as

$$\mathsf{Reach}^{\max}_s(G)=\sup_{\sigma}\Pr^\sigma_s(\diamond G).$$5

where $$\mathsf{Reach}^{\max}_s(G)=\sup_{\sigma}\Pr^\sigma_s(\diamond G).$$6 is a finite set of hidden states, $$\mathsf{Reach}^{\max}_s(G)=\sup_{\sigma}\Pr^\sigma_s(\diamond G).$$7 a finite set of actions, $$\mathsf{Reach}^{\max}_s(G)=\sup_{\sigma}\Pr^\sigma_s(\diamond G).$$8 a finite set of observations, $$\mathsf{Reach}^{\max}_s(G)=\sup_{\sigma}\Pr^\sigma_s(\diamond G).$$9 the transition kernel, $$f(x)_s=\sum_{t}P(s,t)x_t,\quad x|_{G}=1,\;x|_{S\setminus G}=0$$0 the observation kernel, and $$f(x)_s=\sum_{t}P(s,t)x_t,\quad x|_{G}=1,\;x|_{S\setminus G}=0$$1 the initial belief. After action $$f(x)_s=\sum_{t}P(s,t)x_t,\quad x|_{G}=1,\;x|_{S\setminus G}=0$$2 and observation $$f(x)_s=\sum_{t}P(s,t)x_t,\quad x|_{G}=1,\;x|_{S\setminus G}=0$$3, the posterior belief is

$$f(x)_s=\sum_{t}P(s,t)x_t,\quad x|_{G}=1,\;x|_{S\setminus G}=0$$4

With target set $$f(x)_s=\sum_{t}P(s,t)x_t,\quad x|_{G}=1,\;x|_{S\setminus G}=0$$5, reachability reward

$$f(x)_s=\sum_{t}P(s,t)x_t,\quad x|_{G}=1,\;x|_{S\setminus G}=0$$6

and $$f(x)_s=\sum_{t}P(s,t)x_t,\quad x|_{G}=1,\;x|_{S\setminus G}=0$$7, the expected cumulative reward equals the probability of eventually reaching $$f(x)_s=\sum_{t}P(s,t)x_t,\quad x|_{G}=1,\;x|_{S\setminus G}=0$$8: $$f(x)_s=\sum_{t}P(s,t)x_t,\quad x|_{G}=1,\;x|_{S\setminus G}=0$$9 The objective is an $\max_{\alpha}\sum_tP(s,\alpha,t)\cdot x_t$0-optimal policy $\max_{\alpha}\sum_tP(s,\alpha,t)\cdot x_t$1 such that

$\max_{\alpha}\sum_tP(s,\alpha,t)\cdot x_t$2

(Ho et al., 2024).

SVI in this setting maintains two-sided bounds

$\max_{\alpha}\sum_tP(s,\alpha,t)\cdot x_t$3

The lower bound is represented by a finite set $\max_{\alpha}\sum_tP(s,\alpha,t)\cdot x_t$4 of $\max_{\alpha}\sum_tP(s,\alpha,t)\cdot x_t$5-vectors,

$\max_{\alpha}\sum_tP(s,\alpha,t)\cdot x_t$6

seeded initially by “blind-policy” vectors or by a small finite-horizon rollout. The upper bound is represented by a point set $\max_{\alpha}\sum_tP(s,\alpha,t)\cdot x_t$7,

$\max_{\alpha}\sum_tP(s,\alpha,t)\cdot x_t$8

seeded by the fully-observable MDP solution $\max_{\alpha}\sum_tP(s,\alpha,t)\cdot x_t$9 at the simplex corners. Local Bellman backups update both sides: $S$00 Soundness is immediate by monotonicity of $S$01 and the fact that the initial seeds satisfy $S$02 (Ho et al., 2024).

The search procedure builds a directed belief-graph $S$03 rooted at $S$04. Each trial is a depth-first walk from $S$05 of maximum length $S$06. At a visited node $S$07, action selection first forms

$S$08

then selects a UCB action

$S$09

Observation selection uses weighted excess uncertainty: $S$10 The algorithm chooses the maximizing $S$11 while avoiding beliefs already visited on the current trial, expands the resulting successor if new, increments counts, and backtracks with local lower- and upper-bound backups (Ho et al., 2024).

The indefinite horizon introduces loops. The paper states that “the belief-tree is in fact a DAG with possible cycles,” so SVI merges duplicate beliefs into a single node in $S$12. During a trial, observations leading to already-visited nodes are disallowed. In addition, local upper-bound backups may stall in the undiscounted setting because of an “end-component issue.” To guarantee improvability of $S$13, every $S$14 trials the algorithm freezes frontier nodes at their hull-values and runs a full value iterate on the finite graph $S$15 until the least fixed point is reached; this “strictly decreases upper bounds inside each strongly-connected component” (Ho et al., 2024).

The convergence statement is correspondingly conditional: under the assumption that an optimal belief-based policy can be implemented with finite memory, the lower bound $S$16 after $S$17 trials converges from below to $S$18,

$S$19

Per trial, the stated complexity is

$S$20

while every $S$21 trials a global exact VI on $S$22 nodes costs

$S$23

(Ho et al., 2024).

6. Empirical behavior, adjacent methods, and interpretation

The reported empirical results consistently portray SVI as a sound alternative to plain VI that is often particularly effective on probabilistic cycles. The following summary reproduces the settings and headline findings reported in the cited sources.

Source	Setting	Reported findings
(Quatmann et al., 2018)	Storm; 130 benchmarks; $S$24	“often 20 % fewer”; “on average ~20 % faster”
(Azeem et al., 17 Sep 2025)	SG/MDP cyclic examples	retention loop $S$25: “BVI: 682 iterations; SVI: 1 iteration”; small SG: “BVI: 685 iterations; SVI: 2 iterations”
(Ho et al., 2024)	MRPP POMDP benchmarks	$S$26; “$S$27 fewer beliefs”; “$S$28 faster on large grids”

In the original 2018 evaluation, SVI was implemented in the Storm model checker and tested on “130 benchmarks (PRISM MCs/MDPs, CTMCs, Markov automata, multi-objective queries).” Compared against classical interval iteration, with and without Gauss–Seidel or topological SCC optimizations, SVI “consistently required fewer iterations (often 20 % fewer) and was on average ~20 % faster” (Quatmann et al., 2018). The 2025 SG paper reports particularly strong improvements on cyclic examples: for a “Markov chain with retention loop $S$29,” bounded VI required 682 iterations while SVI required 1 iteration, and on a “Small SG example” bounded VI required 685 iterations while SVI required 2 iterations (Azeem et al., 17 Sep 2025). In the POMDP MRPP benchmarks, the 2024 trial-based method is reported to achieve two-sided bounds $S$30 with $S$31, expand $S$32 fewer beliefs than breadth-first methods, run $S$33 faster on large grids, and converge in all benchmarks to the best known reachability probability while providing a rigorous $S$34-certificate $S$35 (Ho et al., 2024).

SVI also sits within a broader family of sound VI procedures. A closely related but distinct method is Optimistic Value Iteration (OVI), which obtains a lower bound via standard VI, guesses an upper bound by setting $S$36 for relative error or $S$37 for absolute error, and then verifies the guess using Park induction: if $S$38, then the least fixed point lies below $S$39 (Hartmanns et al., 2019). This comparison matters because it corrects another common conflation: SVI and OVI are both sound variants of value iteration, but they use different upper-bound mechanisms. The original SVI derives local bounds from the current iterates without requiring a priori start vectors (Quatmann et al., 2018), whereas OVI explicitly “guess[es]” an upper bound and proves its correctness afterward (Hartmanns et al., 2019).

Taken together, the literature presents SVI as a sequence of increasingly specialized sound fixed-point algorithms. The invariant $S$40 is shared across formulations, but the technical machinery varies with the model class: on-the-fly ratio bounds in Markov chains and MDPs, best-exit recursion and delay actions in stochastic games with end components, and $S$41-vector/hull representations with trial-based graph search in undiscounted POMDP reachability. This suggests that “SVI” is best read as a methodological pattern—sound bracketing around value iteration—whose concrete realization depends on the structure of the underlying stochastic model.