Rank-One Modified Value Iteration (R1-VI)

• Rank-One Modified Value Iteration (R1-VI) is an algorithm that uses rank-one approximations of the transition kernel to accelerate planning and reinforcement learning in Markov Decision Processes.
• It improves convergence by applying a rank-one deflation step—leveraging the stationary distribution—to reduce the error component aligned with the maximal eigenvalue.
• R1-VI is effective in both model-based and model-free settings, offering substantial speedups especially in high discount-factor scenarios and well-connected MDPs.

Rank-One Modified Value Iteration (R1-VI) is a class of algorithms for accelerated planning and reinforcement learning in Markov Decision Processes (MDPs). The central paradigm is to enhance (policy or value) iteration by replacing the transition kernel in the policy evaluation step with a rank-one approximation, typically constructed using the stationary distribution of the policy-induced Markov chain. This modification yields provably faster convergence under favorable spectral properties, often at no additional asymptotic cost per iteration. The approach appears in both planning (model-based) and learning (model-free, e.g., Q-learning) contexts, and connects directly to theoretical developments in matrix splitting and deflation from numerical linear algebra.

1. Foundations: MDP Setup and Standard Value Iteration

Consider a finite, discounted MDP defined by:

• State space $S = \{1,2,\ldots,n\}$
• Action space $A = \{1,2,\ldots,m\}$
• Transition kernel $P(s'|s,a)$, representing the probability of transitioning from state $s$ to $s'$ under action $a$
• Reward function $r: S \times A \to \mathbb{R}$
• Discount factor $\gamma \in (0,1)$

For any stationary deterministic policy $\pi: S \to A$, the induced transition matrix is $P^{\pi}_{s,s'} = P(s'|s, \pi(s))$, and the value function $A = \{1,2,\ldots,m\}$0 satisfies the Bellman equation: $A = \{1,2,\ldots,m\}$1 where $A = \{1,2,\ldots,m\}$2.

The classical planning problem is to compute the optimal value function

$A = \{1,2,\ldots,m\}$3

with the Bellman optimality operator $A = \{1,2,\ldots,m\}$4 defined as

$A = \{1,2,\ldots,m\}$5

Standard value iteration (VI) proceeds by repeatedly applying $A = \{1,2,\ldots,m\}$6: $A = \{1,2,\ldots,m\}$7 This process contracts errors at rate $A = \{1,2,\ldots,m\}$8 in $A = \{1,2,\ldots,m\}$9 norm; convergence slows substantially as $P(s'|s,a)$0.

2. Rank-One Approximation and Deflation in Policy Evaluation

Under mild ergodicity conditions, the stochastic matrix $P(s'|s,a)$1 has a unique stationary distribution $P(s'|s,a)$2 (i.e., $P(s'|s,a)$3). The rank-one approximation is constructed as

$P(s'|s,a)$4

with $P(s'|s,a)$5 denoting the all-ones vector. This is the optimal rank-one approximation with respect to the spectral radius under irreducibility and aperiodicity (Kolarijani et al., 3 May 2025). In deflation-based methods, the leading eigenspace contribution is explicitly subtracted: $P(s'|s,a)$6

The stationary distribution $P(s'|s,a)$7 can be efficiently estimated using the Power Method: $P(s'|s,a)$8 Empirically, $P(s'|s,a)$9 power method step per iteration suffices if the policy does not change rapidly.

3. Rank-One Modified Value Iteration: Algorithmic Structure

Two functionally equivalent update paradigms exist: the “policy-iteration with rank-one evaluation” form (Kolarijani et al., 3 May 2025), and the matrix-deflation SOR splitting form (Lee et al., 2024). Both produce the same core iteration up to ordering of substeps.

Policy Iteration–Style R1-VI

At iteration $s$0:

1. Construct the greedy policy $s$1 and build $s$2.
2. Estimate $s$3 (one power step).
3. Compute the Bellman update $s$4.
4. Apply the rank-one correction:

$s$5

Matrix Splitting–Style R1-VI

Given $s$6 such that $s$7:

1. Form $s$8
2. Compute $s$9

This is a direct application of Sherman–Morrison or Woodbury identities to invert $s'$0, enabling efficient closed-form correction at cost $s'$1 per iteration, matching standard VI.

Pseudocode:

$\pi: S \to A$2 (Kolarijani et al., 3 May 2025, Lee et al., 2024)

4. Theoretical Guarantees and Convergence Analysis

For policy evaluation, the rank-one deflation method removes the error component aligned with the top eigenspace. Spectral arguments yield: $s'$2 where $s'$3 is the subdominant eigenvalue of $s'$4. This yields exponentially faster convergence than standard VI ($s'$5) whenever $s'$6 (Lee et al., 2024).

In the control (greedy-improvement) context, the R1-VI iterates $s'$7 converge to the unique Bellman fixed point $s'$8 at least at linear rate $s'$9: $a$0 [(Kolarijani et al., 3 May 2025), Theorem 3.1]. The per-iteration computational complexity is $a$1.

5. Extension to Q-Learning and Model-Free Settings

The rank-one update concept extends to Q-learning by approximating the state-action transition kernel $a$2 with a rank-one matrix: $a$3 with rank-one approximation $a$4, $a$5.

For the model-free setting, empirical Bellman operators are sampled, and the Rank-One Q-Learning (R1-QL) procedure is as follows: $a$6 with $a$7 a Robbins–Monro step size [(Kolarijani et al., 3 May 2025), Algorithm 2]. Under standard stochastic approximation assumptions, $a$8 almost surely at the same sample complexity as classical Q-learning.

6. Empirical Performance and Practical Considerations

Experimental benchmarks include Garnet MDPs and random graph MDPs (Kolarijani et al., 3 May 2025), as well as chain-walk and grid-maze tasks (Lee et al., 2024).

Observed results:

• R1-VI and its rank-one deflation variants converge nearly as fast as full policy iteration.
• Substantially fewer iterations are required than VI, Nesterov-VI, Anderson-VI, or Speedy-Q-learning, particularly as $a$9.
• In model-free settings, R1-QL at least matches or outperforms Speedy-Q, Zap-Q, and standard Q-Learning for both Bellman error and value error.

Implementation and cost:

• To estimate the stationary distribution $r: S \times A \to \mathbb{R}$0 (acting as the deflation vector $r: S \times A \to \mathbb{R}$1), one can use one-step power method per iteration.
• The incremental overhead per iteration is limited to one matrix-vector multiplication and a few inner products.
• Periodic re-estimation or smoothing of $r: S \times A \to \mathbb{R}$2 can control potential instability when $r: S \times A \to \mathbb{R}$3 changes rapidly.

Table: Empirical Comparison (per (Kolarijani et al., 3 May 2025) and (Lee et al., 2024))

Method	Contraction Rate	Per-Iteration Complexity	Practical Speedup
Standard VI	$r: S \times A \to \mathbb{R}$4	$r: S \times A \to \mathbb{R}$5	Baseline
R1-VI / Defl.	$r: S \times A \to \mathbb{R}$6	$r: S \times A \to \mathbb{R}$7	Substantial, esp. $r: S \times A \to \mathbb{R}$8
Policy Iter.	Superlinear (PI)	$r: S \times A \to \mathbb{R}$9	Fastest, costly

7. Recommendations, Limitations, and Generalization

R1-VI offers maximal acceleration when:

• The discount factor $\gamma \in (0,1)$0 is close to 1
• The Markov chain induced by the current policy is well-connected (irreducible, aperiodic), ensuring effective estimation of $\gamma \in (0,1)$1
• The spectral gap $\gamma \in (0,1)$2 is non-negligible

If the Markov chain is reducible or nearly so, the stationary distribution $\gamma \in (0,1)$3 may be highly concentrated, causing the rank-one correction to lose efficacy; in this regime, increasing the number of power iterations (from $\gamma \in (0,1)$4 to $\gamma \in (0,1)$5 or $\gamma \in (0,1)$6) or adding regularization is suggested [(Kolarijani et al., 3 May 2025), Section 6.3].

For policy control tasks with changing $\gamma \in (0,1)$7, recomputing $\gamma \in (0,1)$8 or $\gamma \in (0,1)$9 every few iterations suffices. A fixed $\pi: S \to A$0 taken from the ultimate policy also yields benefits for policy evaluation phases in control-VI (Lee et al., 2024).

Memory requirements are minimal, requiring storage only of $\pi: S \to A$1 and the current greedy action indices.

In summary, Rank-One Modified Value Iteration accelerates convergence in both planning and learning at the same per-iteration complexity as classical first-order methods, and is strongly favored when the transition spectrum has pronounced spectral gap below the leading eigenvalue (Kolarijani et al., 3 May 2025, Lee et al., 2024).