---
title: Rank-One Modified Value Iteration (R1-VI)
url: https://www.emergentmind.com/topics/rank-one-modified-value-iteration-r1-vi
type: topic
---

# Rank-One Modified Value Iteration (R1-VI)

Rank-One Modified Value Iteration (R1-VI) is a class of algorithms for accelerated planning and reinforcement learning in Markov Decision Processes (MDPs). The central paradigm is to enhance (policy or value) iteration by replacing the transition kernel in the policy evaluation step with a rank-one approximation, typically constructed using the stationary distribution of the policy-induced Markov chain. This modification yields provably faster convergence under favorable spectral properties, often at no additional asymptotic cost per iteration. The approach appears in both planning (model-based) and learning (model-free, e.g., Q-learning) contexts, and connects directly to theoretical developments in matrix splitting and deflation from numerical linear algebra.

## 1. Foundations: MDP Setup and Standard Value Iteration

Consider a finite, discounted MDP defined by:
- State space $S = \{1,2,\ldots,n\}$
- Action space $A = \{1,2,\ldots,m\}$
- Transition kernel $P(s'|s,a)$, representing the probability of transitioning from state $s$ to $s'$ under action $a$
- Reward function $r: S \times A \to \mathbb{R}$
- Discount factor $\gamma \in (0,1)$

For any stationary deterministic policy $\pi: S \to A$, the induced transition matrix is $P^{\pi}_{s,s'} = P(s'|s, \pi(s))$, and the value function $V^\pi$ satisfies the Bellman equation:
\[
V^\pi = r^\pi + \gamma P^\pi V^\pi
\]
where $r^\pi(s) = r(s, \pi(s))$.

The classical planning problem is to compute the optimal value function
\[
V^* = \max_\pi V^\pi = \lim_{k\to\infty} T^k(V_0)
\]
with the Bellman optimality operator $T$ defined as
\[
[T(v)](s) = \max_{a\in A} \left\{r(s,a) + \gamma \sum_{s'} P(s'|s,a) v(s')\right\}.
\]

Standard value iteration (VI) proceeds by repeatedly applying $T$:
\[
V_{k+1} = T(V_k).
\]
This process contracts errors at rate $O(\gamma^k)$ in $\ell_\infty$ norm; convergence slows substantially as $\gamma \to 1$.

## 2. Rank-One Approximation and Deflation in Policy Evaluation

Under mild ergodicity conditions, the stochastic matrix $P^\pi$ has a unique stationary distribution $d \in \Delta(S)$ (i.e., $d^\top = d^\top P^\pi$). The rank-one approximation is constructed as
\[
P^\pi \approx \bm{1} d^\top
\]
with $\bm{1}$ denoting the all-ones vector. This is the optimal rank-one approximation with respect to the spectral radius under irreducibility and aperiodicity [2505.01828]. In deflation-based methods, the leading eigenspace contribution is explicitly subtracted:
\[
E = \lambda_1 u v^\top,\ \text{where typically}\ \lambda_1 = 1,\ u = \bm{1},\ v = d.
\]

The stationary distribution $d$ can be efficiently estimated using the Power Method:
\[
f = P^\pi d^{(i)};\quad d^{(i+1)} = \frac{f}{\sum_j f_j}.
\]
Empirically, $I=1$ power method step per iteration suffices if the policy does not change rapidly.

## 3. Rank-One Modified Value Iteration: Algorithmic Structure

Two functionally equivalent update paradigms exist: the “policy-iteration with rank-one evaluation” form [2505.01828], and the matrix-deflation SOR splitting form [2407.10454]. Both produce the same core iteration up to ordering of substeps.

### Policy Iteration–Style R1-VI

At iteration $k$:
1. Construct the greedy policy $\pi_k(s) = \arg\max_a\{r(s,a) + \gamma \sum_{s'} P(s'|s,a) V_k(s')\}$ and build $P_k$.
2. Estimate $d_k \approx P_k d_{k-1}$ (one power step).
3. Compute the Bellman update $T(V_k)$.
4. Apply the rank-one correction:
   \[
   V_{k+1} = T(V_k) + \frac{\gamma}{1-\gamma} \langle d_k, T(V_k) - V_k \rangle \bm{1}
   \]

### Matrix Splitting–Style R1-VI

Given $v$ such that $v^\top \bm{1} = 1$:
1. Form $W = r^\pi + \gamma (P^\pi V^k - (v^\top V^k) \bm{1})$
2. Compute $V^{k+1} = W + \frac{\gamma}{1-\gamma} \bm{1} (v^\top W)$

This is a direct application of Sherman–Morrison or Woodbury identities to invert $I - \gamma E$, enabling efficient closed-form correction at cost $O(n^2 m)$ per iteration, matching standard VI.

### Pseudocode:

```matlab
initialize V, d_{-1} in Δ(S)
for k = 0,1,...
    compute greedy policy π_k, build P_k
    d_k ← P_k d_{k-1} / sum(P_k d_{k-1})
    T_vk ← T(V_k)
    V_{k+1} ← T_vk + (γ/(1-γ)) * <d_k, T_vk - V_k> * 1
end
```
[2505.01828], [2407.10454]

## 4. Theoretical Guarantees and Convergence Analysis

For policy evaluation, the rank-one deflation method removes the error component aligned with the top eigenspace. Spectral arguments yield:
\[
\| V^k - V^\pi \| = O\left((\gamma |\lambda_2|)^k\right)
\]
where $|\lambda_2|$ is the subdominant eigenvalue of $P^\pi$. This yields exponentially faster convergence than standard VI ($O(\gamma^k)$) whenever $|\lambda_2| < 1$ [2407.10454].

In the control (greedy-improvement) context, the R1-VI iterates $\{V_k\}$ converge to the unique Bellman fixed point $V^*$ at least at linear rate $\gamma$:
\[
\| V_{k+1} - V^* \|_\infty \leq \gamma^{k+1} \left( \| V_0 - V^* \|_\infty + \frac{1}{1-\gamma} \|T(V_0) - V_0\|_\infty \right)
\]
[2505.01828, Theorem 3.1]. The per-iteration computational complexity is $O(n^2 m)$.

## 5. Extension to Q-Learning and Model-Free Settings

The rank-one update concept extends to Q-learning by approximating the state-action transition kernel ${\cal P}_k$ with a rank-one matrix:
\[
Q_{k+1} = Q_k + (I - \gamma {\cal P}_k)^{-1} ({\cal T}(Q_k) - Q_k)
\]
with rank-one approximation ${\cal P}_k \approx \bm{1} d_k^\top$, $d_k \in \Delta(S \times A)$.

For the model-free setting, empirical Bellman operators are sampled, and the Rank-One Q-Learning (R1-QL) procedure is as follows:
\[
\begin{cases}
d_k \gets \text{one-step power method on estimated } {\cal P}_k \\
\alpha_k = \frac{\gamma \lambda_k}{1-\gamma} \langle d_k, \widehat{\cal T}_k(Q_k) - Q_k \rangle \\
Q_{k+1} = (1-\lambda_k) Q_k + \lambda_k \widehat{\cal T}_k(Q_k) + \alpha_k \bm{1}
\end{cases}
\]
with $\lambda_k$ a Robbins–Monro step size [2505.01828, Algorithm 2]. Under standard stochastic approximation assumptions, $Q_k \to Q^*$ almost surely at the same sample complexity as classical Q-learning.

## 6. Empirical Performance and Practical Considerations

Experimental benchmarks include Garnet MDPs and random graph MDPs [2505.01828], as well as chain-walk and grid-maze tasks [2407.10454].

**Observed results:**
- R1-VI and its rank-one deflation variants converge nearly as fast as full policy iteration.
- Substantially fewer iterations are required than VI, Nesterov-VI, Anderson-VI, or Speedy-Q-learning, particularly as $\gamma \to 1$.
- In model-free settings, R1-QL at least matches or outperforms Speedy-Q, Zap-Q, and standard Q-Learning for both Bellman error and value error.

**Implementation and cost:**
- To estimate the stationary distribution $d$ (acting as the deflation vector $v$), one can use one-step power method per iteration.
- The incremental overhead per iteration is limited to one matrix-vector multiplication and a few inner products.
- Periodic re-estimation or smoothing of $v$ can control potential instability when $P^\pi$ changes rapidly.

**Table: Empirical Comparison (per [2505.01828] and [2407.10454])**

| Method         | Contraction Rate         | Per-Iteration Complexity | Practical Speedup     |
|----------------|-------------------------|--------------------------|-----------------------|
| Standard VI    | $\gamma^k$              | $O(n^2 m)$               | Baseline              |
| R1-VI / Defl.  | $(\gamma |\lambda_2|)^k$| $O(n^2 m)$               | Substantial, esp. $\gamma\to1$ |
| Policy Iter.   | Superlinear (PI)        | $O(n^3)$                 | Fastest, costly       |

## 7. Recommendations, Limitations, and Generalization

R1-VI offers maximal acceleration when:
- The discount factor $\gamma$ is close to 1
- The Markov chain induced by the current policy is well-connected (irreducible, aperiodic), ensuring effective estimation of $d$
- The spectral gap $1 - |\lambda_2|$ is non-negligible

If the Markov chain is reducible or nearly so, the stationary distribution $d$ may be highly concentrated, causing the rank-one correction to lose efficacy; in this regime, increasing the number of power iterations (from $I=1$ to $2$ or $3$) or adding regularization is suggested [2505.01828, Section 6.3].

For policy control tasks with changing $P^\pi$, recomputing $d$ or $v$ every few iterations suffices. A fixed $v$ taken from the ultimate policy also yields benefits for policy evaluation phases in control-VI [2407.10454].

Memory requirements are minimal, requiring storage only of $d_{k-1}$ and the current greedy action indices.

In summary, Rank-One Modified Value Iteration accelerates convergence in both planning and learning at the same per-iteration complexity as classical first-order methods, and is strongly favored when the transition spectrum has pronounced spectral gap below the leading eigenvalue [2505.01828, 2407.10454].

Source: https://www.emergentmind.com/topics/rank-one-modified-value-iteration-r1-vi