---
title: Shifted Successor Measure in RL
url: https://www.emergentmind.com/topics/shifted-successor-measure
type: topic
---

# Shifted Successor Measure in RL

The shifted successor measure is a matrix-valued object in reinforcement learning (RL) that facilitates low-rank representation and estimation by bypassing initial short-range dynamics and focusing on the steady-state behavior of the Markov chain induced by a policy. Formalized for Markov Decision Processes (MDPs) with state-action space $\mathcal X$ and transition kernel $P_\pi$ under a stationary policy $\pi$, the $k$-shifted successor measure is
$$
M_{\pi,k} := P_\pi^k (I-\gamma P_\pi)^{-1} = \sum_{t=0}^\infty \gamma^t\, P_\pi^{t+k} \in \mathbb R^{\mathcal X \times \mathcal X}
$$
where $\gamma\in(0,1)$ is the discount factor. Entrywise, $M_{\pi,k}R(s,a)$ yields the expected discounted return of a function $R$ evaluated at $(s_{t+k}, a_{t+k})$, conditioned on starting from $(s,a)$. This construction alters the spectrum and structure of the classical successor measure, exposing low-rank properties and offering significant improvements in empirical RL settings [2509.05193].

## 1. Mathematical Construction

Given an MDP $(\mathcal X, P_\pi, \gamma)$, the classical successor measure is
$$
M_\pi = (I - \gamma P_\pi)^{-1}
$$
with entries
$$
M_\pi((s,a), (s',a')) = \sum_{t=0}^\infty \gamma^t \Pr(s_t = s', a_t = a' \mid s_0 = s, a_0 = a).
$$
The shifted successor measure of shift $k \geq 0$ is defined as
$$
M_{\pi,k} = P_\pi^k (I-\gamma P_\pi)^{-1} = \sum_{t=0}^\infty \gamma^t\, P_\pi^{t+k},
$$
and can be interpreted as discounting rewards starting from time $k$, instead of $0$. For any reward function $R$, the shifted expected value is
$$
M_{\pi,k} R(s,a) = \sum_{t=0}^\infty \gamma^t\,\mathbb E_\pi [ R(s_{t+k}, a_{t+k})\mid s_0 = s, a_0 = a ].
$$
This formalism subsumes the classical ($k=0$) case as a particular instance.

## 2. Spectral Properties and Low-Rank Structure

Classical successor measures are typically full-rank under entrywise and spectral norms, even for small, regular MDPs. Proposition 1 [2509.05193] bounds singular values $\sigma_i(M_{\pi,k})$ as
$$
\frac{\sigma_i(P_\pi^k)}{1+\gamma} \leq \sigma_i(M_{\pi,k}) \leq \frac{\sigma_i(P_\pi^k)}{1-\gamma},
$$
with $\sigma_i(P_\pi^k)$ the $i$th singular value of $P_\pi^k$. Since $P_\pi^k$ retains large norm for small $k$, $M_\pi$ does not admit a quality low-rank approximation for small $k$. However, as $k$ increases, ergodicity ensures $P_\pi^k \to \mathbf 1 \nu_{\rm inv}$, a rank-1 object, and $M_{\pi,k}$ becomes effectively low-rank for moderate $k$.

Spectral recoverability $\xi(M)$, defined as
$$
\xi(M) := \max_{x\in\mathcal X} \sum_{i=1}^n \sigma_i\,\psi_i(x)^2,
$$
quantifies the entrywise “concentration” of mass in the top singular vectors. Entrywise SVD-truncation error satisfies
$$
\| M - [M]_r \|_{2,\infty} \leq \sqrt{\sigma_{r+1} \xi(M) }
$$
(Lemma 1), so rapid decay of $\sigma_{r+1}(M_{\pi,k})$ and $\xi(M_{\pi,k})$ yields small truncation error for moderate $r$. Empirically and in theory, increasing $k$ markedly sharpens this decay, enabling low-rank approximation.

## 3. Finite-Sample Approximation and Estimation Analysis

Consider the empirical estimate $\widehat M_{\pi,k}$ from $Z_{s,a}$ samples per transition. Theorem 1 [2509.05193] bounds the SVD truncation error as
$$
\| [\widehat M_{\pi, k}]_r - M_{\pi, k} \|_{2, \infty} \leq C \left(\mathcal E_{\rm approx}+\mathcal E_{\rm estim}\right)
$$
with high probability. Here,
- $\mathcal E_{\rm approx} = \sqrt{\sigma_{r+1}(M_{\pi,k}) \xi(M_{\pi,k})}$ (approximation error from truncation);
- $\mathcal E_{\rm estim}$ captures empirical estimation error, scaling with top singular values, spectral gaps $\sigma_r - \sigma_{r+1}$, maximum entrywise norms, and the sampling distribution.

The critical insight is that both error terms are governed by the spectral recoverability and singular value decay of $M_{\pi,k}$, each sharply improved by appropriate shift $k$.

## 4. Type II Poincaré Inequalities and Shift Selection

While classical Poincaré inequalities yield exponential $2$–$\infty$ norm decay only with global mixing, “Type II Poincaré” inequalities (Theorem 2) extend this to settings dominated by local mixing:
$$
\lambda \|f\|_2^2 \leq \mathcal E_{P P^\dagger}(f,f) + C \lambda \|f\|_1^2
$$
for some constants $\lambda, C \geq 0$. This leads to the bound
$$
\|P^k\|_{2, \infty}^2 \leq (\nu_{\min}^{-1} - C) (1-\lambda)^k + C,
$$
quantifying how the off-diagonal “noise” decays in $P^k$. A refinement in Theorem 3 involves higher-order singular values and subspace coherence:
$$
\frac{1-\sigma_{r+1}^2}{2}\|f\|_2^2 \leq \mathcal E_{P P^\dagger}(f,f) + (1-\sigma_{r+1}^2)\|U_r\|_{2,\infty}^2\|f\|_1^2.
$$
When applied to $M_{\pi,k}$, these results indicate that moderate shift $k$ (on the order of local mixing time) suffices for $\xi(M_{\pi,k})=O(1)$, exposing robust low-rank structure.

## 5. Connection to Local Mixing and Induced Chains

Mixing phenomena in large or structured state spaces is rarely uniform. Decomposing $\mathcal X = S_1 \cup \cdots \cup S_m$ into locally mixing components, Proposition 4 gives that if each $P_{S_i}$ (induced chain on $S_i$) enjoys a local Type II Poincaré gap $\lambda_i$, then a global inequality can be established, and the overall shift required for global low-rank structure is determined by the slowest local rate. For example, in the “4-room” environment, the local mixing time within a room dictates the minimal effective $k$.

This establishes a principled, environment-dependent method for choosing $k$: set $k \approx \frac{\log(\nu_{\min}^{-1} r)}{\lambda_{\rm loc}}$ where $\lambda_{\rm loc}$ is the smallest local gap. In practice, scanning $k\in[1,10]$ suffices, as the optimum is frequently broad and robust.

## 6. Empirical Characterization and Practical Implications

Empirical studies with the Medium PointMaze, U-maze, and Large-maze environments [2509.05193] demonstrate the following:
- Shifting ($k>0$) sharpens the spectral decay of $M_{\pi,k}$, dramatically reducing $\sigma_{r+1}$ for moderate $k$;
- Low-rank SVD approximations of $M_{\pi,k}$ attain peak policy accuracy (both exact and relaxed goal-reaching metrics) for $k$ in the range $3$–$5$;
- Unshifted measures ($k=0$) perform significantly worse, failing to distill the global structure;
- Excessive $k$ can harm performance, especially with finite data, as estimation error from long-horizon transitions compounds;
- Substantial sample efficiency gains ($\sim 2\times$) are attainable at optimal $k$.

In practical terms, shifting the successor measure suppresses short-range, high-variance transitions and aligns the approximation with long-range, globally informative transitions, unlocking the effectiveness of low-rank function approximation in RL.

## 7. Summary of Theoretical and Practical Insights

The shifted successor measure $M_{\pi,k}$ achieves two core objectives: it filters out local, short-horizon noise and reveals the low-rank structure dictated by the environment's long-range mixing properties. The optimal shift $k$ is typically small, linked to local mixing times, and admits both theoretical characterization (via spectral recoverability and Type II Poincaré inequalities) and empirical validation. The result is a robust technique enhancing both statistical and computational efficiency in goal-conditioned and reward-free RL [2509.05193].

Source: https://www.emergentmind.com/topics/shifted-successor-measure