Shifted Successor Measure in RL

• Shifted Successor Measure is a matrix-valued tool in RL that bypasses short-term dynamics to emphasize steady-state behavior and unveil low-rank structures.
• It applies spectral analysis and Type II Poincaré inequalities to bound singular values and ensure effective low-rank approximations in MDPs.
• Empirical studies demonstrate that optimal shifts (typically k ≈ 3–5) improve policy accuracy and nearly double sample efficiency in complex environments.

The shifted successor measure is a matrix-valued object in reinforcement learning (RL) that facilitates low-rank representation and estimation by bypassing initial short-range dynamics and focusing on the steady-state behavior of the Markov chain induced by a policy. Formalized for Markov Decision Processes (MDPs) with state-action space $\mathcal X$ and transition kernel $P_\pi$ under a stationary policy $\pi$, the $k$-shifted successor measure is

$$M_{\pi,k} := P_\pi^k (I-\gamma P_\pi)^{-1} = \sum_{t=0}^\infty \gamma^t\, P_\pi^{t+k} \in \mathbb R^{\mathcal X \times \mathcal X}$$

where $\gamma\in(0,1)$ is the discount factor. Entrywise, $M_{\pi,k}R(s,a)$ yields the expected discounted return of a function $R$ evaluated at $(s_{t+k}, a_{t+k})$, conditioned on starting from $(s,a)$. This construction alters the spectrum and structure of the classical successor measure, exposing low-rank properties and offering significant improvements in empirical RL settings (Dubail et al., 5 Sep 2025).

1. Mathematical Construction

Given an MDP $P_\pi$0, the classical successor measure is

$P_\pi$1

with entries

$P_\pi$2

The shifted successor measure of shift $P_\pi$3 is defined as

$P_\pi$4

and can be interpreted as discounting rewards starting from time $P_\pi$5, instead of $P_\pi$6. For any reward function $P_\pi$7, the shifted expected value is

$P_\pi$8

This formalism subsumes the classical ($P_\pi$9) case as a particular instance.

2. Spectral Properties and Low-Rank Structure

Classical successor measures are typically full-rank under entrywise and spectral norms, even for small, regular MDPs. Proposition 1 (Dubail et al., 5 Sep 2025) bounds singular values $\pi$0 as

$\pi$1

with $\pi$2 the $\pi$3th singular value of $\pi$4. Since $\pi$5 retains large norm for small $\pi$6, $\pi$7 does not admit a quality low-rank approximation for small $\pi$8. However, as $\pi$9 increases, ergodicity ensures $k$0, a rank-1 object, and $k$1 becomes effectively low-rank for moderate $k$2.

Spectral recoverability $k$3, defined as

$k$4

quantifies the entrywise “concentration” of mass in the top singular vectors. Entrywise SVD-truncation error satisfies

$k$5

(Lemma 1), so rapid decay of $k$6 and $k$7 yields small truncation error for moderate $k$8. Empirically and in theory, increasing $k$9 markedly sharpens this decay, enabling low-rank approximation.

3. Finite-Sample Approximation and Estimation Analysis

Consider the empirical estimate $$M_{\pi,k} := P_\pi^k (I-\gamma P_\pi)^{-1} = \sum_{t=0}^\infty \gamma^t\, P_\pi^{t+k} \in \mathbb R^{\mathcal X \times \mathcal X}$$0 from $$M_{\pi,k} := P_\pi^k (I-\gamma P_\pi)^{-1} = \sum_{t=0}^\infty \gamma^t\, P_\pi^{t+k} \in \mathbb R^{\mathcal X \times \mathcal X}$$1 samples per transition. Theorem 1 (Dubail et al., 5 Sep 2025) bounds the SVD truncation error as

$$M_{\pi,k} := P_\pi^k (I-\gamma P_\pi)^{-1} = \sum_{t=0}^\infty \gamma^t\, P_\pi^{t+k} \in \mathbb R^{\mathcal X \times \mathcal X}$$2

with high probability. Here,

• $$M_{\pi,k} := P_\pi^k (I-\gamma P_\pi)^{-1} = \sum_{t=0}^\infty \gamma^t\, P_\pi^{t+k} \in \mathbb R^{\mathcal X \times \mathcal X}$$3 (approximation error from truncation);
• $$M_{\pi,k} := P_\pi^k (I-\gamma P_\pi)^{-1} = \sum_{t=0}^\infty \gamma^t\, P_\pi^{t+k} \in \mathbb R^{\mathcal X \times \mathcal X}$$4 captures empirical estimation error, scaling with top singular values, spectral gaps $$M_{\pi,k} := P_\pi^k (I-\gamma P_\pi)^{-1} = \sum_{t=0}^\infty \gamma^t\, P_\pi^{t+k} \in \mathbb R^{\mathcal X \times \mathcal X}$$5, maximum entrywise norms, and the sampling distribution.

The critical insight is that both error terms are governed by the spectral recoverability and singular value decay of $$M_{\pi,k} := P_\pi^k (I-\gamma P_\pi)^{-1} = \sum_{t=0}^\infty \gamma^t\, P_\pi^{t+k} \in \mathbb R^{\mathcal X \times \mathcal X}$$6, each sharply improved by appropriate shift $$M_{\pi,k} := P_\pi^k (I-\gamma P_\pi)^{-1} = \sum_{t=0}^\infty \gamma^t\, P_\pi^{t+k} \in \mathbb R^{\mathcal X \times \mathcal X}$$7.

4. Type II Poincaré Inequalities and Shift Selection

While classical Poincaré inequalities yield exponential $$M_{\pi,k} := P_\pi^k (I-\gamma P_\pi)^{-1} = \sum_{t=0}^\infty \gamma^t\, P_\pi^{t+k} \in \mathbb R^{\mathcal X \times \mathcal X}$$8–$$M_{\pi,k} := P_\pi^k (I-\gamma P_\pi)^{-1} = \sum_{t=0}^\infty \gamma^t\, P_\pi^{t+k} \in \mathbb R^{\mathcal X \times \mathcal X}$$9 norm decay only with global mixing, “Type II Poincaré” inequalities (Theorem 2) extend this to settings dominated by local mixing:

$\gamma\in(0,1)$0

for some constants $\gamma\in(0,1)$1. This leads to the bound

$\gamma\in(0,1)$2

quantifying how the off-diagonal “noise” decays in $\gamma\in(0,1)$3. A refinement in Theorem 3 involves higher-order singular values and subspace coherence:

$\gamma\in(0,1)$4

When applied to $\gamma\in(0,1)$5, these results indicate that moderate shift $\gamma\in(0,1)$6 (on the order of local mixing time) suffices for $\gamma\in(0,1)$7, exposing robust low-rank structure.

5. Connection to Local Mixing and Induced Chains

Mixing phenomena in large or structured state spaces is rarely uniform. Decomposing $\gamma\in(0,1)$8 into locally mixing components, Proposition 4 gives that if each $\gamma\in(0,1)$9 (induced chain on $M_{\pi,k}R(s,a)$0) enjoys a local Type II Poincaré gap $M_{\pi,k}R(s,a)$1, then a global inequality can be established, and the overall shift required for global low-rank structure is determined by the slowest local rate. For example, in the “4-room” environment, the local mixing time within a room dictates the minimal effective $M_{\pi,k}R(s,a)$2.

This establishes a principled, environment-dependent method for choosing $M_{\pi,k}R(s,a)$3: set $M_{\pi,k}R(s,a)$4 where $M_{\pi,k}R(s,a)$5 is the smallest local gap. In practice, scanning $M_{\pi,k}R(s,a)$6 suffices, as the optimum is frequently broad and robust.

6. Empirical Characterization and Practical Implications

Empirical studies with the Medium PointMaze, U-maze, and Large-maze environments (Dubail et al., 5 Sep 2025) demonstrate the following:

• Shifting ($M_{\pi,k}R(s,a)$7) sharpens the spectral decay of $M_{\pi,k}R(s,a)$8, dramatically reducing $M_{\pi,k}R(s,a)$9 for moderate $R$0;
• Low-rank SVD approximations of $R$1 attain peak policy accuracy (both exact and relaxed goal-reaching metrics) for $R$2 in the range $R$3–$R$4;
• Unshifted measures ($R$5) perform significantly worse, failing to distill the global structure;
• Excessive $R$6 can harm performance, especially with finite data, as estimation error from long-horizon transitions compounds;
• Substantial sample efficiency gains ($R$7) are attainable at optimal $R$8.

In practical terms, shifting the successor measure suppresses short-range, high-variance transitions and aligns the approximation with long-range, globally informative transitions, unlocking the effectiveness of low-rank function approximation in RL.

7. Summary of Theoretical and Practical Insights

The shifted successor measure $R$9 achieves two core objectives: it filters out local, short-horizon noise and reveals the low-rank structure dictated by the environment's long-range mixing properties. The optimal shift $(s_{t+k}, a_{t+k})$0 is typically small, linked to local mixing times, and admits both theoretical characterization (via spectral recoverability and Type II Poincaré inequalities) and empirical validation. The result is a robust technique enhancing both statistical and computational efficiency in goal-conditioned and reward-free RL (Dubail et al., 5 Sep 2025).