n-Step Return in RL & Random Walks

• n-step return is a measure quantifying the probability or cumulative reward obtained after exactly n steps, essential in Markov process analysis and RL.
• It bridges Monte Carlo and temporal-difference methods, enabling a bias–variance trade-off that fine-tunes value estimation for diverse applications.
• It underpins efficient computational algorithms in network analysis and RL, facilitating multi-step updates and improved convergence properties.

An n-step return, also referred to in some contexts as k-step return, is a foundational tool in the analysis of Markov processes, random walks on networks, and reinforcement learning (RL) algorithms. It captures the probability or expected value associated with returning to a particular state—often the origin—after exactly n steps, or accumulates rewards and bootstraps value estimates over a finite trajectory segment before resorting to bootstrapping. The n-step return unifies key concepts across probability theory, spectral graph analysis, and modern RL, and underpins a wide variety of bias–variance trade-offs and algorithmic innovations.

1. Definition and Mathematical Foundations

In Markov processes and random walks, the n-step return probability is typically defined as the probability that a process returns to its starting state after exactly n steps. For a random walk $\{X_i\}_{i\ge1}$ on $\mathbb{Z}$, with increments distributed according to $p_k = P(X_i = k)$, the n-step return probability $r(n)$ is:

$r(n) = P(S_n = 0) = p^{*n}(0)$

where $S_n = X_1 + \dots + X_n$ and $p^{*n}$ denotes the n-fold convolution of the step distribution (Zhou, 2015).

On undirected graphs, the n-step return probability at vertex $i$ is $P_{ii}^{(n)}$, i.e., the probability that a random walk starting at $i$ is back at $i$ after n steps. In RL, the n-step return $G_t^{(n)}$ is

$$G_t^{(n)} = \sum_{k=0}^{n-1} \gamma^k r_{t+k} + \gamma^n V(s_{t+n})$$

where $r_{t+k}$ are observed rewards, $\gamma$ is the discount factor, and $V$ is a value estimate (Mandal et al., 2023, Daley et al., 2024). The return, therefore, interpolates between pure Monte Carlo (large n) and temporal-difference (TD) bootstrapping (n=1).

2. Analytical Properties in Probability and Spectral Theory

The n-step return probability on integer lattices and graphs admits several analytic forms. For random walks on $\mathbb{Z}$, the ordinary generating function $R(z) = \sum_{n=0}^\infty r(n)z^n$ satisfies

$$R(z) = \frac{1}{2\pi}\int_{-\pi}^\pi \frac{d\theta}{1 - z \varphi(\theta)}$$

where $\varphi(\theta)$ is the characteristic function of the step distribution. Symmetry and primitivity of $p_k$ are required for the sequence $\{r(n)\}$ to uniquely determine the walk (Zhou, 2015).

For random walks on transitive graphs, the spectral radius $\rho$ constrains the normalized return probabilities $a_n = u_n/\rho^n$, with sharp asymptotic bounds such as $a_n = O(n^{-3/2})$ in amenable, non-unimodular cases (Tang, 2021). Ballot theorem–type estimates and Doob $h$-transforms are used to obtain such local-limit results.

Closed-form expressions for return and survival probabilities in 1D biased random walks have recently been developed for arbitrary $N$, involving hypergeometric functions and explicit dependence on the bias parameter (Mookerjee et al., 2024).

3. Computational Aspects and Algorithms

Efficient calculation of n-step return distributions on graphs is critical for scalable network analysis and RL. For simple connected undirected graphs, Dronen & Lv (Dronen et al., 2011) define a modified matrix powering procedure:

• $P^{(1)} = P$ (transition probability matrix)
• For $k\ge2$, $P^{(k)} = \text{zd}(P^{(k-1)}) P$, where $\text{zd}(M)$ zeros out the diagonal of $M$.

$P^{(k)}_{ii}$ gives the probability that a random walk started at $i$ returns to $i$ at time $k$ for the first time. Their O($n+m$) algorithm enables linear-time computation for k=2 (the Pólya Power Index), and the full distribution for small k.

In RL, implementation hinges on stacking reward and value estimates over multiple timesteps, resulting in multistep targets used in TD learning, actor-critic, and policy-gradient algorithms (Daley et al., 2024, Yang et al., 2021, Łyskawa et al., 15 Dec 2025).

4. Bias–Variance Trade-offs and Compound Returns

The n-step return parameterizes a fundamental bias–variance trade-off. Small n yields high bias/low variance (fast, albeit myopic, propagation of value information), while large n gives Monte Carlo estimates with low bias but potentially high variance and delayed learning signal. Analytical models (assuming equal-variance, possibly correlated TD errors) yield

$$\operatorname{Var}[G_t^{(n)}|S_t] = (1-\rho) \Gamma^{(2)}_n \kappa + \rho (\Gamma^{(1)}_n)^2 \kappa$$

where $\Gamma^{(c)}_n = \sum_{i=0}^{n-1} \gamma^{ci}$ and $\rho$ models TD-error correlation (Daley et al., 2024). Compound returns—convex combinations of multiple n-step returns—strictly reduce variance at fixed contraction modulus. Piecewise λ-returns (averages of two or more n-step returns) inherit this benefit and offer practical variance reduction for deep RL agents (Daley et al., 2024).

5. n-Step Returns in Advanced RL: Algorithms and Applications

Modern RL leverages n-step returns in mechanisms such as:

• n-step temporal difference (TD) learning: Value update via $V(S_t) \gets V(S_t) + \alpha (G_t^{(n)} - V(S_t))$ (Mandal et al., 2023).
• Knowledge distillation: KETCHUP applies k-step returns to reduce policy-gradient variance in RL-based sequential KD, leveraging multi-step Bellman expansions and telescoping teacher Q-values (Fan et al., 26 Apr 2025).
• Multi-step Hindsight Experience Replay (MHER): n-step relabeling accelerates learning in sparse-reward, multi-goal RL, but suffers from off-policy bias unless mitigated by λ-return averaging (MHER(λ)), model-based expansions (MMHER), or standard importance sampling (Yang et al., 2021).
• Soft Actor-Critic with n-step returns (SACₙ): Combines multi-step returns, maximal-entropy regularization, and numerically stabilized importance sampling for efficient off-policy RL (Łyskawa et al., 15 Dec 2025). The use of τ-sampled entropy estimators and batch-wise clipped importance weights is pivotal for stability and sample efficiency.

Algorithmic selection of optimal n can be formulated as a discrete stochastic optimization problem, minimizing RMSE via techniques such as one-simulation SPSA, robustly adapting n online (Mandal et al., 2023).

6. Connections to Centrality Measures in Networks

n-step return probabilities generalize classic network centrality metrics. The k-step return distribution captures local and nonlocal node prominence. The 2-step return (Pólya Power Index, PPI) is equivalent to negative-β beta centrality and relates closely to graph-theoretical power indices like GPI (Dronen et al., 2011). Higher-k return distributions approximate subgraph centrality, offering fine-grained, computationally efficient centrality measures for large-scale networks.

Empirical runtime comparisons confirm the advantage of k-step return methods—the calculation of 10-step distributions is up to twice as fast as subgraph centrality via eigendecomposition on networks with 1,000–2,000 nodes (Dronen et al., 2011).

7. Implications, Examples, and Limitations

n-step returns enable precise characterizations of random walks and learning algorithms:

• In random walks, sequences $\{r(n)\}$ can, under symmetry and primitivity, uniquely determine the underlying random walk law (Zhou, 2015).
• On transitive, non-unimodular graphs, local return probabilities decay as $n^{-3/2}$, with explicit dependence on geometric and group-theoretic structure; examples include Diestel–Leader graphs, Port–trees, and Cartesian products of certain graphs (Tang, 2021).
• Closed-form recurrence and last-return probabilities for biased random walks yield quantitative predictions for biological molecular motors and illuminate transitions in return-time distributions as a function of bias (Mookerjee et al., 2024).
• In all RL settings, careful tuning or dynamic adaptation of n is critical. Excessive n risks destabilizing learning via increased variance, but in high-discount or reward-sparse tasks, moderate n delivers substantial improvements in convergence and policy quality (Daley et al., 2024, Łyskawa et al., 15 Dec 2025, Fan et al., 26 Apr 2025).

Compound returns, λ-returns, and carefully designed multi-step estimators exploit the full bias–variance interplay, and advances in their theoretical analysis and practical instantiation continue to shape the state of RL methodology and large-scale network analysis (Daley et al., 2024, Łyskawa et al., 15 Dec 2025, Yang et al., 2021).