Cross-Episode Credit Assignment

• Cross-episode credit assignment is a mechanism in reinforcement learning that credits long-range actions across multiple episodes.
• It uses approaches like Predecessor Features and meta-policy gradients to propagate delayed reward signals effectively.
• These techniques improve learning in sparse reward environments and multi-agent tasks despite increased computational demands.

Cross-episode credit assignment refers to mechanisms within reinforcement learning (RL) that attribute observed outcomes not merely to recent actions or states within a single episode, but to a broader and richer set of possible predecessors, including those drawn from preceding episodes or expected across the agent's historical trajectory distribution. This contrasts with traditional within-episode credit assignment, as found in standard eligibility traces or temporal-difference (TD) learning, which limit the propagation of reward information primarily to temporally local or contiguous events. Recent research formalizes cross-episode credit assignment both in model-free RL with Predecessor Features (Bailey et al., 2022) and in multi-agent reinforcement learning (MARL) with meta-learning-based objectives such as the Meta-Policy Gradient for Mixing Networks (MNMPG) (Shao et al., 2021).

1. Conceptual Foundations

In classical RL frameworks, credit assignment is addressed by associating reward signals with the actions or states that most likely contributed to them. Eligibility traces, as employed in TD(λ), allocate TD-errors to a decaying window of past states within the same episode. However, realistic tasks frequently exhibit long time horizons or causal dependencies that span beyond readily observed local trajectories. Cross-episode credit assignment generalizes this by allowing the RL system to assign temporal credit to states or actions that may have occurred long before the observed outcome or even in episodes prior to the current one. Two foundational approaches to cross-episode credit assignment include Predecessor Features, which learns expected traces across episodes, and meta-learning frameworks in MARL, which exploit meta-objectives evaluating the impact of parameter adjustments on future episode returns.

2. Predecessor Features and Expected Trace Formulation

The Predecessor Features approach formalizes cross-episode credit assignment as learning an expected trace z(s) for each state s, defined as the expectation over all predecessor occupancies that could plausibly have led to s under the current or stationary policy. For feature representation x(s) ∈ ℝ^d, the Predecessor Feature vector is:

$$z(s) = \mathbb{E}\left[e_t \mid S_t = s\right] = \mathbb{E}\left[\sum_{n=0}^\infty (\gamma \lambda)^n x(S_{t-n}) \;\middle|\; S_t = s \right]$$

This expectation aggregates the discounted sum of all feature vectors for states that might precede s, weighted appropriately. In contrast to forward-looking Successor Representation (SR), which focuses on future states, Predecessor Features assign TD-errors to the expected set of past states or features, enabling assignment of credit to states not directly visited in the current episode but known (in expectation) to have contributed causally to high-value states (Bailey et al., 2022).

The learning protocol involves bootstrapped temporal-difference updates not only for value weights w, but also for the parameters of the mapping function z_θ(s) (or the matrix Ψ in the linear case). The bootstrapped target for z(s) satisfies a Bellman-like recursion:

$$z(s_t) = x(s_t) + \gamma \lambda \mathbb{E}[z(S_{t-1}) | S_t = s_t]$$

Updates proceed as:

• Value update: $w \leftarrow w + \alpha \delta_t z_\theta(S_t)$
• Predecessor feature update: $$\theta \leftarrow \theta - \beta (z_\theta(S_t) - y_t) \nabla_\theta z_\theta(S_t)$$ where $y_t = x(S_t) + \gamma \lambda z_\theta(S_{t-1})$

Because z_θ(s) is improved across episodes, the learned expected trace allows credit to propagate along plausible historical paths, not just within the latest sampled episode (Bailey et al., 2022).

3. Meta-Policy Gradient in Multi-Agent Credit Assignment

In multi-agent RL, particularly under the Centralized Training with Decentralized Execution (CTDE) paradigm, cross-episode credit assignment has been advanced through meta-learning objectives that evaluate parameter updates in the context of their influence on subsequent episode returns. The MNMPG algorithm (Shao et al., 2021) treats the parameters θᵤ of the mixing network (which combines agents' utilities) as meta-actions, and measures the effect of an "exercise update" (Q-learning step) on θᵤ, followed by a rollout with the updated parameters θᵤ′.

The cross-episode mechanism is as follows:

1. Exercise Update: Collect a trajectory D₀ under θᵤ; compute the exercise loss $L_{exercise}(\theta_u; D_0)$; update θᵤ → θᵤ′ via gradient step(s).
2. Excitation Signal: Run a new episode D₁ under θᵤ′, compute returns R(θᵤ′) and R(θᵤ) for D₁ and D₀ respectively, and define the meta-reward ΔR = R(θᵤ′) – R(θᵤ).
3. Meta-Objective: The meta-objective is $$J_{meta}(\theta_u) = \mathbb{E}_{D_0 \sim \pi_{\theta_u}}[\Delta R]$$. Its REINFORCE-style gradient:

$$\nabla_{\theta_u} J_{meta} = \mathbb{E}_{D_0} [\Delta R \cdot \sum_{t=1}^T \nabla_{\theta_u} \log \pi(z_t | s_t; \theta_u)]$$

This formalism propagates credit not just within an episode but across the episodic boundary, reinforcing updates that demonstrably improve future episode returns. The training loop alternates conventional TD-learning with these cross-episode meta-updates (Shao et al., 2021).

4. Distinctions Between Within-Episode and Cross-Episode Assignment

Traditional within-episode assignment, exemplified by TD(λ) or standard mixing-network training, propagates TD-errors to recently experienced states using sample-based eligibility traces. All credit resides within the temporally local context of a given episode. In contrast, cross-episode assignment, as realized by Predecessor Features or MNMPG, either

• propagates TD-errors to all potential predecessor states or features regardless of when (or whether) they were visited in the current episode, by maintaining and updating an expected trace that persists across episodes; or
• evaluates the causal effect of parameter changes (in e.g., a mixing network) with respect to their influence on future episodes' outcomes, rather than only on immediate loss reduction.

A key practical implication is that cross-episode methods can accelerate learning in domains with sparse or delayed rewards, or in tasks with long causal chains, by avoiding the slow "random walk" propagation of credit inherent in purely local techniques (Bailey et al., 2022). However, these advantages come with potential drawbacks, such as increased computational and sample cost (due to multiple full rollouts per meta-step in MNMPG) and increased variance in meta-gradients (from REINFORCE-style updates) (Shao et al., 2021).

5. Empirical Analyses and Hyperparameter Regimes

Empirical evaluation of Predecessor Features on tasks such as a tabular 6×6 “Plinko” grid and deep-cartpole demonstrate that the fully bootstrapped expected-trace approach converges more rapidly and robustly than TD(λ), particularly when learning parameters for the expected trace are properly tuned (Bailey et al., 2022). In multi-agent settings, experiments on StarCraft II micromanagement “super-hard” maps reveal that MNMPG increases win rates from approximately 50% (QMIX baseline) to over 90%, attributed to more focused exploration and robust discovery of high-reward states by exploiting the learned cross-episode global hierarchy (Shao et al., 2021).

Notable hyperparameter settings for MNMPG include:

Parameter	Typical Value in MNMPG	Description
Inner-loop learning rate α	$5 \times 10^{-4}$	Step-size for exercise update
Meta-learning rate β	$1 \times 10^{-3}$	Step-size for meta-gradient update
Number of exercise steps K	1–5 (often 1)	Inner-loop gradient steps per meta-iteration
Meta-update frequency	Every 500 env steps	Control over cost vs. adaptation speed
Global hierarchy dimension z	3	Dimensionality of latent variable

These regimes are reported to yield significant performance improvements in MARL benchmarks (Shao et al., 2021).

6. Implications, Limitations, and Research Directions

Cross-episode credit assignment mechanisms enlarge the class of RL tasks amenable to efficient solution, especially where long-term dependencies span episode boundaries or when reward signals are both sparse and delayed. Predecessor Features enable assignment of TD-errors to paths or features not observed in the current episode but inferred through a persistently learned expected occupancy structure. MNMPG enables causal attribution across episodes by linking parameter changes to subsequent episodic returns.

Key limitations include:

• Computational cost: Both approaches may require increased computation or parameter storage (e.g., O(d²) in PF for high-dimensional features or multiple rollouts per meta-iteration in MNMPG).
• Variance and Stability: Meta-gradient estimation via REINFORCE can introduce significant variance, while bootstrapped updates may bias estimates in PF. Robustness may depend on effective alternation of standard and meta-updates, as well as function approximator expressivity (Bailey et al., 2022, Shao et al., 2021).
• Approximation error: Misspecification or insufficient capacity in the predecessor mapping or mixing network can undermine assignment accuracy.

Possible research directions suggested in existing work include generalized ET(λ,η) interleavings for PF to balance bias-variance, integration with off-policy methods, incorporation of attention mechanisms for richer predecessor modeling, and further advances in meta-learning protocols for multi-agent or hierarchical RL (Bailey et al., 2022).

7. Summary of Principal Methods

A concise comparison of within-episode and cross-episode credit assignment architectures:

Approach	Core Mechanism	Cross-Episode Credit?
TD(λ), standard mixing network	Sample eligibility traces, episode-local TD-error assignment	No
Predecessor Features (PF)	Expected trace z(s), bootstrapped across episodes	Yes
MNMPG in CTDE (MARL)	Meta-gradient on mixing network by exercise/return difference	Yes

Cross-episode credit assignment generalizes and strictly exceeds within-episode methods in its capacity for long-range, efficient propagation of value information, supporting rapid convergence and improved robustness in challenging RL domains (Bailey et al., 2022, Shao et al., 2021).