---
title: Cross-Episode Credit Assignment
url: https://www.emergentmind.com/topics/cross-episode-credit-assignment
type: topic
---

# Cross-Episode Credit Assignment

Cross-episode credit assignment refers to mechanisms within reinforcement learning (RL) that attribute observed outcomes not merely to recent actions or states within a single episode, but to a broader and richer set of possible predecessors, including those drawn from preceding episodes or expected across the agent's historical trajectory distribution. This contrasts with traditional within-episode credit assignment, as found in standard eligibility traces or temporal-difference (TD) learning, which limit the propagation of reward information primarily to temporally local or contiguous events. Recent research formalizes cross-episode credit assignment both in model-free RL with Predecessor Features [2206.00303] and in multi-agent reinforcement learning (MARL) with meta-learning-based objectives such as the Meta-Policy Gradient for Mixing Networks (MNMPG) [2102.12957].

## 1. Conceptual Foundations

In classical RL frameworks, credit assignment is addressed by associating reward signals with the actions or states that most likely contributed to them. Eligibility traces, as employed in TD(λ), allocate TD-errors to a decaying window of past states within the same episode. However, realistic tasks frequently exhibit long time horizons or causal dependencies that span beyond readily observed local trajectories. Cross-episode credit assignment generalizes this by allowing the RL system to assign temporal credit to states or actions that may have occurred long before the observed outcome or even in episodes prior to the current one. Two foundational approaches to cross-episode credit assignment include Predecessor Features, which learns expected traces across episodes, and meta-learning frameworks in MARL, which exploit meta-objectives evaluating the impact of parameter adjustments on future episode returns.

## 2. Predecessor Features and Expected Trace Formulation

The Predecessor Features approach formalizes cross-episode credit assignment as learning an expected trace z(s) for each state s, defined as the expectation over all predecessor occupancies that could plausibly have led to s under the current or stationary policy. For feature representation x(s) ∈ ℝ^d, the Predecessor Feature vector is:

$$
z(s) = \mathbb{E}\left[e_t \mid S_t = s\right] = \mathbb{E}\left[\sum_{n=0}^\infty (\gamma \lambda)^n x(S_{t-n}) \;\middle|\; S_t = s \right]
$$

This expectation aggregates the discounted sum of all feature vectors for states that might precede s, weighted appropriately. In contrast to forward-looking Successor Representation (SR), which focuses on future states, Predecessor Features assign TD-errors to the expected set of past states or features, enabling assignment of credit to states not directly visited in the current episode but known (in expectation) to have contributed causally to high-value states [2206.00303].

The learning protocol involves bootstrapped temporal-difference updates not only for value weights w, but also for the parameters of the mapping function z_θ(s) (or the matrix Ψ in the linear case). The bootstrapped target for z(s) satisfies a Bellman-like recursion:

$$
z(s_t) = x(s_t) + \gamma \lambda \mathbb{E}[z(S_{t-1}) | S_t = s_t]
$$

Updates proceed as:

- Value update: $w \leftarrow w + \alpha \delta_t z_\theta(S_t)$
- Predecessor feature update: $\theta \leftarrow \theta - \beta (z_\theta(S_t) - y_t) \nabla_\theta z_\theta(S_t)$
 where $y_t = x(S_t) + \gamma \lambda z_\theta(S_{t-1})$

Because z_θ(s) is improved across episodes, the learned expected trace allows credit to propagate along plausible historical paths, not just within the latest sampled episode [2206.00303].

## 3. Meta-Policy Gradient in Multi-Agent Credit Assignment

In multi-agent RL, particularly under the Centralized Training with Decentralized Execution (CTDE) paradigm, cross-episode credit assignment has been advanced through meta-learning objectives that evaluate parameter updates in the context of their influence on subsequent episode returns. The MNMPG algorithm [2102.12957] treats the parameters θᵤ of the mixing network (which combines agents' utilities) as meta-actions, and measures the effect of an "exercise update" (Q-learning step) on θᵤ, followed by a rollout with the updated parameters θᵤ′.

The cross-episode mechanism is as follows:

1. **Exercise Update:** Collect a trajectory D₀ under θᵤ; compute the exercise loss $L_{exercise}(\theta_u; D_0)$; update θᵤ → θᵤ′ via gradient step(s).
2. **Excitation Signal:** Run a new episode D₁ under θᵤ′, compute returns R(θᵤ′) and R(θᵤ) for D₁ and D₀ respectively, and define the meta-reward ΔR = R(θᵤ′) – R(θᵤ).
3. **Meta-Objective:** The meta-objective is $J_{meta}(\theta_u) = \mathbb{E}_{D_0 \sim \pi_{\theta_u}}[\Delta R]$. Its REINFORCE-style gradient:
$$
\nabla_{\theta_u} J_{meta} = \mathbb{E}_{D_0} [\Delta R \cdot \sum_{t=1}^T \nabla_{\theta_u} \log \pi(z_t | s_t; \theta_u)]
$$

This formalism propagates credit not just within an episode but across the episodic boundary, reinforcing updates that demonstrably improve future episode returns. The training loop alternates conventional TD-learning with these cross-episode meta-updates [2102.12957].

## 4. Distinctions Between Within-Episode and Cross-Episode Assignment

Traditional within-episode assignment, exemplified by TD(λ) or standard mixing-network training, propagates TD-errors to recently experienced states using sample-based eligibility traces. All credit resides within the temporally local context of a given episode. In contrast, cross-episode assignment, as realized by Predecessor Features or MNMPG, either

- propagates TD-errors to all potential predecessor states or features regardless of when (or whether) they were visited in the current episode, by maintaining and updating an expected trace that persists across episodes; or
- evaluates the causal effect of parameter changes (in e.g., a mixing network) with respect to their influence on future episodes' outcomes, rather than only on immediate loss reduction.

A key practical implication is that cross-episode methods can accelerate learning in domains with sparse or delayed rewards, or in tasks with long causal chains, by avoiding the slow "random walk" propagation of credit inherent in purely local techniques [2206.00303]. However, these advantages come with potential drawbacks, such as increased computational and sample cost (due to multiple full rollouts per meta-step in MNMPG) and increased variance in meta-gradients (from REINFORCE-style updates) [2102.12957].

## 5. Empirical Analyses and Hyperparameter Regimes

Empirical evaluation of Predecessor Features on tasks such as a tabular 6×6 “Plinko” grid and deep-cartpole demonstrate that the fully bootstrapped expected-trace approach converges more rapidly and robustly than TD(λ), particularly when learning parameters for the expected trace are properly tuned [2206.00303]. In multi-agent settings, experiments on StarCraft II micromanagement “super-hard” maps reveal that MNMPG increases win rates from approximately 50% (QMIX baseline) to over 90%, attributed to more focused exploration and robust discovery of high-reward states by exploiting the learned cross-episode global hierarchy [2102.12957].

Notable hyperparameter settings for MNMPG include:

| Parameter                     | Typical Value in MNMPG         | Description                                 |
|-------------------------------|-------------------------------|---------------------------------------------|
| Inner-loop learning rate α    | \( 5 \times 10^{-4} \)        | Step-size for exercise update               |
| Meta-learning rate β          | \( 1 \times 10^{-3} \)        | Step-size for meta-gradient update          |
| Number of exercise steps K    | 1–5 (often 1)                 | Inner-loop gradient steps per meta-iteration|
| Meta-update frequency         | Every 500 env steps           | Control over cost vs. adaptation speed      |
| Global hierarchy dimension z  | 3                             | Dimensionality of latent variable           |

These regimes are reported to yield significant performance improvements in MARL benchmarks [2102.12957].

## 6. Implications, Limitations, and Research Directions

Cross-episode credit assignment mechanisms enlarge the class of RL tasks amenable to efficient solution, especially where long-term dependencies span episode boundaries or when reward signals are both sparse and delayed. Predecessor Features enable assignment of TD-errors to paths or features not observed in the current episode but inferred through a persistently learned expected occupancy structure. MNMPG enables causal attribution across episodes by linking parameter changes to subsequent episodic returns.

Key limitations include:

- Computational cost: Both approaches may require increased computation or parameter storage (e.g., O(d²) in PF for high-dimensional features or multiple rollouts per meta-iteration in MNMPG).
- Variance and Stability: Meta-gradient estimation via REINFORCE can introduce significant variance, while bootstrapped updates may bias estimates in PF. Robustness may depend on effective alternation of standard and meta-updates, as well as function approximator expressivity [2206.00303, 2102.12957].
- Approximation error: Misspecification or insufficient capacity in the predecessor mapping or mixing network can undermine assignment accuracy.

Possible research directions suggested in existing work include generalized ET(λ,η) interleavings for PF to balance bias-variance, integration with off-policy methods, incorporation of attention mechanisms for richer predecessor modeling, and further advances in meta-learning protocols for multi-agent or hierarchical RL [2206.00303].

## 7. Summary of Principal Methods

A concise comparison of within-episode and cross-episode credit assignment architectures:

| Approach                       | Core Mechanism                                                | Cross-Episode Credit?   |
|---------------------------------|---------------------------------------------------------------|------------------------|
| TD(λ), standard mixing network  | Sample eligibility traces, episode-local TD-error assignment  | No                     |
| Predecessor Features (PF)       | Expected trace z(s), bootstrapped across episodes             | Yes                    |
| MNMPG in CTDE (MARL)            | Meta-gradient on mixing network by exercise/return difference | Yes                    |

Cross-episode credit assignment generalizes and strictly exceeds within-episode methods in its capacity for long-range, efficient propagation of value information, supporting rapid convergence and improved robustness in challenging RL domains [2206.00303, 2102.12957].

Source: https://www.emergentmind.com/topics/cross-episode-credit-assignment