---
title: Topological Experience Replay in RL
url: https://www.emergentmind.com/topics/topological-experience-replay
type: topic
---

# Topological Experience Replay in RL

Searching arXiv for the specified papers and closely related work on topological or graph-structured experience replay.
Topological Experience Replay denotes a family of experience replay mechanisms that replace or augment a flat replay buffer with graph-structured storage and graph-aware sampling. In the reinforcement-learning literature, the term is used most directly for a method that organizes the agent’s experience into a directed graph whose vertices represent visited states and whose edges record one-step transitions, then performs value backups by a breadth-first search starting from terminal states and moving backward along predecessor edges [2203.15845]. Closely related formulations appear in topological map abstraction for reinforcement learning, where a “vertex memory” stores transitions by abstract graph vertex and replays transitions from the vertices lying on a planned shortest path [2005.06061], and in map-based replay memories that compress transitions into prototype nodes and transition edges using a Grow-When-Required self-organizing network [2305.02054]. In continual graph learning, the phrase also refers to topology-aware replay over graph data, where global topological importance is used to rank and retain nodes [2407.19429]. Across these variants, the common principle is that replay is structured by topological relations rather than treated as an unorganized multiset of transitions.

## 1. Conceptual scope and definitions

In deep Q-learning, standard replay buffers update Q-values using state transition tuples sampled from stored experience. The sampling strategy often uniformly and randomly samples or prioritizes data sampling based on measures such as the temporal difference error [2203.15845]. The central objection raised by Topological Experience Replay is that such sampling can be inefficient at learning the Q-function because a state’s Q-value depends on the Q-value of successor states. If the sampling policy ignores the precision of the Q-value estimate of the next state, updates can become useless or incorrect [2203.15845].

The graph-based TER formulation addresses this by making state-dependency explicit. It maintains an unweighted, directed graph $\mathcal{G}=(V,E)$ in which vertices represent distinct visited states or their hash codes and edges encode one-step transitions [2203.15845]. Each edge therefore represents a dependency between the Q-value at a predecessor state-action pair and the value estimates at successor states. This converts replay from an index-based sampling problem into a traversal problem over the topology of state transitions.

A separate but related use of the term appears in TOMA, where the state space is abstracted into a topological map $G=(V,E)$ whose vertices correspond one-to-one with landmark states and whose edges represent observed transitions between landmark neighborhoods [2005.06061]. In that setting, replay is “topological” because transitions are stored in vertex-centric buffers and recalled according to a shortest path in the abstract graph. The paper’s summary states that this is the essence of Topological Experience Replay [2005.06061].

A third line of work uses topological organization primarily for compression and forgetting mitigation. Map-based Experience Replay replaces a flat replay memory by a compact graph of prototype states and directed transition edges, merging similar samples into shared nodes and storing averaged actions, rewards, and terminal flags on edges [2305.02054]. In continual graph learning, topology-aware replay is defined over graph-structured datasets rather than MDP trajectories; there, replay selection uses both feature significance and topological relevance, with Hodge Potential Score providing a global node ranking [2407.19429].

## 2. Graph constructions and storage schemes

The directed-graph TER method assumes a mapping $\phi:\mathcal{S}\to\mathbb{R}^d$, where $\phi(s)$ serves as a hash key for state $s$; in the method description, $\phi$ is a fixed random projection [2203.15845]. The vertex set is
$$
V=\{\phi(s)\mid s \text{ has been observed}\},
$$
and the edge set satisfies
$$
E\subseteq V\times V,
$$
with $(v,v')\in E$ if and only if the agent executed an action in a state mapped to $v$ and reached a next state mapped to $v'$ [2203.15845]. To each edge $(v,v')$, the method attaches a FIFO list of all tuples $(s,a,r,s')$ observed for that transition. Insertions add the hashed states to $V$, add the directed edge to $E$, and append the transition to the edge-local list [2203.15845].

TOMA uses a different abstraction level. Its topological map is an undirected graph $G=(V,E)$, where $V=\{v_1,\dots,v_N\}$ is a dynamic set of vertices and each vertex $v_i$ corresponds one-to-one with a landmark state $l_i\in S$ [2005.06061]. A learned embedding $\phi_\theta:S\to\mathbb{R}^d$ assigns a new state $s$ to the nearest landmark vertex through
$$
v(s)=\arg\min_{v_j\in V}\|\phi_\theta(s)-\phi_\theta(l_j)\|_2,
$$
and a K-d tree is maintained over the landmark embeddings for $O(\log |V|)$ nearest-neighbor lookup [2005.06061]. The replay structure is vertex-centric: for each vertex $v_i$, a fixed-capacity replay buffer $\mathcal{M}_i$ stores transitions whose landing state was clustered to $v_i$, while a standard HER buffer $\mathcal{H}$ is kept globally [2005.06061].

Map-based Experience Replay adopts a prototype graph rather than exact-state or landmark indexing. Incoming states are compared against current node prototypes $w_i$ by Euclidean distance,
$$
d_i(x)=\|x-w_i\|_2,
$$
with activation
$$
a_i(x)=\exp(-d_i(x)).
$$
The Best-Matching Unit is the node with maximal activation [2305.02054]. If activation and habituation thresholds are not satisfied, a new node is inserted; otherwise the state is merged into the closest prototypes through competitive Hebbian updates. Directed temporal edges between consecutive Best-Matching Units then accumulate counts, averaged actions, averaged rewards, and terminal information [2305.02054]. This produces a graph of prototypes and transition summaries rather than raw transition storage.

In continual graph learning, FTF-ER does not build an MDP transition graph. Instead, it evaluates the current training subgraph $G_t=(V_t,E_t)$ through graph-theoretic differential operators and Hodge decomposition. The topological quantity of interest is a node potential $s\in\Omega^0(G)$ obtained from solving a Laplacian system, and replay selects nodes according to the resulting Hodge Potential Score [2407.19429]. This is topological replay in a different sense: the replay unit is the node in a graph dataset rather than the RL transition tuple.

## 3. Sampling order, backup topology, and replay dynamics

The distinctive feature of TER in Q-learning is that it reorders experience replay according to value-dependency structure. The Bellman target
$$
Q(s,a)\leftarrow r+\gamma\max_{a'}Q(s',a')
$$
shows that updates at $(s,a)$ depend on the value estimates at successor states [2203.15845]. TER therefore performs a breadth-first search on the replay graph starting from terminal states and moving backward along incoming edges. A queue is initialized with a sample of terminal vertices, predecessor sets are retrieved through
$$
Adj(v')=\{v\in V\mid (v,v')\in E\},
$$
and stored transitions on predecessor edges are enqueued until enough transitions have been collected for a mini-batch [2203.15845]. Each BFS layer corresponds to states at a fixed graph distance from terminal nodes.

The method is explicitly contrasted with Uniform Experience Replay and Prioritized Experience Replay. Uniform replay samples each transition with probability $1/N$, whereas prioritized replay uses probabilities proportional to powers of the TD error. TER instead mixes a fraction of each batch from BFS-based reverse sweeps and a fraction by PER:
$$
p_{TER}(i)=(1-\eta)\,p_{BFS}(i)+\eta\,p_{PER}(i),
$$
where $\eta\in[0,1]$ is the mixing ratio [2203.15845]. This makes the replay density highly skewed toward transitions near terminals in the graph, while preserving nonzero probability for all transitions through the mixed component.

TOMA’s replay dynamics are path-conditioned rather than globally reverse-topological. At the beginning of an episode, or periodically every $K$ steps, an ultimate graph goal $g\in S$ is selected, with either a farthest-vertex or least-visited-vertex criterion [2005.06061]. The current vertex and the goal vertex determine a shortest path
$$
P=(v^{(0)}=v_{cur},v^{(1)},\dots,v^{(k)}=v_{goal}),
$$
computed by Dijkstra on $(V,E)$ with unit edge weights [2005.06061]. A mixed minibatch of size $B$ is assembled by taking $B_{vm}=\lfloor B/2\rfloor$ samples from the union of per-vertex buffers along the path,
$$
\mathcal{M}_P=\bigcup_{j=0\ldots k}\mathcal{M}_{v^{(j)}},
$$
and $B-B_{vm}$ samples from the HER buffer [2005.06061]. In the basic implementation, sampling over the path-conditioned union is uniform [2005.06061].

Map-based Experience Replay uses a simpler topological sampler. A node is sampled uniformly from the prototype graph, a successor node is then sampled with probability proportional to the transition count $TC_{ij}$, and the replayed tuple is reconstructed from the prototype node weights and stored edge statistics [2305.02054]. This does not enforce a reverse-sweep schedule, but it uses graph topology to reconstruct compact surrogate experiences.

## 4. Exploration, forgetting, and topological abstraction

TOMA assigns an explicitly exploratory role to topological replay. Goal selection is described as a graph-based bonus because choosing either the farthest or least-visited vertex forces the agent to drive exploration outward along the graph frontier [2005.06061]. The topological map abstracts the state space into clusters, and vertex memory stores “how I got into each cluster” [2005.06061]. When the agent plans a route to a distant vertex, it retrieves transitions that led into each intermediate cluster, thereby replaying the border-crossing skills required at successive stages of the route [2005.06061].

The same account argues that this combats catastrophic forgetting of earlier sub-skills: even as the agent learns to go deeper into the graph, it continues replaying earlier cluster-entry transitions [2005.06061]. The mixed use of HER and vertex memory ensures that half the gradient steps reinforce task-specific skills associated with reaching the next border landmark and half reinforce general replayed goals [2005.06061]. A plausible implication is that TOMA treats replay not only as variance reduction for off-policy learning, but also as a mechanism for maintaining competence on reusable navigation primitives.

Map-based Experience Replay addresses forgetting through memory compaction and de-correlation. By merging only very similar states into single prototypes, it enforces that each stored node is “maximally informative,” increasing the mean pairwise distance among samples and making mini-batches more diverse than those drawn from a naïve buffer [2305.02054]. The learned directed graph records frequently used transitions more strongly through the transition-count matrix, so replay naturally focuses on novel or under-sampled parts of the state space [2305.02054]. The authors explicitly connect this to reduced bias toward recent experience and stabilization of value-function updates over long horizons [2305.02054].

In FTF-ER, the target pathology is catastrophic forgetting in continual graph learning rather than RL exploration. The method argues that existing experience replay methods focus either on feature significance or topological relevance, and that topology-based methods relying on neighboring nodes consider only local topological information while increasing memory overhead [2407.19429]. Hodge Potential Score is introduced to capture global topological information, and the feature-topology fusion score combines structural centrality with feature-based informativeness [2407.19429]. This suggests a broader interpretation of topological replay: topology can govern not only the order of Bellman backups, but also which structural units are worth retaining under a fixed memory budget.

## 5. Theoretical framing and methodological comparisons

TER’s theoretical motivation is rooted in value-backup ordering. The method notes two known facts: in acyclic MDPs, Bertsekas (2000) shows that a reverse topological sweep from terminal states to predecessors is optimal for value iteration, and Dai and Hansen (2007) show that interleaving arbitrary priority mechanisms with uniform sampling preserves Q-learning convergence [2203.15845]. Because TER mixes reverse-sweep sampling with a nonzero random component, it is described as retaining the standard contraction argument for Bellman operators and ensuring convergence to $Q^*$ under usual assumptions [2203.15845]. The formulation is given informally as a lemma requiring replay density $p(i)\ge \epsilon/N$ for all transitions and standard Robbins-Monro conditions on the learning rate [2203.15845].

This theoretical framing distinguishes TER from Prioritized Experience Replay. PER ranks transitions by immediate TD error,
$$
\delta_i=\left|r_i+\gamma\max_{a'}Q(s_i',a')-Q(s_i,a_i)\right|,
$$
and samples according to probabilities proportional to $(|\delta_i|+\epsilon)^\alpha$ with importance-sampling correction weights [2203.15845]. TER’s criticism is not that TD error is irrelevant, but that TD error alone neglects whether the bootstrapped successor values are already reliable. The graph encodes the dependencies needed to choose a more appropriate update order [2203.15845].

TOMA differs methodologically from TER despite sharing topological replay terminology. It does not perform a backward sweep from terminal states. Instead, it uses graph abstraction, shortest-path planning, landmark-conditioned subgoals, and vertex-wise replay. Its replay unit is determined by the landing state’s assigned abstract vertex, and batch construction is tied to the current planned route [2005.06061]. This makes it a planning-and-exploration framework in which replay is route-aware.

Map-based Experience Replay differs again. It is not primarily about ordering Bellman backups or planning toward subgoals, but about replacing raw storage with a concise environment-model-like network of state nodes and transition edges [2305.02054]. The topological advantage is memory efficiency and sample diversity rather than explicit successor-precision reasoning.

The following table summarizes the principal variants described in the cited work.

| Method | Core graph unit | Replay mechanism |
|---|---|---|
| TER [2203.15845] | Distinct visited states or hash codes | BFS reverse sweep from terminal states, mixed with PER |
| TOMA vertex memory [2005.06061] | Landmark vertices in an abstract topological map | Uniform sampling from vertex buffers along the current shortest path, mixed with HER |
| Map-based Experience Replay [2305.02054] | Prototype state-nodes and directed transition-edges | Sampling node-successor pairs from a compact GWR-R graph |
| FTF-ER [2407.19429] | Nodes in a training subgraph | Ranking and retaining nodes by fused feature and topological scores |

## 6. Empirical findings, limitations, and adjacent developments

The directed-graph TER paper reports experiments on NChain, the Minigrid suite with 40×40 RGB images, Sokoban puzzles with 84×84 images, and additional ablations on stochastic dynamics and Atari control tasks without terminal-goal structure [2203.15845]. The metric is average normalized return versus environment steps or versus number of Q-updates, with curves showing mean ± 95% bootstrap confidence intervals over 5 seeds [2203.15845]. The reported findings are that TER converges in approximately 30 backups on NChain versus more than 100 for UER, PER, and DisCor; reaches maximal success rate 2–5× faster than PER or EBU on Minigrid tasks; achieves nonzero success on harder Sokoban levels while baselines often fail or over-estimate Q; and retains its advantage even when replay ratio for PER or UER is increased, because higher replay ratios cause more over-estimation [2203.15845]. The paper also states that TER scales readily to high-dimensional observations because graph vertices are merely hash keys and no state decoding or generative modeling is needed [2203.15845].

The same source enumerates several limitations. Collisions in $\phi(s)$ or truly one-off observations limit stitching of episodes; if no repeated states occur, TER degenerates to simple reverse-trajectory replay, identified with EBU [2203.15845]. The method also requires terminal-goal tasks with sparse final reward, and extension to continuing tasks would need “pseudo-terminals” or other high-value seeds [2203.15845]. Fixed hashing ignores task-relevant state similarity, and the paper suggests that learned embeddings, pseudo-terminal nodes, probabilistic predecessor weighting for stochastic MDPs, and integration with model-based successor or predecessor learning are plausible extensions [2203.15845].

For TOMA, the summary states that TOMA with vertex memory solves sparse-reward mazes far more reliably than HER alone or TOMA without vertex memory [2005.06061]. The broader paper claim is that TOMA can generate an abstract graph representation for MDPs with much less memory and computation cost than existing methods, can facilitate exploration through planning to explore, and can outperform existing methods to achieve state-of-the-art performance [2005.06061]. Since the supplied details do not reproduce full benchmark tables, those claims are best understood at the level stated in the paper summary.

Map-based Experience Replay reports explicit memory-performance trade-offs. The simplest memory reduction factor is
$$
R_{mem}=|V|/N,
$$
with complementary savings $1-R_{mem}$ [2305.02054]. The reported savings range from 40% to 80%, meaning that only 20% to 60% of the original transitions need to be stored [2305.02054]. The same source reports that moderate abstraction, corresponding to 40% to 60% memory reduction, yields less than 10% performance loss, while very aggressive merging beyond 80% savings begins to hurt [2305.02054].

In continual graph learning, FTF-ER reports a significant improvement of 3.6% in AA and 7.1% in AF on the OGB-Arxiv dataset in the class-incremental learning setting [2407.19429]. The detailed comparison also reports gains in Average Accuracy up to +2.2%, reduction in Average Forgetting up to −0.03, memory savings of approximately 20–30%, and about 20% faster per epoch on Reddit relative to specific baselines [2407.19429]. Although this is outside reinforcement learning proper, it shows that topological replay has become a broader design pattern for memory-limited continual learning over structured data.

A common misconception is that all topological replay methods are instances of the same algorithm. The literature instead supports a narrower conclusion: “Topological Experience Replay” names at least one specific reverse-BFS replay method for Q-learning [2203.15845], while also serving as a descriptive label for route-conditioned vertex replay in TOMA [2005.06061], prototype-graph replay for memory compression [2305.02054], and topology-aware node selection in continual graph learning [2407.19429]. What unifies them is not a single implementation, but the use of graph topology to determine what is stored, what is replayed, and in what order.

Source: https://www.emergentmind.com/topics/topological-experience-replay