Topological Experience Replay in RL

• Topological Experience Replay is a method that replaces flat replay buffers with graph-structured storage to capture state dependencies.
• It employs reverse-BFS or path-conditioned sampling strategies to enhance Q-learning convergence and reduce memory overhead.
• Applications span sparse-reward RL and continual learning, offering improved efficiency and robustness against forgetting.

Searching arXiv for the specified papers and closely related work on topological or graph-structured experience replay. Topological Experience Replay denotes a family of experience replay mechanisms that replace or augment a flat replay buffer with graph-structured storage and graph-aware sampling. In the reinforcement-learning literature, the term is used most directly for a method that organizes the agent’s experience into a directed graph whose vertices represent visited states and whose edges record one-step transitions, then performs value backups by a breadth-first search starting from terminal states and moving backward along predecessor edges (Hong et al., 2022). Closely related formulations appear in topological map abstraction for reinforcement learning, where a “vertex memory” stores transitions by abstract graph vertex and replays transitions from the vertices lying on a planned shortest path (Yin et al., 2020), and in map-based replay memories that compress transitions into prototype nodes and transition edges using a Grow-When-Required self-organizing network (Hafez et al., 2023). In continual graph learning, the phrase also refers to topology-aware replay over graph data, where global topological importance is used to rank and retain nodes (Pang et al., 2024). Across these variants, the common principle is that replay is structured by topological relations rather than treated as an unorganized multiset of transitions.

1. Conceptual scope and definitions

In deep Q-learning, standard replay buffers update Q-values using state transition tuples sampled from stored experience. The sampling strategy often uniformly and randomly samples or prioritizes data sampling based on measures such as the temporal difference error (Hong et al., 2022). The central objection raised by Topological Experience Replay is that such sampling can be inefficient at learning the Q-function because a state’s Q-value depends on the Q-value of successor states. If the sampling policy ignores the precision of the Q-value estimate of the next state, updates can become useless or incorrect (Hong et al., 2022).

The graph-based TER formulation addresses this by making state-dependency explicit. It maintains an unweighted, directed graph $\mathcal{G}=(V,E)$ in which vertices represent distinct visited states or their hash codes and edges encode one-step transitions (Hong et al., 2022). Each edge therefore represents a dependency between the Q-value at a predecessor state-action pair and the value estimates at successor states. This converts replay from an index-based sampling problem into a traversal problem over the topology of state transitions.

A separate but related use of the term appears in TOMA, where the state space is abstracted into a topological map $G=(V,E)$ whose vertices correspond one-to-one with landmark states and whose edges represent observed transitions between landmark neighborhoods (Yin et al., 2020). In that setting, replay is “topological” because transitions are stored in vertex-centric buffers and recalled according to a shortest path in the abstract graph. The paper’s summary states that this is the essence of Topological Experience Replay (Yin et al., 2020).

A third line of work uses topological organization primarily for compression and forgetting mitigation. Map-based Experience Replay replaces a flat replay memory by a compact graph of prototype states and directed transition edges, merging similar samples into shared nodes and storing averaged actions, rewards, and terminal flags on edges (Hafez et al., 2023). In continual graph learning, topology-aware replay is defined over graph-structured datasets rather than MDP trajectories; there, replay selection uses both feature significance and topological relevance, with Hodge Potential Score providing a global node ranking (Pang et al., 2024).

2. Graph constructions and storage schemes

The directed-graph TER method assumes a mapping $\phi:\mathcal{S}\to\mathbb{R}^d$, where $\phi(s)$ serves as a hash key for state $s$; in the method description, $\phi$ is a fixed random projection (Hong et al., 2022). The vertex set is

$V=\{\phi(s)\mid s \text{ has been observed}\},$

and the edge set satisfies

$E\subseteq V\times V,$

with $(v,v')\in E$ if and only if the agent executed an action in a state mapped to $v$ and reached a next state mapped to $G=(V,E)$0 (Hong et al., 2022). To each edge $G=(V,E)$1, the method attaches a FIFO list of all tuples $G=(V,E)$2 observed for that transition. Insertions add the hashed states to $G=(V,E)$3, add the directed edge to $G=(V,E)$4, and append the transition to the edge-local list (Hong et al., 2022).

TOMA uses a different abstraction level. Its topological map is an undirected graph $G=(V,E)$5, where $G=(V,E)$6 is a dynamic set of vertices and each vertex $G=(V,E)$7 corresponds one-to-one with a landmark state $G=(V,E)$8 (Yin et al., 2020). A learned embedding $G=(V,E)$9 assigns a new state $\phi:\mathcal{S}\to\mathbb{R}^d$0 to the nearest landmark vertex through

$\phi:\mathcal{S}\to\mathbb{R}^d$1

and a K-d tree is maintained over the landmark embeddings for $\phi:\mathcal{S}\to\mathbb{R}^d$2 nearest-neighbor lookup (Yin et al., 2020). The replay structure is vertex-centric: for each vertex $\phi:\mathcal{S}\to\mathbb{R}^d$3, a fixed-capacity replay buffer $\phi:\mathcal{S}\to\mathbb{R}^d$4 stores transitions whose landing state was clustered to $\phi:\mathcal{S}\to\mathbb{R}^d$5, while a standard HER buffer $\phi:\mathcal{S}\to\mathbb{R}^d$6 is kept globally (Yin et al., 2020).

Map-based Experience Replay adopts a prototype graph rather than exact-state or landmark indexing. Incoming states are compared against current node prototypes $\phi:\mathcal{S}\to\mathbb{R}^d$7 by Euclidean distance,

$\phi:\mathcal{S}\to\mathbb{R}^d$8

with activation

$\phi:\mathcal{S}\to\mathbb{R}^d$9

The Best-Matching Unit is the node with maximal activation (Hafez et al., 2023). If activation and habituation thresholds are not satisfied, a new node is inserted; otherwise the state is merged into the closest prototypes through competitive Hebbian updates. Directed temporal edges between consecutive Best-Matching Units then accumulate counts, averaged actions, averaged rewards, and terminal information (Hafez et al., 2023). This produces a graph of prototypes and transition summaries rather than raw transition storage.

In continual graph learning, FTF-ER does not build an MDP transition graph. Instead, it evaluates the current training subgraph $\phi(s)$0 through graph-theoretic differential operators and Hodge decomposition. The topological quantity of interest is a node potential $\phi(s)$1 obtained from solving a Laplacian system, and replay selects nodes according to the resulting Hodge Potential Score (Pang et al., 2024). This is topological replay in a different sense: the replay unit is the node in a graph dataset rather than the RL transition tuple.

3. Sampling order, backup topology, and replay dynamics

The distinctive feature of TER in Q-learning is that it reorders experience replay according to value-dependency structure. The Bellman target

$\phi(s)$2

shows that updates at $\phi(s)$3 depend on the value estimates at successor states (Hong et al., 2022). TER therefore performs a breadth-first search on the replay graph starting from terminal states and moving backward along incoming edges. A queue is initialized with a sample of terminal vertices, predecessor sets are retrieved through

$\phi(s)$4

and stored transitions on predecessor edges are enqueued until enough transitions have been collected for a mini-batch (Hong et al., 2022). Each BFS layer corresponds to states at a fixed graph distance from terminal nodes.

The method is explicitly contrasted with Uniform Experience Replay and Prioritized Experience Replay. Uniform replay samples each transition with probability $\phi(s)$5, whereas prioritized replay uses probabilities proportional to powers of the TD error. TER instead mixes a fraction of each batch from BFS-based reverse sweeps and a fraction by PER:

$\phi(s)$6

where $\phi(s)$7 is the mixing ratio (Hong et al., 2022). This makes the replay density highly skewed toward transitions near terminals in the graph, while preserving nonzero probability for all transitions through the mixed component.

TOMA’s replay dynamics are path-conditioned rather than globally reverse-topological. At the beginning of an episode, or periodically every $\phi(s)$8 steps, an ultimate graph goal $\phi(s)$9 is selected, with either a farthest-vertex or least-visited-vertex criterion (Yin et al., 2020). The current vertex and the goal vertex determine a shortest path

$s$0

computed by Dijkstra on $s$1 with unit edge weights (Yin et al., 2020). A mixed minibatch of size $s$2 is assembled by taking $s$3 samples from the union of per-vertex buffers along the path,

$s$4

and $s$5 samples from the HER buffer (Yin et al., 2020). In the basic implementation, sampling over the path-conditioned union is uniform (Yin et al., 2020).

Map-based Experience Replay uses a simpler topological sampler. A node is sampled uniformly from the prototype graph, a successor node is then sampled with probability proportional to the transition count $s$6, and the replayed tuple is reconstructed from the prototype node weights and stored edge statistics (Hafez et al., 2023). This does not enforce a reverse-sweep schedule, but it uses graph topology to reconstruct compact surrogate experiences.

4. Exploration, forgetting, and topological abstraction

TOMA assigns an explicitly exploratory role to topological replay. Goal selection is described as a graph-based bonus because choosing either the farthest or least-visited vertex forces the agent to drive exploration outward along the graph frontier (Yin et al., 2020). The topological map abstracts the state space into clusters, and vertex memory stores “how I got into each cluster” (Yin et al., 2020). When the agent plans a route to a distant vertex, it retrieves transitions that led into each intermediate cluster, thereby replaying the border-crossing skills required at successive stages of the route (Yin et al., 2020).

The same account argues that this combats catastrophic forgetting of earlier sub-skills: even as the agent learns to go deeper into the graph, it continues replaying earlier cluster-entry transitions (Yin et al., 2020). The mixed use of HER and vertex memory ensures that half the gradient steps reinforce task-specific skills associated with reaching the next border landmark and half reinforce general replayed goals (Yin et al., 2020). A plausible implication is that TOMA treats replay not only as variance reduction for off-policy learning, but also as a mechanism for maintaining competence on reusable navigation primitives.

Map-based Experience Replay addresses forgetting through memory compaction and de-correlation. By merging only very similar states into single prototypes, it enforces that each stored node is “maximally informative,” increasing the mean pairwise distance among samples and making mini-batches more diverse than those drawn from a naïve buffer (Hafez et al., 2023). The learned directed graph records frequently used transitions more strongly through the transition-count matrix, so replay naturally focuses on novel or under-sampled parts of the state space (Hafez et al., 2023). The authors explicitly connect this to reduced bias toward recent experience and stabilization of value-function updates over long horizons (Hafez et al., 2023).

In FTF-ER, the target pathology is catastrophic forgetting in continual graph learning rather than RL exploration. The method argues that existing experience replay methods focus either on feature significance or topological relevance, and that topology-based methods relying on neighboring nodes consider only local topological information while increasing memory overhead (Pang et al., 2024). Hodge Potential Score is introduced to capture global topological information, and the feature-topology fusion score combines structural centrality with feature-based informativeness (Pang et al., 2024). This suggests a broader interpretation of topological replay: topology can govern not only the order of Bellman backups, but also which structural units are worth retaining under a fixed memory budget.

5. Theoretical framing and methodological comparisons

TER’s theoretical motivation is rooted in value-backup ordering. The method notes two known facts: in acyclic MDPs, Bertsekas (2000) shows that a reverse topological sweep from terminal states to predecessors is optimal for value iteration, and Dai and Hansen (2007) show that interleaving arbitrary priority mechanisms with uniform sampling preserves Q-learning convergence (Hong et al., 2022). Because TER mixes reverse-sweep sampling with a nonzero random component, it is described as retaining the standard contraction argument for Bellman operators and ensuring convergence to $s$7 under usual assumptions (Hong et al., 2022). The formulation is given informally as a lemma requiring replay density $s$8 for all transitions and standard Robbins-Monro conditions on the learning rate (Hong et al., 2022).

This theoretical framing distinguishes TER from Prioritized Experience Replay. PER ranks transitions by immediate TD error,

$s$9

and samples according to probabilities proportional to $\phi$0 with importance-sampling correction weights (Hong et al., 2022). TER’s criticism is not that TD error is irrelevant, but that TD error alone neglects whether the bootstrapped successor values are already reliable. The graph encodes the dependencies needed to choose a more appropriate update order (Hong et al., 2022).

TOMA differs methodologically from TER despite sharing topological replay terminology. It does not perform a backward sweep from terminal states. Instead, it uses graph abstraction, shortest-path planning, landmark-conditioned subgoals, and vertex-wise replay. Its replay unit is determined by the landing state’s assigned abstract vertex, and batch construction is tied to the current planned route (Yin et al., 2020). This makes it a planning-and-exploration framework in which replay is route-aware.

Map-based Experience Replay differs again. It is not primarily about ordering Bellman backups or planning toward subgoals, but about replacing raw storage with a concise environment-model-like network of state nodes and transition edges (Hafez et al., 2023). The topological advantage is memory efficiency and sample diversity rather than explicit successor-precision reasoning.

The following table summarizes the principal variants described in the cited work.

Method	Core graph unit	Replay mechanism
TER (Hong et al., 2022)	Distinct visited states or hash codes	BFS reverse sweep from terminal states, mixed with PER
TOMA vertex memory (Yin et al., 2020)	Landmark vertices in an abstract topological map	Uniform sampling from vertex buffers along the current shortest path, mixed with HER
Map-based Experience Replay (Hafez et al., 2023)	Prototype state-nodes and directed transition-edges	Sampling node-successor pairs from a compact GWR-R graph
FTF-ER (Pang et al., 2024)	Nodes in a training subgraph	Ranking and retaining nodes by fused feature and topological scores

6. Empirical findings, limitations, and adjacent developments

The directed-graph TER paper reports experiments on NChain, the Minigrid suite with 40×40 RGB images, Sokoban puzzles with 84×84 images, and additional ablations on stochastic dynamics and Atari control tasks without terminal-goal structure (Hong et al., 2022). The metric is average normalized return versus environment steps or versus number of Q-updates, with curves showing mean ± 95% bootstrap confidence intervals over 5 seeds (Hong et al., 2022). The reported findings are that TER converges in approximately 30 backups on NChain versus more than 100 for UER, PER, and DisCor; reaches maximal success rate 2–5× faster than PER or EBU on Minigrid tasks; achieves nonzero success on harder Sokoban levels while baselines often fail or over-estimate Q; and retains its advantage even when replay ratio for PER or UER is increased, because higher replay ratios cause more over-estimation (Hong et al., 2022). The paper also states that TER scales readily to high-dimensional observations because graph vertices are merely hash keys and no state decoding or generative modeling is needed (Hong et al., 2022).

The same source enumerates several limitations. Collisions in $\phi$1 or truly one-off observations limit stitching of episodes; if no repeated states occur, TER degenerates to simple reverse-trajectory replay, identified with EBU (Hong et al., 2022). The method also requires terminal-goal tasks with sparse final reward, and extension to continuing tasks would need “pseudo-terminals” or other high-value seeds (Hong et al., 2022). Fixed hashing ignores task-relevant state similarity, and the paper suggests that learned embeddings, pseudo-terminal nodes, probabilistic predecessor weighting for stochastic MDPs, and integration with model-based successor or predecessor learning are plausible extensions (Hong et al., 2022).

For TOMA, the summary states that TOMA with vertex memory solves sparse-reward mazes far more reliably than HER alone or TOMA without vertex memory (Yin et al., 2020). The broader paper claim is that TOMA can generate an abstract graph representation for MDPs with much less memory and computation cost than existing methods, can facilitate exploration through planning to explore, and can outperform existing methods to achieve state-of-the-art performance (Yin et al., 2020). Since the supplied details do not reproduce full benchmark tables, those claims are best understood at the level stated in the paper summary.

Map-based Experience Replay reports explicit memory-performance trade-offs. The simplest memory reduction factor is

$\phi$2

with complementary savings $\phi$3 (Hafez et al., 2023). The reported savings range from 40% to 80%, meaning that only 20% to 60% of the original transitions need to be stored (Hafez et al., 2023). The same source reports that moderate abstraction, corresponding to 40% to 60% memory reduction, yields less than 10% performance loss, while very aggressive merging beyond 80% savings begins to hurt (Hafez et al., 2023).

In continual graph learning, FTF-ER reports a significant improvement of 3.6% in AA and 7.1% in AF on the OGB-Arxiv dataset in the class-incremental learning setting (Pang et al., 2024). The detailed comparison also reports gains in Average Accuracy up to +2.2%, reduction in Average Forgetting up to −0.03, memory savings of approximately 20–30%, and about 20% faster per epoch on Reddit relative to specific baselines (Pang et al., 2024). Although this is outside reinforcement learning proper, it shows that topological replay has become a broader design pattern for memory-limited continual learning over structured data.

A common misconception is that all topological replay methods are instances of the same algorithm. The literature instead supports a narrower conclusion: “Topological Experience Replay” names at least one specific reverse-BFS replay method for Q-learning (Hong et al., 2022), while also serving as a descriptive label for route-conditioned vertex replay in TOMA (Yin et al., 2020), prototype-graph replay for memory compression (Hafez et al., 2023), and topology-aware node selection in continual graph learning (Pang et al., 2024). What unifies them is not a single implementation, but the use of graph topology to determine what is stored, what is replayed, and in what order.