Experience Graph Overview

• Experience graph is a structured representation of past experiences using nodes and edges to capture and reuse relational information in dynamic environments.
• In motion planning, techniques like Thunder leverage experience graphs to reduce memory by up to 98.8% and speed up path retrieval by reusing subpaths.
• In reinforcement learning and LLM-agent systems, experience graphs enable prioritized replay and skill auditing, enhancing update efficiency and semantic reuse.

Searching arXiv for papers on “experience graph” and closely related graph-structured experience memory/planning formulations. Experience graph denotes a graph-structured representation of accumulated experience in which reusable structure is preserved explicitly rather than left implicit in flat logs, isolated trajectories, or uniformly sampled replay buffers. Across the literature, the term is used for several related constructions: motion-planning graphs that store prior feasible configurations and transitions for later reuse, replay graphs that encode state dependencies for more effective value backup or sample selection, and agent-memory graphs that record executable artifacts, tool outputs, rewards, sibling comparisons, mutable search statistics, and causal lineage as durable state (Liao et al., 29 Jun 2026). The literature therefore suggests that “experience graph” is not a single canonical object, but a family of graph-based experience representations specialized to different workloads.

1. Conceptual scope and recurrent structure

A common feature of experience-graph formulations is that experience is treated as structured data with explicit nodes, edges, and update rules. Nodes may denote robot configurations, observed states, replay candidates, queries, tool invocations, reasoning traces, answers, skills, or entire experiences. Edges may denote feasible motion transitions, temporal transitions, semantic similarity, causal lineage, tool-to-tool dependency, or composition contracts. What is preserved is not only what happened, but also how one episode relates to others and which substructures are reusable.

A second recurrent feature is that graph structure is coupled to a task-specific control operation. In motion planning, this operation is shortest-path retrieval with lazy validation or repair. In reinforcement learning, it is reverse breadth-first search or prioritized replay over dependency structure. In recent LLM-agent systems, it is graph retrieval, diffusion, applicability-aware traversal, or verifier-backed promotion of reusable skills. The graph is therefore not merely archival; it is an operational substrate for search, replay, and adaptation.

Domain	Typical graph contents	Representative papers
Motion planning	Configurations, feasible transitions, reused path segments	(Coleman et al., 2014, Lai et al., 2021)
RL and continual learning	States, transitions, replay nodes, feature/topology scores	(Hong et al., 2022, Zhou et al., 2020, Pang et al., 2024)
Agent memory and tool use	Artifacts, rewards, tool dependencies, state summaries, skills	(Liao et al., 29 Jun 2026, Feng et al., 29 May 2026, Li et al., 8 Dec 2025)

This breadth matters because claims about experience graphs are often domain-specific. A sparse roadmap spanner for humanoid motion planning and a heterogeneous query-thinking-answer graph for reward prediction solve different problems even though both are graph-structured memories of prior experience.

2. Motion-planning lineage

Some of the clearest early experience-graph formulations arose in motion planning. The Thunder framework stores experiences in a single Sparse Roadmap Spanner rather than in individual paths, with the explicit aim of eliminating redundant information, enabling path-segment reuse, and imposing a theoretical limit on graph size (Coleman et al., 2014). In that formulation, solution paths are discretized into states and selectively incorporated into SPARS under coverage, connectivity, and quality criteria. Because shared subpaths are stored once, the graph becomes both a memory-saving device and a retrieval structure.

Thunder demonstrated that graph-based storage can change the asymptotics and empirical profile of experience reuse. In environments with variable obstacles and stability constraints, it was reported to be on average an order of magnitude faster than Lightning and planning from scratch, while using 98.8% less memory after 10,000 trials (Coleman et al., 2014). The significance of these results is not merely quantitative. They establish a core design principle that reappears in later work: reusable experience is often more naturally represented as a graph of overlapping substructures than as a database of whole episodes.

A later development is the lazy experience graph in LTR. Here the graph is persistent and incrementally constructed, but edge validity is checked only upon usage, not during initial graph construction (Lai et al., 2021). LTR grows bidirectional trees for each new task, stores new nodes and edges in a temporary graph, merges them into a persistent graph, and then attempts lazy shortest-path search through the accumulated structure when subsequent tasks arise. Because the workspace changes after each pick-and-place operation, prior paths cannot simply be replayed as fixed assets. The lazy graph instead supports selective revalidation, edge removal when collisions appear, and continued growth in newly free regions. The method is accompanied by proofs of probabilistic completeness and almost-surely asymptotic optimal guarantees (Lai et al., 2021).

These motion-planning papers illustrate a durable interpretation of experience graph: a compact reusable connectivity skeleton over a changing search space. The graph stores not only successes, but also enough structure to support repair when previous solutions become invalid.

3. Replay, dependency structure, and continual graph learning

In reinforcement learning, experience graphs arise when the dependency structure among updates is made explicit. Topological Experience Replay organizes experience as a directed graph $\mathcal{G} = (V,E)$ in which vertices are encoded states and each edge denotes an experienced transition $(s,a,r,s')$ (Hong et al., 2022). Instead of sampling transitions uniformly or solely by TD error, TER performs value backups by reverse breadth-first search starting from terminal states. This is motivated by the observation that the Q-value of a state depends on the Q-values of successor states, so backup order matters. On NChain, TER was reported to achieve the optimal policy in 30 value backups, whereas UER, PER, and DisCor took 3–4x more backups and EBU was still 2x slower (Hong et al., 2022).

Continual graph learning introduces a related but distinct use of experience graphs. ER-GNN addresses catastrophic forgetting by storing representative nodes from earlier graph tasks and replaying them during learning of later tasks (Zhou et al., 2020). Its buffer update is governed by node-selection strategies—mean of feature, coverage maximization, and influence maximization—and its training loss combines current-task data and replayed experiences through a dynamic weight,

$$\mathcal{L}_{T_i}'=\beta \mathcal{L}_{T_i}(D_i^{\mathrm{tr}})+(1-\beta)\mathcal{L}_{T_i}(B), \qquad \beta=\frac{|B|}{|\mathcal{D}_i^{\mathrm{tr}}|+|B|}.$$

The central idea is that experience replay for graphs must choose representative nodes rather than merely accumulate past batches.

FTF-ER extends this logic by arguing that replay selection should fuse feature informativeness with global topological importance rather than privileging only one of them (Pang et al., 2024). Its node score is

$$\mathbf{S}^{mix}(v)=\beta \cdot \mathbf{S}^{feature}(v)+(1-\beta)\cdot \mathbf{S}^{topo}(v),$$

and the topology term is computed using Hodge Potential Score, derived from Hodge decomposition on graphs. HPS is intended to provide a global node ranking and to avoid the memory overhead of neighbor sampling. On OGB-Arxiv in the class-incremental setting, FTF-ER reported a 3.6% improvement in AA and a 7.1% improvement in AF relative to state-of-the-art methods (Pang et al., 2024).

Taken together, these papers show two distinct replay roles for experience graphs. One role is to encode transition dependency so that update order becomes structurally informed, as in TER. The other is to inform which graph elements should be retained and replayed under severe memory constraints, as in ER-GNN and FTF-ER.

4. External memory for agentic systems

Recent LLM-agent work generalizes the experience-graph concept from transitions and configurations to richer semantic and executable objects. MemReward constructs a heterogeneous graph in which queries, thinking processes, and answers are separate node types connected by similarity and structural edges, and a GNN propagates rewards from labeled to unlabeled rollouts during online optimization (Luo et al., 13 Mar 2026). With only 20% labels, it reported 97.3% of Oracle performance on Qwen2.5-3B and 96.6% on Qwen2.5-1.5B, reaching 99.4% of Oracle at 70% labels (Luo et al., 13 Mar 2026). Here the experience graph acts as a semi-supervised reward-propagation substrate rather than a planner or replay buffer.

Other systems use graph memory to support exploration and multi-turn adaptation. KG-Agent organizes GUI interaction history into a persistent State-Action Knowledge Graph whose nodes are GUI states, whose similarity edges connect functionally analogous but visually distinct states, and whose skill edges represent successful execution of skills between states (Tang et al., 17 Oct 2025). SIT-Graph represents tool-use history as

$G=(V,E,D,W,I),$

where tool nodes and directed tool-dependency edges are augmented with compact state summaries attached to edges (Li et al., 8 Dec 2025). The objective in both cases is to reuse partially overlapping experience rather than retrieve whole trajectories as indivisible units.

A more clinically structured variant appears in GSEM, which uses a dual-layer memory graph: an entity layer for internal decision structure and an experience layer for relations among full clinical experiences (Han et al., 23 Mar 2026). Each experience is represented as

$e_i=(c_i,s_i,z_i,Q_i),$

with condition, strategy, indication or contraindication, and quality score. Across MedR-Bench and MedAgentsBench, GSEM achieved the highest average accuracy among baselines, reaching 70.90% with DeepSeek-V3.2 and 69.24% with Qwen3.5-35B (Han et al., 23 Mar 2026). The same design impulse appears in trainable graph memory for LLM agents, where queries, FSM-abstracted transition paths, and meta-cognitions form a multi-layer graph whose weights are optimized by reinforcement learning (Xia et al., 11 Nov 2025).

A further step is to convert experience not merely into memories but into reusable audited capabilities. ASG-SI compiles successful trajectory fragments into a directed multigraph of skills, with verifier-backed replay, contract checks, evidence bundles, and append-only audit traces controlling promotion into the graph (Huang et al., 28 Dec 2025). In this formulation, the experience graph is explicitly tied to governance and reproducibility rather than only to retrieval quality.

5. Retrieval, evolution, and database semantics

A central technical issue is how an experience graph is queried and updated. ExpGraph provides one explicit answer: trajectories are summarized into “skills” and “failure lessons,” inserted as nodes in a self-evolving graph, connected to semantically similar neighbors, and retrieved through graph diffusion plus utility-aware ranking (Feng et al., 29 May 2026). Its retrieval copilot chooses diffusion depth and similarity-versus-utility weighting, while node utilities are updated online from downstream task outcomes. On ExpSuite, ExpGraph improved over the strongest baseline by 12.2% and 4.7% on static tasks with smaller and larger executors, and by 21.4% and 12.7% in agentic environments, while reducing average interaction steps by 12.7% and 21.6% (Feng et al., 29 May 2026).

GSEM provides a complementary update mechanism in which only the node qualities and edge weights of activated experiences are adjusted after each case, while the content of stored experiences is left unchanged (Han et al., 23 Mar 2026). This distinction is important: in some systems the graph evolves by adding and pruning memory items, whereas in others it evolves by recalibrating trust and relation strength over a fixed semantic substrate.

The most expansive systems view is given by Trellis, which treats the experience graph as first-class database state rather than as process-local memory (Liao et al., 29 Jun 2026). In that account, frontier selection is a query, cross-session reuse is vector-seeded graph retrieval, training-data extraction is a materialized view, and reconstructing what an agent knew at any past step is a time-travel query. The logical payload includes executable artifacts, tool outputs, objective rewards, sibling comparisons, mutable search statistics, and causal lineage. In KernelEvolve, this design reportedly allowed cross-session reuse to reach a target speedup roughly 10x faster at 52% lower token cost (Liao et al., 29 Jun 2026).

These systems collectively show that the experience graph is increasingly treated as a dynamic data plane. Retrieval is hybrid—graph, vector, relational, and temporal. Updating is often online and feedback-driven. In the most developed formulations, persistence, provenance, and audit are not peripheral engineering details but part of the graph’s semantics.

6. Misconceptions, limits, and emerging synthesis

A common misconception is that an experience graph is equivalent to any stored history. The recent literature explicitly rejects that equivalence. Trellis contrasts an experience graph with episodic logs by emphasizing queryability, traversal, versioning, mutability of search statistics, and direct extraction of training views (Liao et al., 29 Jun 2026). Likewise, GSEM argues that storing experiences as independent records without explicit relational structure can introduce noisy retrieval, unreliable reuse, and in some cases even hurt performance compared to direct LLM inference (Han et al., 23 Mar 2026).

A second misconception is that graph structure alone guarantees safe reuse. Several formulations retain explicit safeguards. Thunder and LTR* use lazy collision checking and repair because previously useful paths may become invalid in changed environments (Coleman et al., 2014, Lai et al., 2021). ASG-SI requires verifier-backed replay and contract checks before promotion into the skill graph (Huang et al., 28 Dec 2025). KG-Agent falls back on trial-and-error when high-quality skills fail, while continuing to update the SA-KG (Tang et al., 17 Oct 2025). The graph is therefore a memory of possibilities and dependencies, not a substitute for validation.

A third issue concerns granularity. Whole-trajectory retrieval is often too coarse for stateful or long-horizon workloads, but atomizing experience too aggressively can destroy compositional meaning. SIT-Graph addresses this by attaching compact state summaries to tool-transition edges rather than retrieving whole past episodes, while trainable graph memory for LLM agents abstracts trajectories first into FSM paths and then into meta-cognitions (Li et al., 8 Dec 2025, Xia et al., 11 Nov 2025). This suggests that successful experience-graph designs operate at an intermediate level of abstraction: structured enough for recombination, but semantically rich enough to remain actionable.

The emerging synthesis is that experience graphs function as reusable substrates for cumulative learning under non-i.i.d., long-horizon, or dynamically changing conditions. Their instantiations differ sharply across motion planning, RL, continual graph learning, clinical reasoning, GUI exploration, and self-improving LLM agents, yet they repeatedly converge on the same architectural claim: experience becomes more useful when its relational structure is retained, updated, and queried as a graph rather than discarded as disposable process state.