---
title: 'CORE Planner: Contextual Memory RL'
url: https://www.emergentmind.com/topics/contextual-memory-oriented-reinforcement-learning-core-planner
type: topic
---

# CORE Planner: Contextual Memory RL

Contextual-memory Oriented Reinforcement-learning (CORE) Planner is a reinforcement-learning-based navigation framework for unknown environments in which a robot must reach a predefined goal without any prior map while continuously updating its belief about free space, obstacles, and unknown regions. The method combines a sparse visibility graph for structured environment representation, a Transformer network for holistic environmental understanding, a graph sparsification method for large-scale scenes, and a contextual memory mechanism that encodes historical motion to alleviate local optima and improve navigation efficiency. In the reported experiments, CORE Planner is trained solely on image-based environments and transfers to real robots in a zero-shot manner, with no fine-tuning [2606.29222].

## 1. Problem setting and design rationale

CORE Planner is formulated for autonomous navigation in unknown environments under partial observability. At each step, the robot has only a local and incomplete belief of the world, and must infer useful next-waypoint decisions from evolving sensor observations. The paper emphasizes a characteristic failure mode of such settings: a waypoint that appears locally promising may later become invalid or lead into a dead end, producing back-and-forth motion or oscillation. This failure is described as **Navigation Deadlock Oscillation (NDO)**, especially in room-to-room navigation, where the robot may repeatedly alternate between apparently favorable waypoints [2606.29222].

The method is motivated by limitations in two established families of navigation systems. Traditional planners, including Dijkstra, A*, D*, LPA*, D* Lite, RRT variants, FAR Planner, E-Planner, and FPS, rely on hand-crafted rules and heuristic costs; dense occupancy-grid representations can also become computationally heavy in large unknown environments. Prior learning-based planners, including NavRL, YOPO, CTSAC, CADRL, and GDAE-style hybrid planners, often use fixed-size CNN or RGB-D inputs, have short memory or shallow sensor history, and are frequently local planners rather than long-horizon planners. CORE Planner is presented as combining the sparse, structured geometry of visibility graphs with learned decision-making and a contextual memory mechanism that records historical motion, thereby addressing both local optima and scalability [2606.29222].

A central conceptual move is to reject purely greedy waypoint selection. The classical next-waypoint objective is written as
$$
p^*(t)=\arg\min_{p\in\mathcal{P}(t)}\; \mathrm{Cost}\!\big(p;\; S(t-1), B(t-1)\big),
$$
where $\mathcal{P}(t)$ is the set of admissible next waypoints or local plans. CORE extends this to a history-aware objective,
$$
p^*(t)=\arg\min_{p\in\mathcal{P}(t)}\; \mathrm{Cost}\!\big(p;\; S(0),\ldots,S(t-1), B(t-1)\big),
$$
but does not literally preserve the full trajectory in raw form; instead, it compresses history into node-wise contextual memory [2606.29222].

## 2. Formal RL formulation

The environment is modeled as a $2$D occupancy grid map $E$, partitioned into free space $E_f$ and obstacle space $E_o$. The robot maintains a belief over believed obstacle space $B_o$, unknown space $B_u$, and believed free space $B_f$, and updates this belief through sensor measurements $M$ according to
$$
B = B \cup M.
$$
This provides the continuously updated partial world model on which planning operates [2606.29222].

CORE Planner is cast as an MDP,
$$
\{S, A, P, R, \gamma\},
$$
with policy objective
$$
\pi^* = \arg\max_{\pi} \mathbb{E}_{\pi}\!\left[ \sum_{\tau=0}^{T_{\mathrm{ep}}} \gamma^{\tau} R(s_{\tau}, a_{\tau}) \right].
$$
After constructing a visibility graph, the action is to choose one of the neighboring visible nodes as the next waypoint. Because the candidate set is variable-sized, the action selector is implemented as a Pointer Network [2606.29222].

The reward is a weighted sum of five terms,
$$
r = \lambda_{1}r_{\mathrm{goal}} + \lambda_{2}r_{\mathrm{frontier}} + \lambda_{3}r_{\mathrm{ds}} + \lambda_4r_{\mathrm{stay}} + \lambda_5r_{\mathrm{f}}.
$$
The goal-distance term encourages progress toward the goal,
$$
r_{\mathrm{goal}} = \frac{1}{\|r_t-\mathrm{goal}\|}.
$$
The frontier term rewards newly observed frontier points,
$$
r_{\mathrm{frontier}} = F_t - F_{t-1}.
$$
The position-difference term penalizes stagnation,
$$
r_{\mathrm{ds}} = -\| r_t - r_{t-1} \|.
$$
The stagnation penalty is
$$
r_{\mathrm{stay}} =
\begin{cases}
\mathrm{stay} - 5, & \text{if } \mathrm{stay} > 5,\\
0, & \text{otherwise},
\end{cases}
$$
and the finish reward is
$$
r_{\mathrm{f}} =
\begin{cases}
20, & \text{if } \| r_t - \mathrm{goal} \| < \epsilon,\\
0, & \text{otherwise}.
\end{cases}
$$
Taken together, these terms reward goal progress, exploration, movement continuity, escape from deadlock, and task completion [2606.29222].

## 3. Visibility-graph state representation and contextual memory

CORE Planner represents the navigation state at time $t$ as a graph
$$
G_t = (N_t, E_t),
$$
where $N_t$ are nodes and $E_t$ are edges between collision-free visible nodes. Each node $n_i \in N_t$ carries four components: spatial coordinates $p_i$, utility $u_i$, distance to goal $d_i$, and trajectory indicator $v_i$. The corresponding node feature is
$$
\mathbf{x}_i = [p_i \oplus u_i \oplus d_i \oplus v_i].
$$
The use of a visibility graph rather than a dense occupancy grid is justified as a topological abstraction that retains essential geometry and connectivity while removing redundant spatial cells, thereby reducing computational overhead in large-scale exploration [2606.29222].

The paper adds a further graph sparsification stage. Nodes within a non-sparsified region around the robot are kept unchanged, while farther nodes are clustered using a distance threshold and visibility-connectivity constraints, then replaced by centroid representatives. For a cluster $C$, the centroid is
$$
\mathbf{p}_{\text{centroid}} = \frac{1}{|C|} \sum_{n_j \in C} \mathbf{p}_{n_j}.
$$
The resulting complexity is described as
$$
T_{\mathrm{sparsify}} \approx O(N \log N), \qquad
T_{\mathrm{transformer}} = O(N'^2), \qquad
T_{\mathrm{total}} = O(N \log N + N'^2),
$$
in contrast to the non-sparsified
$$
T_{\mathrm{raw}} = O(N^2).
$$
The paper notes that $\alpha = N'/N$ is empirically small, often $\alpha \leq 0.5$ in large scenes [2606.29222].

The contextual memory mechanism is encoded through the trajectory indicator
$$
v_i = \sum_{\tau=1}^{t} \mathbb{I}\!\left(D(r_\tau, p_i) < \epsilon\right),
$$
which counts how many times the robot has visited or come near node $n_i$. This is explicitly more informative than a binary visited or unvisited flag. Because $v_i$ is embedded directly into $\mathbf{x}_i$, the policy can identify repeatedly chosen dead-end regions, suppress cyclic behavior, and maintain long-term consistency in path reasoning. The paper argues that this augmentation preserves Markovity by defining an augmented state space
$$
\mathcal{S}' = \mathcal{S} \times \mathcal{V},
$$
with transition
$$
P(s'_{t+1} \mid s'_t, a_t) = P(s_{t+1}, v_{t+1} \mid s_t, v_t, a_t),
$$
since the memory update is deterministic given actions [2606.29222].

## 4. Transformer architecture and SAC training

The network backbone is a Transformer encoder-decoder operating on graph-structured inputs. Node features are stacked as
$$
\mathbf{X} = \big[\mathbf{x}_1\ \mathbf{x}_2\ \dots\ \mathbf{x}_N\big]^{\top} \in \mathbb{R}^{N \times D_x},
$$
then projected into latent space through
$$
\mathbf{H}^{(0)} = \text{ReLU}(\mathbf{X}\cdot W_{emb} + \mathbf{B}_{emb}),
$$
where $W_{emb} \in \mathbb{R}^{D_x \times D_h}$ and $\mathbf{B}_{emb} \in \mathbb{R}^{N \times D_h}$. A graph-based mask $M$ constrains encoder attention so that nodes attend according to graph connectivity rather than globally. The encoder output, denoted $h_e$, is described as **environment-aware features** combining spatial structure, node utilities, goal distance, and visitation memory. In the decoder, the current robot node feature $h_r$ serves as the query, while $h_e$ serves as keys and values. The pointer attention layer converts the decoded representation and neighboring node embeddings into attention weights $\theta$, which are directly interpreted as the action distribution over next waypoints [2606.29222].

Training uses **Soft Actor-Critic (SAC)**. The paper gives the soft Q-function
$$
Q^\pi(s,a) = r(s,a) + \gamma \mathbb{E}_{s' \sim p}\big[V^\pi(s')\big],
$$
the soft value function
$$
V^\pi(s) = \mathbb{E}_{a\sim\pi}\big[Q^\pi(s,a) - \alpha\log\pi(a|s)\big],
$$
the soft Bellman operator
$$
(\mathcal{T}^\pi Q)(s,a) = r(s,a) + \gamma \mathbb{E}_{s' \sim p}\big[V^\pi(s')\big],
$$
and the entropy-regularized objective
$$
\pi^\ast = \arg\max_\pi \mathbb{E}_{s\sim\mathcal{D}, a\sim\pi} \big[ Q^\pi(s,a) - \alpha\log\pi(a|s) \big].
$$
The paper states that convergence is preserved because node features are bounded, $\|x_i\|_2 \le B$, and the augmented state remains Markovian [2606.29222].

The training environment uses the environment generator from Chen et al., with map size $500 \times 500$ grid, physical size $100m \times 100m$, and sensor range $16m$. Training includes simple maps and complex maps; testing uses only complex unseen configurations. The reported optimization details are maximum episode length $250$, Adam, learning rate $2 \times 10^{-5}$, replay buffer size $20{,}000$, and batch size $256$, on an Intel Xeon E5-2660 v4 CPU and $3 \times$ NVIDIA RTX 2080 Ti GPUs. Training is reported to converge in about five hours in an image-based simulated environment, without Gazebo or Isaac Sim [2606.29222].

## 5. Empirical performance, ablations, and sim-to-real transfer

The evaluation covers image-based unseen environments, Gazebo simulation, and real-world deployment, using success rate $S$, travel distance $D(m)$, inference time $T_e(s)$, planning time $T(s)$, and human interventions $I$ as metrics. Across these settings, the paper reports that CORE Planner consistently outperforms the traditional FAR Planner and all learning-based baselines, with larger gains in more complex environments [2606.29222].

| Setting | Baseline(s) | Reported CORE outcome |
|---|---|---|
| Image-based unseen environments | CADRL: success \(97\%\), distance \(391.74\), time \(0.21s\) | success \(100\%\), distance \(310.10\), time \(0.30s\); travel distance reduced by **20.8%** |
| Gazebo simulation | FAR Planner, CTSAC, NavRL, YOPO | up to **13% reduction** in travel distance over FAR Planner; simple indoor: **18.3%** vs CTSAC, **23.3%** vs NavRL, **14.7%** vs YOPO; complex indoor: **37.1%** vs YOPO; forest: **48%** vs NavRL and **30.1%** vs YOPO |
| Real-world Scout Mini | FAR: \(D=132.21\), \(T=0.73\), \(I=3\) | \(D=109.07\), \(T=0.12\), \(I=0\) |
| Real-world LYNX M20 | No quantitative FAR comparison given | completed all trials without intervention |

The image-based result is notable because the baseline CADRL also performs strongly, with \(97\%\) success, yet CORE reaches \(100\%\) and shortens travel distance from \(391.74\) to \(310.10\). In Gazebo, the paper highlights that CTSAC and NavRL fail in the complex indoor setting, and CTSAC fails in the forest setting, whereas CORE remains operational. Inference time for CORE is reported as around \(5\) ms in the simple indoor environment and around \(10\) ms in the forest environment, while FAR Planner’s inference time grows from \(0.002s\) in Simple to \(0.120s\) in Forest, which the authors attribute to CORE’s sparse topological representation [2606.29222].

The ablation study isolates the two main modules. **CORE w/o Sparse** retains contextual memory but removes graph sparsification; **CORE w/o Mem** retains sparsification but removes contextual memory. In the simple scenario, the variants behave similarly because sparsification is rarely activated. In larger environments, however, removing memory causes substantial degradation. In Indoor, **CORE w/o Mem** produces \(1036.1\) m travel distance with \(5\) interventions, versus CORE at \(681.1\) m and \(0\) interventions. In Forest, **CORE w/o Mem** yields \(386.8\) m versus CORE at \(272.0\) m. The paper interprets this as evidence that sparsification primarily improves efficiency, whereas contextual memory is responsible for robustness against oscillatory motion and NDO [2606.29222].

The real-world experiments use two different platforms: a **LYNX M20 quadruped** with a ZED2 RGB-D camera and NVIDIA Jetson AGX Orin, and a **Scout Mini UGV** with a Livox Mid-360 LiDAR and NVIDIA Jetson Orin NX. Tasks include multi-room local-optimum escape and dynamic obstacle avoidance. The reported zero-intervention performance on both platforms is presented as evidence of cross-sensor, cross-robot, and cross-hardware generalization under zero-shot sim-to-real transfer [2606.29222].

## 6. Position within contextual-memory RL research and open issues

Within the recent literature, CORE Planner belongs to a broader class of architectures that treat planning quality and memory organization as central design variables rather than auxiliary modules. In long-horizon digital-task automation, a planner-actor-memory decomposition has been used to show that scaling the planner yields the largest marginal gain, while execution and memory management can remain smaller; the same work reports that planner-only GRPO training with frozen actor and memory manager is the most stable RL strategy [2605.02168]. In memory-augmented LLM systems, Mem-\(\alpha\) formulates memory construction itself as a sequential decision problem over `memory_insert`, `memory_update`, and `memory_delete`, with GRPO optimizing a write policy over core, episodic, and semantic memory [2509.25911]. CoDA likewise separates a short strategic Planner context from a temporary Executor context to mitigate “Context Explosion,” using PECO for joint RL optimization of both roles [2512.12716]. These results suggest a more general research tendency: contextual memory is increasingly being treated as a mechanism for preserving strategic state while limiting the computational and statistical costs of long-horizon decision making.

The acronym **CORE** is also used in a distinct line of work on active causal discovery, where it denotes **Causal DiscOvery with REinforcement learning** rather than navigation. That method formulates causal discovery as a POMDP and uses history-based context together with dual DQN controllers for intervention and structural actions [2401.16974]. The naming overlap is terminological rather than methodological; the robot-navigation CORE Planner is specifically the **Contextual-memory Oriented Reinforcement-learning** framework described above.

The navigation paper identifies a concrete limitation: in real-world scenarios, visibility graph extraction may become less reliable in complex environments with irregular geometry, which can make structural nodes less informative and cause failures. The authors also indicate future work on integrating semantic information into the navigation framework [2606.29222]. A plausible implication is that future versions of CORE Planner may move toward richer memory representations that combine geometric topology, trajectory history, and semantic cues, thereby narrowing the gap between robot navigation and the planner-centric contextual-memory systems now appearing in language-agent research.

Source: https://www.emergentmind.com/topics/contextual-memory-oriented-reinforcement-learning-core-planner