ExploRLer: Dual Exploration in RL

• ExploRLer is a dual-use term describing two distinct exploration approaches in reinforcement learning: one for robot mapping in a POMDP and one for enhancing policy updates via zero-order search.
• The robotic version leverages deep RL with graph attention, privileged critic training, and graph rarefaction to efficiently uncover unknown frontiers in large-scale maps.
• The on-policy variant periodically augments PPO/TRPO by exploring sparse checkpoint subspaces with zero-order search to correct surrogate-gradient drift and rescue learning.

Searching arXiv for the two ExploRLer usages to ground the article in current literature. ExploRLer is a name used in recent reinforcement-learning literature for two distinct methods rather than a single standardized algorithm. In "Deep Reinforcement Learning-based Large-scale Robot Exploration" (Cao et al., 2024), it denotes a deep reinforcement learning-based reactive planner for large-scale LiDAR-based autonomous robot exploration in a 2D action space. In "Efficient On-Policy Reinforcement Learning via Exploration of Sparse Parameter Space" (Zhang et al., 30 Sep 2025), it denotes a pluggable augmentation for on-policy policy-gradient methods such as PPO and TRPO, designed to probe sparse neighborhoods in parameter space. The shared label reflects a common emphasis on exploration, but the two systems operate at different levels: one explores physical space through graph-based navigation, whereas the other explores policy parameter space through periodic zero-order search.

1. Terminological scope and conceptual split

The robot-exploration ExploRLer addresses autonomous mapping under partial observability. Its central problem is how an agent should select successive waypoints from a local graph built over the current occupancy belief so as to uncover unknown frontier regions efficiently. The method is reactive, graph-based, and trained with deep reinforcement learning; it also uses privileged learning during training and a graph rarefaction procedure to transfer from small-scale training environments to large-scale deployment (Cao et al., 2024).

The on-policy ExploRLer addresses a different problem: surrogate-gradient updates in PPO and TRPO can drift through parameter space and miss nearby higher-return solutions. This version of ExploRLer preserves the base optimizer entirely and periodically performs an "empty-space search" over recent checkpoints, evaluates candidate policies with rollouts, and jumps to the best candidate before continuing training (Zhang et al., 30 Sep 2025).

A common misconception is to treat ExploRLer as a single RL framework with one architecture and one training protocol. The literature considered here does not support that reading. The same name identifies two separate contributions with different state representations, optimization procedures, and evaluation regimes.

2. ExploRLer as a POMDP for large-scale robot exploration

In the robotic formulation, exploration is cast as a POMDP with true state

$s_t = (E, M_t, \psi_{1:t}),$

where $E=(E_f,E_o)$ is the underlying unknown environment partitioned into free and occupied cells, $M_t=(M_f,M_o,M_u)$ is the robot's current occupancy-grid belief over free, occupied, and unknown cells, and $\psi_{1:t}=[p_1,\ldots,p_t]$ is the sequence of executed positions (Cao et al., 2024). The observation space is the belief

$o_t = (M_t,\psi_{1:t}),$

and the action space consists of neighboring nodes in a local collision-free graph $G$ constructed over $M_t$:

$A_t = \{ v_i \in V_t : (p_t,v_i)\in E_t \}.$

An action $a_t \sim \pi(\cdot \mid o_t)$ selects the next waypoint.

The transition and observation models are implicitly defined by SLAM plus sensing and are treated as unknown in a model-free approach. The reward at each step is

$r_t = a\cdot r_c + b\cdot r_e + r_f,$

where $E=(E_f,E_o)$0 is the negative traversal cost, $E=(E_f,E_o)$1 is the number of newly observed frontier cells uncovered by choosing $E=(E_f,E_o)$2, and $E=(E_f,E_o)$3 is a fixed finishing bonus awarded when exploration completes. The reported typical scaling is $E=(E_f,E_o)$4, $E=(E_f,E_o)$5, and $E=(E_f,E_o)$6. The policy objective is the expected discounted sum of shaped rewards with entropy regularization,

$E=(E_f,E_o)$7

with $E=(E_f,E_o)$8 set to $E=(E_f,E_o)$9.

This formulation is notable for combining path cost, information gain, and completion incentive in a single shaped reward. A plausible implication is that the method is designed not merely to reach frontiers greedily, but to trade off immediate motion cost against longer-horizon map revelation.

3. Graph representation, attention architecture, and privileged training

The robot-exploration ExploRLer uses an attention-based encoder-decoder over a graph representation of the map (Cao et al., 2024). The collision-free graph $M_t=(M_f,M_o,M_u)$0 is built by sampling points uniformly over known free cells $M_t=(M_f,M_o,M_u)$1. Each node $M_t=(M_f,M_o,M_u)$2 carries two scalar features: a utility

$M_t=(M_f,M_o,M_u)$3

and a guidepost bit

$M_t=(M_f,M_o,M_u)$4

which equals $M_t=(M_f,M_o,M_u)$5 if the node has been visited. The informative graph is

$M_t=(M_f,M_o,M_u)$6

The encoder is a stacked self-attention network. Each node receives an initial embedding $M_t=(M_f,M_o,M_u)$7 from a shared MLP. For each of $M_t=(M_f,M_o,M_u)$8 layers, the model computes

$M_t=(M_f,M_o,M_u)$9

then

$\psi_{1:t}=[p_1,\ldots,p_t]$0

with attention masked by graph adjacency so that $\psi_{1:t}=[p_1,\ldots,p_t]$1 if $\psi_{1:t}=[p_1,\ldots,p_t]$2. The updated node feature is

$\psi_{1:t}=[p_1,\ldots,p_t]$3

The stated purpose of stacking six layers is to propagate information across multiple hops and capture long-range spatial dependencies.

The decoder is pointer-style. If $\psi_{1:t}=[p_1,\ldots,p_t]$4 is the encoded feature of the node at the robot's current position, the model first forms a global context

$\psi_{1:t}=[p_1,\ldots,p_t]$5

then

$\psi_{1:t}=[p_1,\ldots,p_t]$6

For each neighbor $\psi_{1:t}=[p_1,\ldots,p_t]$7 it computes

$\psi_{1:t}=[p_1,\ldots,p_t]$8

and selects the action distribution by

$\psi_{1:t}=[p_1,\ldots,p_t]$9

Training adopts discrete-action Soft Actor-Critic. The distinctive element is privileged critic training: the critic receives a richer graph $o_t = (M_t,\psi_{1:t}),$0 sampled over the true free cells $o_t = (M_t,\psi_{1:t}),$1 rather than the belief free cells $o_t = (M_t,\psi_{1:t}),$2. Ground-truth node features are

$o_t = (M_t,\psi_{1:t}),$3

where $o_t = (M_t,\psi_{1:t}),$4 indicates whether that node has already been explored. The paper states that this reduces variance in the Q-loss by making the critic solve a fully observed MDP. The losses are the standard SAC critic, actor, and temperature objectives,

$o_t = (M_t,\psi_{1:t}),$5

$o_t = (M_t,\psi_{1:t}),$6

$o_t = (M_t,\psi_{1:t}),$7

$o_t = (M_t,\psi_{1:t}),$8

with optimization solving

$o_t = (M_t,\psi_{1:t}),$9

Reported auxiliary settings include replay-buffer size $G$0 steps, batch size $G$1, $G$2 updates per episode, target-network updates every $G$3 parameter updates, learning rates of $G$4 for policy and critic and $G$5 for $G$6, and training on $G$7 randomized dungeon environments for approximately $G$8 days on an RTX4080.

4. Graph rarefaction, scaling behavior, and hardware validation

To transfer a policy trained in small-scale environments to large maps, the robotic ExploRLer introduces graph rarefaction (Cao et al., 2024). When the map becomes large, a uniform $G$9-NN informative graph can be prohibitively large. The method therefore extracts a sparse informative graph

$M_t$0

that preserves all nonzero-utility nodes together with a minimal set of connector nodes. The procedure keeps all frontier-adjacent nodes, groups nearby frontier nodes by adjacency within a visibility radius $M_t$1, runs A* shortest-path searches from the robot position to representative utility nodes, and inserts intermediate connector waypoints whenever line-of-sight or distance constraints are violated. The stated per-search complexity is $M_t$2, but the search is run once per utility group rather than once per node. Empirically, the resulting sparse graph has approximately $M_t$3-$M_t$4 of the nodes of the full informative graph in large office-scale maps while preserving long-range task-relevant nodes and keeping inference below $M_t$5 ms.

The main quantitative results are reported on a $M_t$6 Gazebo office benchmark with 3D LiDAR and real-time drive constraints, as well as on $M_t$7 small-scale dungeon maps.

Planner	Path length $M_t$8	Makespan $M_t$9 / planning time $A_t = \{ v_i \in V_t : (p_t,v_i)\in E_t \}.$0
DSVP	1462 m	870 s / 0.90 s
TARE	1158 m	634 s / 0.24 s
ExploRLer	1020 m	590 s / 0.15 s

Relative to TARE, the reported office-scale performance for ExploRLer is $A_t = \{ v_i \in V_t : (p_t,v_i)\in E_t \}.$1 in path length, $A_t = \{ v_i \in V_t : (p_t,v_i)\in E_t \}.$2 in makespan, and $A_t = \{ v_i \in V_t : (p_t,v_i)\in E_t \}.$3 in per-step planning time. On the small-scale benchmark, TARE-Local achieves approximately $A_t = \{ v_i \in V_t : (p_t,v_i)\in E_t \}.$4 px path length and ExploRLer approximately $A_t = \{ v_i \in V_t : (p_t,v_i)\in E_t \}.$5 px, a reduction of $A_t = \{ v_i \in V_t : (p_t,v_i)\in E_t \}.$6. All reported improvements pass a paired $A_t = \{ v_i \in V_t : (p_t,v_i)\in E_t \}.$7-test at $A_t = \{ v_i \in V_t : (p_t,v_i)\in E_t \}.$8.

Hardware validation uses a custom four-wheel TurtleBot3 with maximum speed $A_t = \{ v_i \in V_t : (p_t,v_i)\in E_t \}.$9 m/s, onboard 3D LiDAR, and LOAM for pose and mapping in an $a_t \sim \pi(\cdot \mid o_t)$0 cluttered lab with moving pedestrians. The reported settings are OctoMap resolution $a_t \sim \pi(\cdot \mid o_t)$1 m, node spacing $a_t \sim \pi(\cdot \mid o_t)$2 m, local $a_t \sim \pi(\cdot \mid o_t)$3-NN $a_t \sim \pi(\cdot \mid o_t)$4, sparse $a_t \sim \pi(\cdot \mid o_t)$5-NN $a_t \sim \pi(\cdot \mid o_t)$6, $a_t \sim \pi(\cdot \mid o_t)$7 m, and replanning every $a_t \sim \pi(\cdot \mid o_t)$8 s. Exploration completes in approximately $a_t \sim \pi(\cdot \mid o_t)$9 min with per-step planning around $r_t = a\cdot r_c + b\cdot r_e + r_f,$0 s, and no further fine-tuning or domain randomization is used. This suggests that the graph abstraction, rather than domain-specific visual adaptation, carries much of the sim-to-real robustness claimed by the authors.

5. ExploRLer as iteration-level parameter-space exploration for on-policy RL

In the second usage, ExploRLer is an augmentation for on-policy reinforcement learning in parameter space rather than a robot planner (Zhang et al., 30 Sep 2025). The starting point is the expected discounted return

$r_t = a\cdot r_c + b\cdot r_e + r_f,$1

and the policy-gradient theorem

$r_t = a\cdot r_c + b\cdot r_e + r_f,$2

PPO uses the clipped surrogate

$r_t = a\cdot r_c + b\cdot r_e + r_f,$3

with

$r_t = a\cdot r_c + b\cdot r_e + r_f,$4

The paper's premise is that surrogate gradients can be poorly aligned with the true return landscape, so PPO or TRPO checkpoints may drift past nearby higher-return regions.

ExploRLer leaves the underlying PPO/TRPO update unchanged. Once every $r_t = a\cdot r_c + b\cdot r_e + r_f,$5 iterations, it collects the iteration-end checkpoints into an anchor set

$r_t = a\cdot r_c + b\cdot r_e + r_f,$6

runs a zero-order "empty-space search" (ESA) in the subspace spanned by those anchors, evaluates the resulting candidates by full-episode rollouts, and replaces the current parameters with the best candidate. The high-level pipeline is:

1. Collect rollouts with $r_t = a\cdot r_c + b\cdot r_e + r_f,$7.
2. For each on-policy epoch $r_t = a\cdot r_c + b\cdot r_e + r_f,$8, update $r_t = a\cdot r_c + b\cdot r_e + r_f,$9 via PPO or TRPO on the original surrogate objective.
3. Let $E=(E_f,E_o)$00 and store it in $E=(E_f,E_o)$01.
4. If $E=(E_f,E_o)$02, apply ESA to generate candidates $E=(E_f,E_o)$03, evaluate each candidate with $E=(E_f,E_o)$04 full episodes to estimate $E=(E_f,E_o)$05, and set $E=(E_f,E_o)$06; otherwise continue with $E=(E_f,E_o)$07.

The reported experimental setting is $E=(E_f,E_o)$08 and $E=(E_f,E_o)$09, so the anchor set contains $E=(E_f,E_o)$10 checkpoints. ESA uses $E=(E_f,E_o)$11 agents, neighborhood size $E=(E_f,E_o)$12, search length $E=(E_f,E_o)$13, and step size $E=(E_f,E_o)$14. The agent interactions are governed by a Lennard-Jones-style potential

$E=(E_f,E_o)$15

with force

$E=(E_f,E_o)$16

where $E=(E_f,E_o)$17 is the distance to the nearest anchor neighbor and $E=(E_f,E_o)$18 is the unit vector. The total candidate count per ESA call is $E=(E_f,E_o)$19, comprising rollout initial anchors plus the $E=(E_f,E_o)$20 search agents. No gradient information is used in ESA.

6. Empirical behavior, visualization, and limitations of the on-policy method

The on-policy ExploRLer is evaluated as ExploRLer-P with PPO and ExploRLer-T with TRPO on seven tasks: Pendulum, BipedalWalker, and five MuJoCo locomotion environments (Zhang et al., 30 Sep 2025). Reported maximum average returns are given as mean $E=(E_f,E_o)$21 standard deviation over $E=(E_f,E_o)$22 seeds.

Environment	ExploRLer-P vs PPO	ExploRLer-T vs TRPO
Ant	4574.0±190.2 vs 4433.7±71.0	3653.3±951.6 vs 3613.6±693.1
Hopper	3318.5±82.6 vs 2234.0±934.8	1060.5±640.3 vs 556.3±76.0
Walker2d	3762.5±467.4 vs 3647.2±506.4	3778.4±515.2 vs 3263.8±575.4
Humanoid	739.4±51.0 vs 547.3±121.8	686.96±83.9 vs 568.9±78.1
HalfCheetah	2262.2±1162.4 vs 2237.4±1029.8	2748.2±1359.4 vs 2589.4±1360.4
Pendulum	-152.9±129.9 vs -154.7±128.4	-222.6±96.6 vs -222.3±97.2
BipedalWalker	293.2±6.8 vs 284.1±11.0	174.7±117.1 vs -18.2±31.7

The paper also reports training-curve behavior in which ESA jumps often rescue learning when PPO alone would plateau or dip. A further line of evidence comes from visualization: at three training stages, the authors project $E=(E_f,E_o)$23 epoch checkpoints into two-dimensional PCA space, fit a Gaussian, sample $E=(E_f,E_o)$24 probe parameters, and plot contour maps of average returns. These maps reveal empty valleys of high reward lying near, but off, the surrogate-gradient trajectory. This supports the paper's interpretation that iteration-level checkpoint clouds contain exploitable local structure missed by standard on-policy updates.

The method's stated strengths are zero additional gradient overhead at the mini-batch level, plug-in compatibility with on-policy algorithms, and empirical consistency across classic control and MuJoCo tasks. Its stated limitations are extra sample cost for policy evaluation, off-policy integration that remains nontrivial, and hyperparameter sensitivity. The reported evaluation overhead is that running $E=(E_f,E_o)$25 candidate policies for $E=(E_f,E_o)$26 episodes each adds an average of $E=(E_f,E_o)$27 environment rollouts per iteration, although the rollouts can be parallelized. Preliminary tests with SAC and pure off-policy evaluation through FQE are described as unstable. Suggested future directions include learned world models, DualDICE or FQE hybrids, safe insertion of new actor parameters into off-policy algorithms such as SAC or TD3, adaptive schedules for ESA invocation, and richer probing strategies such as Bayesian optimization in the local anchor subspace.

Taken together, the two ExploRLer usages illustrate two different meanings of exploration in contemporary RL. In one case, exploration concerns a robot's traversal of an unknown occupancy map using graph attention, privileged critic learning, and sparse graph transfer. In the other, exploration concerns deliberate search of sparse regions in checkpoint-derived parameter subspaces to correct surrogate-objective drift. The shared name therefore marks a thematic affinity rather than a unified technical lineage.