VORL-EXPLORE: A Hybrid Learning Planning Approach to Multi-Robot Exploration in Dynamic Environments

Abstract: Hierarchical multi-robot exploration commonly decouples frontier allocation from local navigation, which can make the system brittle in dense and dynamic environments. Because the allocator lacks direct awareness of execution difficulty, robots may cluster at bottlenecks, trigger oscillatory replanning, and generate redundant coverage. We propose VORL-EXPLORE, a hybrid learning and planning framework that addresses this limitation through execution fidelity, a shared estimate of local navigability that couples task allocation with motion execution. This fidelity signal is incorporated into a fidelity-coupled Voronoi objective with inter-robot repulsion to reduce contention before it emerges. It also drives a risk-aware adaptive arbitration mechanism between global A* guidance and a reactive reinforcement learning policy, balancing long-range efficiency with safe interaction in confined spaces. The framework further supports online self-supervised recalibration of the fidelity model using pseudo-labels derived from recent progress and safety outcomes, enabling adaptation to non-stationary obstacles without manual risk tuning. We evaluate this capability separately in a dedicated severe-traffic ablation. Extensive experiments in randomized grids and a Gazebo factory scenario show high success rates, shorter path length, lower overlap, and robust collision avoidance. The source code will be made publicly available upon acceptance.

• The paper introduces execution fidelity, an online signal that jointly guides Voronoi frontier assignment and switches between A* planning and reactive reinforcement learning.
• The hybrid system achieves success rates above 0.9 across tested traffic levels, including SR 0.96 on 80×80 grids with 64 dynamic obstacles, outperforming degraded baselines.
• Online gate adaptation provides the largest robustness gain in severe traffic, raising success from 0.36 to 0.69 while reducing recoveries from 82.4 to 6.8 per episode.

Motivation and problem statement

Hierarchical multi-robot exploration systems typically decouple frontier allocation from local motion execution, relying on the assumption that each robot can reliably execute its assigned target under local dynamics. The paper argues that this assumption fails in dense and dynamic environments: distance-driven allocators such as BFS-induced Voronoi partitioning dispatch multiple robots toward frontiers that share narrow passages, producing congestion, yielding oscillations, mutual blocking, and cascading replanning. Because allocators lack an online signal about execution difficulty and local controllers do not report execution confidence upstream, contested assignment patterns are regenerated even as routes become infeasible. VORL-EXPLORE addresses this structural brittleness by introducing a shared, continuously updated coupling signal between the task and motion layers.

Execution fidelity as a bidirectional coupling signal

The central construct is execution fidelity $p_{i,t}\in[0,1]$, a lightweight per-robot estimate of whether globally guided navigation will make reliable progress under current local interactions. It is computed by a logistic-regression gate over an eight-dimensional deterministic feature vector comprising local crowding, a stuck flag, normalized distance to goal, feasible-action ratio, unknown-area ratios around robot and goal, blockage density, and planner feasibility. This single signal serves two roles simultaneously:

• Task layer: fidelity modulates a fidelity-coupled Voronoi frontier score $$\Phi_{i,t}[f]=u_t[f]-\lambda(p_{i,t})\,d_{i,t}[f]-\rho(p_{i,t})\,r_{i,t}[f]$$, where low fidelity inflates distance and repulsion penalties, biasing assignment toward nearby, low-conflict frontiers. Repulsion combines pose-based and target-based exponential kernels over BFS distances.
• Motion layer: fidelity drives a hysteresis gate (dual thresholds with dwell length $K$) that arbitrates between global $A^*$ guidance and a reactive RL policy (EPOM-style, trained with PPO), suppressing oscillatory switching.

A recovery override handles planning infeasibility, stalled progress, and switch oscillation via symmetry-breaking maneuvers. Notably, inter-robot repulsion is evaluated only during reassignment rounds rather than per-step execution, keeping per-step computation limited to the gate evaluation.

Online self-supervised adaptation

Rather than fixed hand-crafted risk rules, the gate is recalibrated online from pseudo-labels derived from a sliding-window surrogate quality score combining coverage gain, BFS-based distance reduction, risk, and stall penalties. Updates use regularized binary cross entropy with a margin condition $|Q_{i,t}|\ge m$ to suppress noise-driven drift. This closes the loop: physical progress and safety outcomes update the fidelity estimator, which in turn reshapes both allocation and arbitration — the paper's claimed architectural contribution of a bidirectional closed loop replacing strictly top-down hierarchies.

Experimental results

Evaluation uses held-out $40\times40$ and $80\times80$ grids (30% static obstacle density, dynamic obstacles at speed ratio $\nu=0.5$), 100 runs per configuration, under an idealized synchronized shared-state protocol. Key findings:

Setting	VORL-EXPLORE	Strongest degraded baseline
$40\times40$, 64 dyn. obs.	SR 0.95, Overlap 0.21	PICO SR 0.51, Overlap 0.33
$80\times80$, 64 dyn. obs.	SR 0.96, Overlap 0.31	ICBS SR 0.31, Overlap 0.51

VORL-EXPLORE sustains success rates above 0.9 across all traffic levels while ICBS collapses from 0.85 to 0.31 as obstacles increase from 8 to 64 on $$\Phi_{i,t}[f]=u_t[f]-\lambda(p_{i,t})\,d_{i,t}[f]-\rho(p_{i,t})\,r_{i,t}[f]$$0. The ablated variants isolate the gating mechanism's value: planning-only VORL-A* degrades to SR 0.55 under heavy traffic, whereas policy-only VORL-RL reaches 0.92 but incurs longer exploration in sparse layouts (EL 203.20 vs. 181.32 for the hybrid at 8 obstacles). Allocator-level comparisons show auction allocation plateauing at EL 42.2 at maximum team size versus 25.4 for VORL-EXPLORE, which attains 100% success earlier than all baselines.

The severe-traffic ablation (128 dynamic obstacles) is the most informative result regarding adaptation: a cold static gate achieves only SR 0.36 with 82.4 recoveries per episode and selects the planner for 91.5% of steps; warm initialization alone raises SR to 0.51; but online updates dominate — the cold adaptive variant reaches SR 0.65 with 12.3 recoveries, and warm-plus-online reaches SR 0.69 with 6.8 recoveries and 34.5% planner selection. The implication is that test-time recalibration, not pre-training, is the primary mechanism maintaining gate calibration under non-stationary interaction regimes. A Gazebo study with four Pioneer3 robots and two moving pedestrians shows faster normalized coverage growth than ROS explore_lite without fine-tuning, though this is presented as qualitative proof-of-concept.

Limitations and open questions

The paper is explicit that all results rest on a synchronized shared-state communication assumption: delayed, lossy, or bandwidth-limited communication is not modeled, which is a significant constraint given that the fidelity signal depends on shared teammate poses and goals. Dynamic obstacles are treated as time-varying hard constraints within sensing range at a fixed speed ratio; adversarial or heterogeneous obstacle behavior is untested. The gate is deliberately restricted to logistic regression over eight hand-designed features, so its expressiveness relative to learned representations remains unexamined, and the surrogate quality weights ($$\Phi_{i,t}[f]=u_t[f]-\lambda(p_{i,t})\,d_{i,t}[f]-\rho(p_{i,t})\,r_{i,t}[f]$$1), hysteresis thresholds, and repulsion scales are fixed hyperparameters whose sensitivity is not reported. The Gazebo validation involves only two pedestrians at constant velocity, leaving scalability of the coupled architecture to richer human dynamics an open question. Finally, the objective weights the completion time term far above redundancy ($$\Phi_{i,t}[f]=u_t[f]-\lambda(p_{i,t})\,d_{i,t}[f]-\rho(p_{i,t})\,r_{i,t}[f]$$2), so behavior under alternative trade-off preferences is not characterized.

Conclusion

VORL-EXPLORE contributes a decentralized hybrid learning-and-planning framework in which a single shared execution-fidelity estimate simultaneously modulates Voronoi frontier scoring and arbitrates between $$\Phi_{i,t}[f]=u_t[f]-\lambda(p_{i,t})\,d_{i,t}[f]-\rho(p_{i,t})\,r_{i,t}[f]$$3 guidance and a reactive RL policy, with self-supervised online recalibration closing the allocation–execution loop. Empirically, the framework maintains success rates near 0.95–0.99 across grid benchmarks where conflict-based and learning-based executors degrade sharply under dense dynamic traffic, and ablations attribute the dominant robustness benefit to online gate adaptation rather than warm-start initialization. The results are conditional on idealized synchronized communication, and extending the coupling signal to realistic communication regimes remains unresolved by this work.