PILOC: Deep MARL Framework for Search & Rescue

• PILOC is a multi-agent reinforcement learning framework designed for cooperative search and rescue in dynamic, partially observable grid environments.
• It integrates inverse pheromone guidance with local communication to reduce redundant exploration and enhance decentralized coordination.
• The framework uses centralized training with decentralized execution via MAPPO, achieving high success rates in complex, unknown settings.

Searching arXiv for “PILOC” and closely related entries to ground the article in current papers. Searching arXiv for exact paper (Liu et al., 10 Jul 2025) and nearby similarly named systems to disambiguate terminology. PILOC is a deep multi-agent reinforcement learning framework for cooperative search and rescue in dynamic, partially observable, communication-constrained environments. The name denotes a design built around Pheromone Inverse guidance and LOcal Communication, and the framework is proposed for dynamic target search in completely unknown environments, where the map is not known in advance, targets move over time, each agent has only local sensing, communication is limited, and coordination must remain decentralized at execution time (Liu et al., 10 Jul 2025). Its central premise is to combine two coordination channels: implicit coordination through a decaying virtual pheromone field that repels agents from heavily visited regions, and explicit coordination through proximity-triggered local map exchange. In the reported formulation, PILOC is implemented in a centralized-training, decentralized-execution regime using MAPPO, with the pheromone mechanism embedded directly into the observation space of the learned policy rather than treated as an external planner (Liu et al., 10 Jul 2025).

1. Problem setting and formalization

PILOC addresses Multi-Agent Search and Rescue (MASAR) in a 2D discretized grid map whose cells are either passable or impassable, with the passable region assumed to be fully connected (Liu et al., 10 Jul 2025). The setting is explicitly characterized by a conjunction of difficulties: unknown map structure, dynamic targets, partial observability, limited communication, and a decentralized coordination requirement. This makes the framework distinct from methods that assume static targets, globally known maps, persistent global communication, or centralized planning.

Targets are modeled as multiple moving entities $G$, with each target $g_i \in G$ initialized randomly in free space. At each time step, a target chooses uniformly from the 4-neighbor motion set

$$A_g = \{ \leftarrow, \uparrow, \rightarrow, \downarrow \}.$$

If a sampled move would collide with an obstacle, the action is re-sampled until a feasible move is found. Agents are homogeneous and indexed by

$U \triangleq \{1,2,\dots,U\},$

with each agent assigned a perception radius $v$, a communication radius $c$, and a discrete action space

$$A = \{\leftarrow, \uparrow, \rightarrow, \downarrow\}.$$

Without communication, an agent does not know the global map, target locations, or the explored areas of other agents. The mission objective is to find all targets within a finite time limit (Liu et al., 10 Jul 2025).

The problem is formulated as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP),

$\langle N, S, A, O, \Omega, R, T, \gamma \rangle,$

where $N$ is the number of agents, $S$ the state space, $g_i \in G$0 the joint action space, $g_i \in G$1 the observation spaces, $g_i \in G$2 the observation model, $g_i \in G$3 the reward function, $g_i \in G$4 the transition model, and $g_i \in G$5 the discount factor. Each agent uses a local policy

$g_i \in G$6

with discounted return

$g_i \in G$7

This formalization captures the intended operating regime: local decision-making under uncertainty, with incomplete knowledge of the world state and of other agents’ internal information (Liu et al., 10 Jul 2025).

2. Observation design and overall workflow

PILOC consists of five connected components: local perception and map construction, a pheromone inverse guidance field, local communication and map fusion, observation embedding into a DRL policy, and MAPPO-based policy learning with decentralized execution (Liu et al., 10 Jul 2025). A typical execution cycle is: the agent observes its local environment; updates its obstacle map $g_i \in G$8, exploration map $g_i \in G$9, and local pheromone observation map $$A_g = \{ \leftarrow, \uparrow, \rightarrow, \downarrow \}.$$0; exchanges and merges local maps if neighboring agents are within communication range; feeds the three-map observation into a shared actor network; samples one of the four movement actions; receives exploration-, re-exploration-, collision-, and pheromone-related reward; and during training updates shared actor and critic parameters through MAPPO. During testing, a hybrid fallback planner is triggered if an agent appears stuck.

The observation space has three map-like inputs. The obstacle map $$A_g = \{ \leftarrow, \uparrow, \rightarrow, \downarrow \}.$$1 contains unsearched area, obstacle area, passable area, and historical position information consisting of the current position plus the previous two positions. The exploration map $$A_g = \{ \leftarrow, \uparrow, \rightarrow, \downarrow \}.$$2 contains unexplored area, explored area, and historical positions, but also introduces a time-marking mechanism for explored cells. These cells are assigned a value in $$A_g = \{ \leftarrow, \uparrow, \rightarrow, \downarrow \}.$$3, with smaller values denoting more recently observed areas. This design encodes not only whether a region has been seen, but how stale that information is, which is essential when targets move and can re-enter previously explored areas (Liu et al., 10 Jul 2025).

The third input is the pheromone observation map $$A_g = \{ \leftarrow, \uparrow, \rightarrow, \downarrow \}.$$4. Each agent centers a square observation window of size $$A_g = \{ \leftarrow, \uparrow, \rightarrow, \downarrow \}.$$5 on itself and reads the pheromone values within that local region. A key design decision is that pheromone is not used only as an external heuristic. It is embedded directly into the policy’s observation space and learned end-to-end by DRL. This suggests that PILOC’s coordination logic is split between explicit algorithmic structure and learned policy adaptation, rather than being encoded entirely as a rule-based swarm controller (Liu et al., 10 Jul 2025).

The policy/value model is an Actor-Critic network with four components: a convolutional module, a feature extraction module, an integration module, and an output layer. Each of the three maps is processed through convolution and pooling to extract spatial features; the resulting feature maps are reshaped, passed through a fully connected layer, transposed, and processed by another fully connected layer. Features from the three maps are fused by channel concatenation. The actor outputs a probability distribution over the four actions and samples stochastically; the critic outputs the state value estimate. The paper does not provide exact layer sizes, filter counts, or activation functions (Liu et al., 10 Jul 2025).

3. Pheromone inverse guidance

The most distinctive mechanism in PILOC is the pheromone inverse guidance field (Liu et al., 10 Jul 2025). In classical ant-colony optimization, high pheromone attracts agents to routes that have been used frequently. PILOC reverses this logic: high pheromone means a region has already been visited often, so agents should prefer lower-pheromone regions to reduce redundant exploration. Pheromone therefore functions as a distributed, decaying memory of traffic density over the map.

Let $$A_g = \{ \leftarrow, \uparrow, \rightarrow, \downarrow \}.$$6 denote the pheromone concentration at cell $$A_g = \{ \leftarrow, \uparrow, \rightarrow, \downarrow \}.$$7. When an agent occupies a location, the pheromone there is updated as

$$A_g = \{ \leftarrow, \uparrow, \rightarrow, \downarrow \}.$$8

To prevent unbounded accumulation,

$$A_g = \{ \leftarrow, \uparrow, \rightarrow, \downarrow \}.$$9

with

$U \triangleq \{1,2,\dots,U\},$0

At every time step pheromone evaporates according to

$U \triangleq \{1,2,\dots,U\},$1

with

$U \triangleq \{1,2,\dots,U\},$2

This update-decay rule makes pheromone a fading occupancy memory rather than a permanent trace. The representation captures both dwell time and visit frequency (Liu et al., 10 Jul 2025).

The inverse-guidance interpretation is operationalized in two places. First, the pheromone map is part of the agent observation. Second, the reward includes a pheromone-based term. The full reward is

$U \triangleq \{1,2,\dots,U\},$3

The exploration reward is

$U \triangleq \{1,2,\dots,U\},$4

where $U \triangleq \{1,2,\dots,U\},$5 is the number of newly discovered passable grid cells during a transition. The pheromone reward is written as

$U \triangleq \{1,2,\dots,U\},$6

The manuscript’s typesetting is imperfect, but the intended meaning is explicit: $U \triangleq \{1,2,\dots,U\},$7 is the pheromone concentration in the perception range at the previous time step, $U \triangleq \{1,2,\dots,U\},$8 is the concentration at the current time step, and the term rewards movement toward a region of lower current pheromone than before. The constants are

$U \triangleq \{1,2,\dots,U\},$9

This dual use of pheromone—as observation and as reward shaping—means the mechanism is both representational and behavioral (Liu et al., 10 Jul 2025).

Because pheromone evaporates, previously visited regions gradually lose repulsion and can become attractive again later. In a dynamic-target setting this matters: a hard “never revisit” rule would be inappropriate, because targets can move into areas that were observed earlier. A plausible implication is that PILOC’s inverse-pheromone design functions not merely as dispersion control but as a soft revisitation scheduler (Liu et al., 10 Jul 2025).

4. Local communication and decentralized coordination

The second structural pillar of PILOC is local communication (Liu et al., 10 Jul 2025). Agents communicate only when they are within communication range $v$0. Communication is therefore proximity-based and dynamic, rather than globally synchronized or continuously connected. When a set $v$1 of agents forms a communication group, they exchange their locally built obstacle and exploration maps. The paper does not describe direct exchange of policy parameters, actions, or recurrent hidden states during execution; communication is specifically map-level knowledge sharing.

Obstacle-map fusion is defined as

$v$2

Exploration-map fusion is defined analogously: $v$3 When the same explored cell has different time-mark values across agents, the smaller value is used, because smaller time marks indicate more recent observation. Thus the merged exploration map preserves the freshest available information (Liu et al., 10 Jul 2025).

This communication design reduces dependence on stable global connectivity and is intended to lower communication overhead, lower energy consumption, improve scalability, and increase robustness to disruption. Information can spread opportunistically as agents come into range. That is particularly germane to search-and-rescue scenarios with damaged infrastructure, cluttered urban terrain, or intermittent links.

A common misconception would be to read PILOC as a purely heuristic pheromone method. In fact, local communication is an equal architectural component, and the two mechanisms serve different roles. Pheromone provides implicit coordination even when agents are disconnected; local communication provides explicit synchronization when they meet (Liu et al., 10 Jul 2025). The ablation results reported later support the claim that these mechanisms are complementary rather than redundant.

5. Learning algorithm, hybrid control, and empirical results

PILOC uses MAPPO under centralized training and decentralized execution (CTDE) (Liu et al., 10 Jul 2025). The paper motivates this choice by noting that single-agent RL is inadequate in a multi-agent setting because each agent experiences a non-stationary environment induced by the behavior of others. The return and value quantities are given in standard PPO form: $v$4

$v$5

$v$6

$v$7

All homogeneous agents share the same policy network and value network during training and execution (Liu et al., 10 Jul 2025).

The reward also includes a re-exploration component intended for dynamic targets. Its printed formula suffers from typesetting corruption, but its intent is explicit: older observed cells with larger time-mark values yield more revisitation value, and a separate positive reward is given when an agent finds a target. Each collision incurs a penalty of $v$8. This reward structure makes dynamic-target adaptation a first-class part of the learning objective rather than a post hoc heuristic (Liu et al., 10 Jul 2025).

PILOC also includes a practical hybrid fallback at test time. If an agent revisits its position more than 3 times within the last 10 time steps, it is considered trapped in ineffective local behavior. Control then switches temporarily from the neural policy to a rule-based recovery routine: find the nearest unexplored cell; if all cells are explored, choose the nearest passable cell with the largest time mark; compute a path with $v$9; and follow that path to resume exploration. This is used only in testing (Liu et al., 10 Jul 2025).

Training and testing use the public grid-world dataset from Chen et al. (2019), with 5663 training maps, 5218 test maps, and $c$0 map size. Evaluation uses 2 agents, 6 dynamic targets, 250 test scenarios sampled randomly, and a maximum episode length of 250 steps; failure is declared if not all targets are found within that horizon. Training uses a curriculum over episode horizon, beginning with $c$1 and increasing by 10 when the maximum return does not improve for $c$2 consecutive episodes, until $c$3 (Liu et al., 10 Jul 2025).

The evaluation metrics are Success Rate (SR), Average Steps (AS), Step Variance (SV), and Average Number of Targets Obtained (ANTO). Against the reported baselines—MASAC, IPPO, QMIX, and Frontier-based exploration—PILOC achieves the strongest overall performance. The reported values are:

• PILOC: SR $c$4, AS $c$5, SV $c$6, ANTO $c$7
• IPPO: SR $c$8, AS $c$9, SV $$A = \{\leftarrow, \uparrow, \rightarrow, \downarrow\}.$$0, ANTO $$A = \{\leftarrow, \uparrow, \rightarrow, \downarrow\}.$$1
• MASAC: SR $$A = \{\leftarrow, \uparrow, \rightarrow, \downarrow\}.$$2, AS $$A = \{\leftarrow, \uparrow, \rightarrow, \downarrow\}.$$3, SV $$A = \{\leftarrow, \uparrow, \rightarrow, \downarrow\}.$$4, ANTO $$A = \{\leftarrow, \uparrow, \rightarrow, \downarrow\}.$$5
• QMIX: SR $$A = \{\leftarrow, \uparrow, \rightarrow, \downarrow\}.$$6 in the table, though the result-section text reportedly says $$A = \{\leftarrow, \uparrow, \rightarrow, \downarrow\}.$$7; AS $$A = \{\leftarrow, \uparrow, \rightarrow, \downarrow\}.$$8, SV $$A = \{\leftarrow, \uparrow, \rightarrow, \downarrow\}.$$9, ANTO $\langle N, S, A, O, \Omega, R, T, \gamma \rangle,$0
• Frontier: SR $\langle N, S, A, O, \Omega, R, T, \gamma \rangle,$1, AS $\langle N, S, A, O, \Omega, R, T, \gamma \rangle,$2, SV $\langle N, S, A, O, \Omega, R, T, \gamma \rangle,$3, ANTO $\langle N, S, A, O, \Omega, R, T, \gamma \rangle,$4

The ablation study isolates the effects of the two central modules. PILOC-com-ph removes both local communication and pheromone, PILOC-ph uses local communication only, PILOC-com uses pheromone only, and the full PILOC uses both. The reported results are:

• PILOC-com-ph: SR $\langle N, S, A, O, \Omega, R, T, \gamma \rangle,$5, AS $\langle N, S, A, O, \Omega, R, T, \gamma \rangle,$6
• PILOC-ph: SR $\langle N, S, A, O, \Omega, R, T, \gamma \rangle,$7, AS $\langle N, S, A, O, \Omega, R, T, \gamma \rangle,$8
• PILOC-com: SR $\langle N, S, A, O, \Omega, R, T, \gamma \rangle,$9, AS $N$0
• PILOC: best overall

These numbers show that both modules improve performance, with pheromone alone yielding a larger gain than communication alone in the reported setup, and the combined system performing best (Liu et al., 10 Jul 2025). The paper also studies scalability from 2 to 5 agents. Success rate rises as the number of agents increases, average steps decrease substantially, success reaches 100% at 4 agents and remains 100% at 5 agents, and average steps decrease from 123.51 to 59.01 (Liu et al., 10 Jul 2025).

6. Interpretation, limitations, and nomenclature

PILOC’s reported strengths are closely tied to the specific challenge model it targets. It is designed for unknown environments, moving targets, local sensing, limited communication, and decentralized execution. Its coordination strategy is explicitly hybrid: inverse pheromone guidance reduces redundant overlap even when agents are disconnected, while local communication opportunistically merges partial knowledge when contact occurs (Liu et al., 10 Jul 2025). This suggests that the framework is intended less as a generic MARL benchmark and more as a structured MASAR architecture.

The limitations are also explicit. Agents and targets move only in four directions on a grid; communication is modeled as proximity-triggered full map union rather than packetized networking with delay or asymmetric loss; the network architecture is described qualitatively rather than fully specified; some reward equations suffer from typesetting corruption; the test-time $N$1-based rescue mechanism indicates that learned behavior can still get stuck in local loops; and the framework assumes homogeneous agents, with heterogeneous teams identified as future work (Liu et al., 10 Jul 2025). A plausible implication is that PILOC’s present contribution lies more in coordination design under constrained observability than in high-fidelity physical realism.

The term PILOC is also easy to confuse with several unrelated or only name-adjacent systems on arXiv. In the relevant literature, PIDLoc is a cross-view RGB+LiDAR-to-satellite 3-DoF pose optimization network for autonomous driving and is not PILOC (Lee et al., 4 Mar 2025). POLOCALC is a POLarization Orientation CALibrator for Cosmology for absolute CMB polarization-angle calibration and is likewise unrelated (Nati et al., 2017). The phrase “PILOC” also does not denote the PILOT polarization-leakage correction pipeline for balloon-borne dust-polarization data; that work concerns instrumental corrections in the PILOT experiment rather than multi-agent search (Bernard et al., 2022). Nor is PILOC the UAV geo-localization line PiLoT or PiLoT v2, which address free-view UAV pose estimation against 3D meshes or TDOM/DSM orthographic maps (Liu et al., 30 Jun 2026).

Within its own domain, however, PILOC has a clear and specific meaning: a CTDE MARL framework for cooperative search of moving targets in unknown maps, built around inverse pheromone guidance and local communication to reduce redundant exploration and maintain decentralized coordination under partial observability and communication constraints (Liu et al., 10 Jul 2025).