---
title: 'PILOC: Deep MARL Framework for Search & Rescue'
url: https://www.emergentmind.com/topics/piloc
type: topic
---

# PILOC: Deep MARL Framework for Search & Rescue

Searching arXiv for “PILOC” and closely related entries to ground the article in current papers.
Searching arXiv for exact paper 2507.07376 and nearby similarly named systems to disambiguate terminology.
PILOC is a deep multi-agent reinforcement learning framework for cooperative search and rescue in dynamic, partially observable, communication-constrained environments. The name denotes a design built around **Pheromone Inverse guidance** and **LOcal Communication**, and the framework is proposed for dynamic target search in completely unknown environments, where the map is not known in advance, targets move over time, each agent has only local sensing, communication is limited, and coordination must remain decentralized at execution time [2507.07376]. Its central premise is to combine two coordination channels: implicit coordination through a decaying virtual pheromone field that repels agents from heavily visited regions, and explicit coordination through proximity-triggered local map exchange. In the reported formulation, PILOC is implemented in a centralized-training, decentralized-execution regime using MAPPO, with the pheromone mechanism embedded directly into the observation space of the learned policy rather than treated as an external planner [2507.07376].

## 1. Problem setting and formalization

PILOC addresses Multi-Agent Search and Rescue (MASAR) in a **2D discretized grid map** whose cells are either passable or impassable, with the passable region assumed to be fully connected [2507.07376]. The setting is explicitly characterized by a conjunction of difficulties: unknown map structure, dynamic targets, partial observability, limited communication, and a decentralized coordination requirement. This makes the framework distinct from methods that assume static targets, globally known maps, persistent global communication, or centralized planning.

Targets are modeled as multiple moving entities \(G\), with each target \(g_i \in G\) initialized randomly in free space. At each time step, a target chooses uniformly from the 4-neighbor motion set
\[
A_g = \{ \leftarrow, \uparrow, \rightarrow, \downarrow \}.
\]
If a sampled move would collide with an obstacle, the action is re-sampled until a feasible move is found. Agents are homogeneous and indexed by
\[
U \triangleq \{1,2,\dots,U\},
\]
with each agent assigned a perception radius \(v\), a communication radius \(c\), and a discrete action space
\[
A = \{\leftarrow, \uparrow, \rightarrow, \downarrow\}.
\]
Without communication, an agent does not know the global map, target locations, or the explored areas of other agents. The mission objective is to find all targets within a finite time limit [2507.07376].

The problem is formulated as a **Decentralized Partially Observable Markov Decision Process (Dec-POMDP)**,
\[
\langle N, S, A, O, \Omega, R, T, \gamma \rangle,
\]
where \(N\) is the number of agents, \(S\) the state space, \(A=[A_1,\dots,A_N]\) the joint action space, \(O=[O_1,\dots,O_N]\) the observation spaces, \(\Omega(o,s',a)\sim P(o\mid s',a)\) the observation model, \(R(s)\) the reward function, \(T(s,a,s')\sim P(s'\mid s,a)\) the transition model, and \(\gamma\in[0,1]\) the discount factor. Each agent uses a local policy
\[
\pi(a_i \mid o_i): O_i \times A_i \to [0,1],
\]
with discounted return
\[
G = \sum_{k=0}^{T} \gamma^k r_k.
\]
This formalization captures the intended operating regime: local decision-making under uncertainty, with incomplete knowledge of the world state and of other agents’ internal information [2507.07376].

## 2. Observation design and overall workflow

PILOC consists of five connected components: local perception and map construction, a pheromone inverse guidance field, local communication and map fusion, observation embedding into a DRL policy, and MAPPO-based policy learning with decentralized execution [2507.07376]. A typical execution cycle is: the agent observes its local environment; updates its obstacle map \(M_o\), exploration map \(M_e\), and local pheromone observation map \(O_{\text{ph}}\); exchanges and merges local maps if neighboring agents are within communication range; feeds the three-map observation into a shared actor network; samples one of the four movement actions; receives exploration-, re-exploration-, collision-, and pheromone-related reward; and during training updates shared actor and critic parameters through MAPPO. During testing, a hybrid fallback planner is triggered if an agent appears stuck.

The observation space has three map-like inputs. The **obstacle map** \(M_o\) contains unsearched area, obstacle area, passable area, and historical position information consisting of the current position plus the previous two positions. The **exploration map** \(M_e\) contains unexplored area, explored area, and historical positions, but also introduces a time-marking mechanism for explored cells. These cells are assigned a value in \([0,0.3]\), with smaller values denoting more recently observed areas. This design encodes not only whether a region has been seen, but how stale that information is, which is essential when targets move and can re-enter previously explored areas [2507.07376].

The third input is the **pheromone observation map** \(O_{\text{ph}}\). Each agent centers a square observation window of size \(l \times l\) on itself and reads the pheromone values within that local region. A key design decision is that pheromone is not used only as an external heuristic. It is embedded directly into the policy’s observation space and learned end-to-end by DRL. This suggests that PILOC’s coordination logic is split between explicit algorithmic structure and learned policy adaptation, rather than being encoded entirely as a rule-based swarm controller [2507.07376].

The policy/value model is an Actor-Critic network with four components: a convolutional module, a feature extraction module, an integration module, and an output layer. Each of the three maps is processed through convolution and pooling to extract spatial features; the resulting feature maps are reshaped, passed through a fully connected layer, transposed, and processed by another fully connected layer. Features from the three maps are fused by channel concatenation. The actor outputs a probability distribution over the four actions and samples stochastically; the critic outputs the state value estimate. The paper does not provide exact layer sizes, filter counts, or activation functions [2507.07376].

## 3. Pheromone inverse guidance

The most distinctive mechanism in PILOC is the **pheromone inverse guidance** field [2507.07376]. In classical ant-colony optimization, high pheromone attracts agents to routes that have been used frequently. PILOC reverses this logic: high pheromone means a region has already been visited often, so agents should prefer **lower-pheromone** regions to reduce redundant exploration. Pheromone therefore functions as a distributed, decaying memory of traffic density over the map.

Let \(P(x,y)\) denote the pheromone concentration at cell \((x,y)\). When an agent occupies a location, the pheromone there is updated as
\[
P(x,y) =
\begin{cases}
P(x,y) + 1, & \text{Agent exists at this location} \\
P(x,y), & \text{Agent does not exist at this location}.
\end{cases}
\]
To prevent unbounded accumulation,
\[
P(x,y) \le P_{\max},
\]
with
\[
P_{\max} = 10.
\]
At every time step pheromone evaporates according to
\[
P(x,y) = P(x,y)(1-\lambda),
\]
with
\[
\lambda = 0.02.
\]
This update-decay rule makes pheromone a fading occupancy memory rather than a permanent trace. The representation captures both dwell time and visit frequency [2507.07376].

The inverse-guidance interpretation is operationalized in two places. First, the pheromone map is part of the agent observation. Second, the reward includes a pheromone-based term. The full reward is
\[
r = r_e + r_{re} - r_{co} + r_{ph}. \tag{6}
\]
The exploration reward is
\[
r_e = 0.5 \times n_e, \tag{7}
\]
where \(n_e\) is the number of newly discovered passable grid cells during a transition. The pheromone reward is written as
\[
r_{ph} = \alpha \frac{I_{\text{ph}'} - I_{\text{ph}}}{I_{\text{ph}'} + \beta (I_{\text{ph}'} - I_{\text{ph}})}. \tag{9}
\]
The manuscript’s typesetting is imperfect, but the intended meaning is explicit: \(I_{\text{ph}'}\) is the pheromone concentration in the perception range at the previous time step, \(I_{\text{ph}}\) is the concentration at the current time step, and the term rewards movement toward a region of lower current pheromone than before. The constants are
\[
\alpha = 0.1,\qquad \beta = 0.1.
\]
This dual use of pheromone—as observation and as reward shaping—means the mechanism is both representational and behavioral [2507.07376].

Because pheromone evaporates, previously visited regions gradually lose repulsion and can become attractive again later. In a dynamic-target setting this matters: a hard “never revisit” rule would be inappropriate, because targets can move into areas that were observed earlier. A plausible implication is that PILOC’s inverse-pheromone design functions not merely as dispersion control but as a soft revisitation scheduler [2507.07376].

## 4. Local communication and decentralized coordination

The second structural pillar of PILOC is **local communication** [2507.07376]. Agents communicate only when they are within communication range \(c\). Communication is therefore proximity-based and dynamic, rather than globally synchronized or continuously connected. When a set \(C\) of agents forms a communication group, they exchange their locally built obstacle and exploration maps. The paper does not describe direct exchange of policy parameters, actions, or recurrent hidden states during execution; communication is specifically **map-level knowledge sharing**.

Obstacle-map fusion is defined as
\[
M_o^{(i')} = \bigcup_{j \in C} M_o^j \qquad \forall i \in C. \tag{4}
\]
Exploration-map fusion is defined analogously:
\[
M_e^{(i')} = \bigcup_{j \in C} M_e^j \qquad \forall i \in C. \tag{5}
\]
When the same explored cell has different time-mark values across agents, the smaller value is used, because smaller time marks indicate more recent observation. Thus the merged exploration map preserves the freshest available information [2507.07376].

This communication design reduces dependence on stable global connectivity and is intended to lower communication overhead, lower energy consumption, improve scalability, and increase robustness to disruption. Information can spread opportunistically as agents come into range. That is particularly germane to search-and-rescue scenarios with damaged infrastructure, cluttered urban terrain, or intermittent links.

A common misconception would be to read PILOC as a purely heuristic pheromone method. In fact, local communication is an equal architectural component, and the two mechanisms serve different roles. Pheromone provides **implicit coordination** even when agents are disconnected; local communication provides **explicit synchronization** when they meet [2507.07376]. The ablation results reported later support the claim that these mechanisms are complementary rather than redundant.

## 5. Learning algorithm, hybrid control, and empirical results

PILOC uses **MAPPO** under **centralized training and decentralized execution (CTDE)** [2507.07376]. The paper motivates this choice by noting that single-agent RL is inadequate in a multi-agent setting because each agent experiences a non-stationary environment induced by the behavior of others. The return and value quantities are given in standard PPO form:
\[
R_t = r_{t+1} + \gamma r_{t+2} + \gamma^2 r_{t+3} + \cdots = \sum_{k=0}^{\infty} \gamma^k r_{t+k+1}, \tag{10}
\]
\[
V^{\pi}(s_t) = \mathbb{E}_{\pi}[R_t \mid s_t], \tag{11}
\]
\[
Q^{\pi}(s_t,a_t) = \mathbb{E}_{\pi}[R_t \mid s_t,a_t], \tag{12}
\]
\[
A^{\pi}(s_t,a_t)=Q^{\pi}(s_t,a_t)-V^{\pi}(s_t). \tag{13}
\]
All homogeneous agents share the same policy network and value network during training and execution [2507.07376].

The reward also includes a **re-exploration** component intended for dynamic targets. Its printed formula suffers from typesetting corruption, but its intent is explicit: older observed cells with larger time-mark values yield more revisitation value, and a separate positive reward is given when an agent finds a target. Each collision incurs a penalty of \(-3\). This reward structure makes dynamic-target adaptation a first-class part of the learning objective rather than a post hoc heuristic [2507.07376].

PILOC also includes a practical **hybrid fallback** at test time. If an agent revisits its position more than 3 times within the last 10 time steps, it is considered trapped in ineffective local behavior. Control then switches temporarily from the neural policy to a rule-based recovery routine: find the nearest unexplored cell; if all cells are explored, choose the nearest passable cell with the largest time mark; compute a path with \(A^*\); and follow that path to resume exploration. This is used only in testing [2507.07376].

Training and testing use the public grid-world dataset from Chen et al. (2019), with **5663 training maps**, **5218 test maps**, and **\(60 \times 60\)** map size. Evaluation uses **2 agents**, **6 dynamic targets**, **250 test scenarios sampled randomly**, and a maximum episode length of **250 steps**; failure is declared if not all targets are found within that horizon. Training uses a curriculum over episode horizon, beginning with \(N_s = 10\) and increasing by 10 when the maximum return does not improve for \(M\) consecutive episodes, until \(N_s = 260\) [2507.07376].

The evaluation metrics are **Success Rate (SR)**, **Average Steps (AS)**, **Step Variance (SV)**, and **Average Number of Targets Obtained (ANTO)**. Against the reported baselines—**MASAC**, **IPPO**, **QMIX**, and **Frontier-based exploration**—PILOC achieves the strongest overall performance. The reported values are:
- **PILOC**: SR \(=95.6\%\), AS \(=129.08\), SV \(=2897.15\), ANTO \(=5.716\)
- **IPPO**: SR \(=87.2\%\), AS \(=158.00\), SV \(=3190.97\), ANTO \(=5.148\)
- **MASAC**: SR \(=74.0\%\), AS \(=174.80\), SV \(=2995.33\), ANTO \(=4.344\)
- **QMIX**: SR \(=62.8\%\) in the table, though the result-section text reportedly says \(58.4\%\); AS \(=189.64\), SV \(=3232.90\), ANTO \(=5.40\)
- **Frontier**: SR \(=18.4\%\), AS \(=225.14\), SV \(=4948.43\), ANTO \(=3.82\)

The ablation study isolates the effects of the two central modules. **PILOC-com-ph** removes both local communication and pheromone, **PILOC-ph** uses local communication only, **PILOC-com** uses pheromone only, and the full **PILOC** uses both. The reported results are:
- **PILOC-com-ph**: SR \(=81.6\%\), AS \(=170\)
- **PILOC-ph**: SR \(=88.8\%\), AS \(=146.55\)
- **PILOC-com**: SR \(=91.2\%\), AS \(=139.28\)
- **PILOC**: best overall

These numbers show that both modules improve performance, with pheromone alone yielding a larger gain than communication alone in the reported setup, and the combined system performing best [2507.07376]. The paper also studies scalability from 2 to 5 agents. Success rate rises as the number of agents increases, average steps decrease substantially, success reaches **100%** at **4 agents** and remains **100%** at **5 agents**, and average steps decrease from **123.51** to **59.01** [2507.07376].

## 6. Interpretation, limitations, and nomenclature

PILOC’s reported strengths are closely tied to the specific challenge model it targets. It is designed for unknown environments, moving targets, local sensing, limited communication, and decentralized execution. Its coordination strategy is explicitly hybrid: inverse pheromone guidance reduces redundant overlap even when agents are disconnected, while local communication opportunistically merges partial knowledge when contact occurs [2507.07376]. This suggests that the framework is intended less as a generic MARL benchmark and more as a structured MASAR architecture.

The limitations are also explicit. Agents and targets move only in four directions on a grid; communication is modeled as proximity-triggered full map union rather than packetized networking with delay or asymmetric loss; the network architecture is described qualitatively rather than fully specified; some reward equations suffer from typesetting corruption; the test-time \(A^*\)-based rescue mechanism indicates that learned behavior can still get stuck in local loops; and the framework assumes homogeneous agents, with heterogeneous teams identified as future work [2507.07376]. A plausible implication is that PILOC’s present contribution lies more in coordination design under constrained observability than in high-fidelity physical realism.

The term **PILOC** is also easy to confuse with several unrelated or only name-adjacent systems on arXiv. In the relevant literature, **PIDLoc** is a cross-view RGB+LiDAR-to-satellite **3-DoF pose optimization network** for autonomous driving and is not PILOC [2503.02388]. **POLOCALC** is a **POLarization Orientation CALibrator for Cosmology** for absolute CMB polarization-angle calibration and is likewise unrelated [1704.02704]. The phrase “PILOC” also does not denote the **PILOT** polarization-leakage correction pipeline for balloon-borne dust-polarization data; that work concerns instrumental corrections in the PILOT experiment rather than multi-agent search [2205.03668]. Nor is PILOC the UAV geo-localization line **PiLoT** or **PiLoT v2**, which address free-view UAV pose estimation against 3D meshes or TDOM/DSM orthographic maps [2606.31098].

Within its own domain, however, PILOC has a clear and specific meaning: a CTDE MARL framework for cooperative search of moving targets in unknown maps, built around inverse pheromone guidance and local communication to reduce redundant exploration and maintain decentralized coordination under partial observability and communication constraints [2507.07376].

Source: https://www.emergentmind.com/topics/piloc