Multi-Agent Biased Random Walk

• Multi-agent biased random walk is a stochastic model where agents navigate networks using biased transition probabilities, integrating inherent network characteristics.
• The methodology leverages Markov chain theory and dynamic moving neighborhood graphs to analyze consensus, convergence, and stability in distributed systems.
• Applications span network exploration, consensus algorithms, and reinforcement learning, while addressing computational challenges and optimizing exploration efficiency.

A multi-agent biased random walk is a stochastic process in which multiple agents independently traverse a network or graph, utilizing transition probabilities that systematically favor certain movements or destinations (the “bias”), while their interactions—whether direct or indirect—are mediated by their trajectory coincidences or local environment. This construct generalizes classical random walk models by incorporating agent mobility, spatially and temporally variable interactions, and asymmetric, agent- or context-dependent transition mechanisms. Rigorous formulations often place such agents on weighted directed networks, lattices, random graphs, or hierarchical structures, with broad applications spanning consensus algorithms, exploration and coverage, learning dynamics, network sampling, and collective decision protocols.

1. Formal Models: Biased Random Walks by Multiple Agents

Multi-agent biased random walks typically operate on a finite, strongly connected, weighted directed graph $G = (V, E, W)$, with $|V| = m$. Each of $n$ agents evolves as a (possibly biased) Markov chain on $G$. The one-step transition probability for agent $k$ at node $i$ to move to node $j$ is defined as: $$q_{ij} = \frac{w_{ij}}{d_i}, \qquad d_i = \sum_{j=1}^m w_{ij}$$ where $w_{ij} > 0$ if $(i, j) \in E$, and the weight $w_{ij}$ encodes the potential bias (e.g., favoring movement to certain nodes).

Each agent’s walk is a homogeneous, ergodic Markov chain provided the graph is strongly connected and aperiodic (0909.3475), ensuring convergence to a stationary distribution $T$. When additional structure is present (e.g., trees, lattices, or networks with community/hierarchical organization), the direction and character of the bias may be linked to node degree, distance to targets, neighborhood properties, core-periphery structure, or even dynamically updated features such as local exploration statistics (see core- or degree-biased mechanisms (Mondragon, 2017, Jr. et al., 2023, Calva et al., 2021)).

Biases can be static, determined by fixed network attributes, or dynamic, modulated by controller strategies, agent states, or reinforcement/memory of visitation history (Haslegrave et al., 2020, Agliari et al., 2012, Jr. et al., 2023). In networked multi-agent reinforcement learning, the notion of “biased action information” generalizes this further, biasing not just motion but value estimation based on cooperative or competitive group membership (Ryu et al., 2021).

2. Interaction Mechanisms and Moving Neighborhood Graphs

Communication or interaction among agents is often predicated on coincident occupation of a node: if $Y_i(t) = Y_j(t)$ (agents $i$ and $j$ at node $v$ at time $t$), then a directed (potentially probabilistic) information link from $i$ to $j$ is formed with probability $p_{ij}$ (linkage probability) (0909.3475). The instantaneous communication graph $G(t)$ built this way is called a moving neighborhood random graph.

• The resulting graphical structure is highly dynamic: links exist only when agents physically coincide, and these links are generally unidirectional and may or may not be “balanced” (same total in- and out-degree per agent).
• Under further balance assumptions (e.g., out-degree equals in-degree per node), stochastic consensus protocols admit mean-preserving state evolution and facilitate rigorous stability analysis.

The general discrete-time consensus protocol for such a system is: $$X_i(t+1) = X_i(t) + \epsilon \sum_{j\in N_i(t)} b_{ij}(X_j(t) - X_i(t))$$ with $\epsilon \in (0, 1/4)$ for stability, and $b_{ij}$ quantifying the influence weight from agent $j$ to agent $i$.

This protocol can be expressed compactly via the Laplacian matrix $L(t)$ of $G(t)$: $X(t+1) = (I_n - \epsilon L(t)) X(t)$ which preserves the average state and ensures convergence to consensus under suitable connectivity and stochasticity conditions.

3. Consensus, Convergence, and Stability Analysis

Asymptotic consensus in multi-agent biased random walk systems requires:

• Strong connectivity and aperiodicity in the underlying graph $G$ (ensures ergodicity and unique stationary distribution for each agent’s walk).
• Temporal “richness” of the moving neighborhood union—although $G(t)$ may be disconnected at any instant, the sequence should be such that the union over time is sufficiently connected to propagate information globally.
• Balance conditions for the moving neighborhood graph (the sum of weights of incoming equals outcoming links for each agent), which control the symmetry needed for stability (0909.3475).

Stochastic Lyapunov functions are introduced to quantify the level of disagreement: $$\varphi(t) = X(t)^\top X(t) - n a^2,\quad \text{with } a = \frac{1}{n} \mathbf{1}^\top X(0)$$ where one proves $\varphi(t) \to 0$ almost surely as $t \to \infty$. By splitting $X(t)$ into consensus and disagreement components, the analysis reduces to establishing that the disagreement is driven to zero by the contraction properties of the average Laplacian, leveraging stochastic stability arguments.

In certain settings (e.g., random walks on multi-type Galton–Watson trees (Dembo et al., 2010)) or with memory/reinforcement (Agliari et al., 2012), convergence properties may also exhibit phase transitions in dependence on the bias, with critical regimes (e.g., recurrence vs. transience, ballistic vs. sub-ballistic motion) determined by structural eigenvalues or control parameters.

4. Variations: Structure of Bias and Protocols

The rich class of multi-agent biased random walks encompasses several variations:

• Degree-biased and core-biased walks: The transition probability of agent $k$ from node $i$ to $j$ may be proportional to node degrees ($k_j^\alpha$ for some exponent $\alpha$) or connectivity with a “core set,” approximating maximal entropy random walks (MERW) without requiring global structural knowledge (Mondragon, 2017, Calva et al., 2021).
• Avoiding traps or hubs: In heterogeneous networks, biasing movements away from hubs (e.g., $\alpha < -1$ in the degree-bias exponent) can mitigate trapping and ensure mobile agents or packets are not immobilized by high-degree node congestion (Bastas et al., 2013).
• Soft priorities and action information: Introducing small probabilities for low-priority agents to act, or incorporating friend-or-foe action biases in MARL, balances efficiency and fairness in competitive environments (Bastas et al., 2013, Ryu et al., 2021).
• Self-avoiding and memory-based bias: Probabilities can decay with visitation frequency (true self-avoiding walks), enforcing spatial dispersion and expanding explored territory (Jr. et al., 2023). Dynamic memory effects (“local reinforcement”) can lead to ballistic motion for arbitrarily small bias, with field effects always overcoming memory in certain regimes (Agliari et al., 2012).
• Time-dependent or controlled biases: Centralized or distributed “controllers” adjust the random walk protocol at each step, possibly using history-dependent strategies to optimize cover times, which can yield near-optimal exploration rates but at the expense of high computational complexity (often PSPACE-complete to optimize) (Haslegrave et al., 2020).
• Local bias configurations: Customizing a vector of local biases at each node can minimize global mean first passage times to targets; this local tailoring can be found via combinatorial optimization or heuristics such as simulated annealing (Calva et al., 2021).

5. Applications: Exploration, Sampling, and Distributed Algorithms

Multi-agent biased random walk models directly inform and underpin algorithms for:

• Consensus and distributed averaging: Mobile robotic or sensor swarms, where agents effectively exchange information only upon physical proximity, benefit from such time-varying and spatially triggered network dynamics (0909.3475).
• Network exploration and coverage: Coverage times and exploration efficiency on random graphs can be dramatically improved by biasing walks toward low-degree or underexplored regions, with the cover time reduced to $O(n \log n)$ on $G_{n,p}$, and further reductions possible under controller-based (E-TBRW) strategies (Cooper et al., 2017, Haslegrave et al., 2020).
• Network embedding and sampling: Biased random walk sampling processes power context-based node embedding methods (e.g., node2vec, BiasedWalk), enabling both local (BFS-like) and global (DFS-like) graph structure preservation in the resulting embeddings and robust link prediction accuracy (Nguyen et al., 2018, Jr. et al., 2023). Surprisingly, structural properties are often well recapitulated regardless of specific walk bias, reflecting the resilience of embedding methods to the walk dynamics.
• Epidemic modeling and information spreading: Directional bias (e.g., modeling commuting behavior) and mobility restrictions play key roles in the spatiotemporal spread of diseases and information, with reduced mobility among infected agents suppressing epidemic size but prolonging duration (Ichinose et al., 2018).
• Optimization in design and control: Local and targeted biasing strategies (nodewise parameterization, core estimation, adaptive policies) open avenues for optimal transport, exploration, and task assignment in urban networks and beyond (Calva et al., 2021, Smith et al., 2014).

6. Theoretical Insights and Key Results

Several fundamental insights have been established:

• Ergodicity, balance, and strong connectivity are essential for consensus in mobile multi-agent systems on time-varying graphs (0909.3475).
• Phase transitions (recurrence/transience, ballistic/sub-ballistic) can emerge as bias strength is tuned, with critical points dictated by global network characteristics (e.g., eigenvalues of branching matrices, or ratios of conductance and exponential decay rates) (Dembo et al., 2010, Croydon et al., 2019).
• Rigorous bounds on cover times and convergence rates can be given in terms of graph parameters, with controller-enabled strategies achieving near-optimal performance at the cost of increased algorithmic complexity (Haslegrave et al., 2020).
• Dynamic and group-aware biases (friend-or-foe, group priorities, learning without recall) enhance learning and coordination in multispecies or game-theoretic systems, and biases can be designed to decay as system coordination improves (Rahimian et al., 2015, Ryu et al., 2021).
• Robustness to walk structure in learning from paths: In embedding and structural recovery tasks, various walk biases (degree-based, self-avoiding, node2vec parametric variations) yield only mild differences, implying walk-based sampling strategies are broadly interchangeable for many downstream tasks (Jr. et al., 2023).

7. Challenges, Limitations, and Future Directions

• Computational constraints: Locally optimal or controller-based strategies for cover and exploration (especially when time- or history-dependent) can be computationally hard (e.g., PSPACE-complete for directed graphs), suggesting practical implementations must leverage decentralized, heuristic, or structure-aware rules (Haslegrave et al., 2020).
• Analysis of non-stationary and adaptive bias: Understanding the long-term effects of adaptive or learning-based biases, particularly in competitive and adversarial multi-agent scenarios, remains a frontier topic (Ryu et al., 2021).
• Impact of spatial disorder, boundaries, and resetting: Recent formalism (Giuggioli et al., 2023) demonstrates how lattice geometry, heterogeneous disorder, and boundary/radiation conditions can dramatically impact first-passage times, encounter statistics, and overall transmission efficiency in multi-agent systems.
• Integration with reinforcement and MARL frameworks: Biases in action information and learning targets are emerging as powerful inductive tools for faster coordination, reward maximization, and stability in multi-agent reinforcement learning, particularly in mixed cooperative-competitive tasks.

---

In synopsis, the multi-agent biased random walk framework is a mathematically robust, versatile paradigm that encapsulates the complex interplay among agent mobility, stochastic communication, network topology, and local/dynamic biasing mechanisms. It bridges deep probabilistic theory with scalable, decentralized algorithmic solutions for distributed coordination, search, learning, and networked control in heterogeneous, dynamic environments.