Networked Restless Multi-Armed Bandits

• Networked RMABs are a framework where each arm's state evolution depends on its own action and those of its neighbors, capturing positive externalities and cascading effects.
• They leverage graph-aware index policies, submodular action selection, and networked reinforcement learning to achieve tractable, near-optimal decision-making in high-dimensional settings.
• Applications include resource allocation, epidemic control, social information gathering, and mobile interventions, addressing both intrinsic dynamics and network-induced influences.

Networked Restless Multi-Armed Bandits (Networked RMABs) generalize the classical RMAB framework by embedding arms within a network structure, such that an arm's state evolution and reward may depend not only on its own action but also on the actions selected for neighboring arms. These models capture positive externalities, propagative or cascading effects, and action-induced state transitions frequently observed in resource allocation, epidemiology, social information gathering, and mobile intervention domains. Substantial algorithmic innovations—including graph-aware index policies, efficient greedy action selection, and network-coupled reinforcement learning—enable tractable, near-optimal decision-making in these complex and high-dimensional control settings.

1. Formal Model Definition

Networked RMABs are defined over a collection of $N$ arms situated on the vertex set $V$ of a (typically directed or undirected) graph $G = (V, E)$ that encodes potential externality or communication pathways between arms. For each arm $i \in V$:

• The local state space $S_i$ is typically finite (binary or multi-level).
• The action space $A_i = \{0, 1\}$, where $a_i(t) = 1$ denotes resource allocation (pull) at time $t$.
• The reward function $r_i(s_i(t), a_i(t))$ measures instantaneous utility, commonly $r_i(s_i, a_i) = s_i$, reflecting success-state occupancy.

Transitions in Networked RMABs can occur via multiple mechanisms:

• Intrinsic transitions: When arm $V$0 is pulled, it evolves according to its own Markov kernel $V$1; if not pulled, via $V$2.
• Network-induced transitions/externalities: If a neighbor $V$3 is pulled, arm $V$4 may accrue augmented transition probabilities, formalized as

$V$5

with $V$6 the externality strength and $V$7 normalized contributions (Herlihy et al., 2022).

A global budget constraint restricts the sum of pulls per time step: $V$8. The control objective is to maximize the expected total discounted reward

$V$9

with $G = (V, E)$0 (Herlihy et al., 2022).

Extensions feature more general arms (continuous/finite state), availability constraints, local population effects, or communication patterns for parameter sharing (Mehta et al., 2018, Zhao et al., 2024).

2. Algorithmic Approaches: Whittle Index and Beyond

Whittle Index Formulation

For classical RMABs, the Whittle index policy decouples the arms by relaxing the hard budget constraint via a Lagrange multiplier $G = (V, E)$1 (subsidy for inaction), solving each single-arm MDP: $G = (V, E)$2 with the Bellman equation

$G = (V, E)$3

(Mehta et al., 2018, Herlihy et al., 2022).

Indexability is fundamental: the set of states where $G = (V, E)$4 is optimal must monotonically expand as $G = (V, E)$5 increases. The Whittle index $G = (V, E)$6 is the unique $G = (V, E)$7 where actions are equally valuable. In networked settings, the Whittle index must be modified to reflect not only the arm's own action but also the propagated benefits/costs due to actions on its neighbors (Herlihy et al., 2022, Ou et al., 2022).

Graph-Aware Index Policies

The “Greta” algorithm constitutes a tractable network-aware Whittle-style heuristic (Herlihy et al., 2022):

• At each time, compute pull-only and pull+message (externality) allocations in budget chunks, selecting assignments via index sums.
• Edge-level indices $G = (V, E)$8 are maintained; allocations compare pull-costs to message-benefit combinations.
• Greta’s reward is provably bounded between the classic Threshold Whittle policy (graph-agnostic) and the optimal baseline for zero message cost (Herlihy et al., 2022).

Concavity and indexability properties enable efficient search for optimal periodic policies in mobile-intervention domains, with auxiliary subroutines for synchronized scheduling (Ou et al., 2022).

Submodular Action Selection and Greedy Guarantees

In frameworks combining RMAB with network-propagation models (e.g., independent cascade), the state/action space grows exponentially. However, the per-step $G = (V, E)$9 set function is submodular in targeted nodes, enabling greedy or hill-climbing action selection with $i \in V$0 optimality guarantees under the Nemhauser bound. Convergence of such approximate Bellman updates follows via meta-MDP contraction analysis (Zhang et al., 6 Dec 2025).

Networked RL and Communication-Aware Learning

In settings with partial observability, systematic data errors, or unknown dynamics, deep RL approaches—incorporating network effects—are critical:

• Communication learning selects which arms should exchange Q-network parameters, optimizing joint utility of parameter sharing. Sparse-neighbor schemes and communication-MDP decompositions support scalable and decentralized coordination, provably improving learning efficiency under sufficient coverage/mixing (Zhao et al., 2024).
• Hill-climbing DQN architectures with GNN featurization address intractable state/action spaces and explicitly process network-induced outcome cascades (Zhang et al., 6 Dec 2025).

3. Computational Challenges and Structural Properties

Combinatorial state/action spaces arising from network externalities, action-induced dependencies, and large-scale population models present significant computational bottlenecks:

• For budgeted activation with $i \in V$1 arms, the action space is $i \in V$2.
• Value-function storage and dynamic programming are infeasible; tractable policies rely on decomposability (Whittle), submodularity (greedy maximization), or mean-field/surrogate relaxations (Herlihy et al., 2022, Zhang et al., 6 Dec 2025, Ou et al., 2022).

Concavity and monotonicity properties are leveraged for period-finding relaxations and to guarantee indexability (ensuring indices exist and can be computed via binary search) (Ou et al., 2022).

Propositions clarify when network coupling is beneficial or even detrimental; selective communication and low-bandwidth protocols (one-neighbor schemes) attain near-dense communication sample complexity under standard assumptions (Zhao et al., 2024).

4. Empirical Validation and Performance

Representative empirical evaluations across various domains confirm the practical significance of exploiting network effects in RMABs:

Algorithm	Typical IB or Reward Relative to Optimum	Key Scenario
Greta (graph-aware)	100% IB (by definition)	N=100 arms, cluster/random mappings
Myopic	76-88% IB	Underperforms Greta except for low $i \in V$3
Threshold Whittle	73-84% IB	Baseline RMAB (no network), N=100
Random	64-75% IB	Non-adaptive allocation

• Greta’s advantage increases with the budget, diminishes as message cost $i \in V$4 rises, and remains positive across homogeneous (assortative) or heterogeneous (cross-group) graphs (Herlihy et al., 2022).
• ENGAge algorithm shows 10–35% per-round rewards above network-agnostic methods, is robust to moderate network perturbations, and does not underserve high-need populations (Ou et al., 2022).
• Hill-climbing DQN with GNN features achieves ≈82% active-state fraction at $i \in V$5, outperforming myopic and network-blind policies on real village contact graphs (Zhang et al., 6 Dec 2025).
• Communication learning recaptures 75–80% of the noise-free reward versus ≈60% for zero-comm protocols and strictly outperforms nearest-neighbor, fixed, or random communication baselines (Zhao et al., 2024).

5. Theoretical Guarantees

Key theoretical contributions underpin networked RMAB tractability and performance:

• Indexability: Sufficient conditions (e.g., arm-level concavity/monotonicity in waiting/exposure) ensure Whittle indices exist and can be efficiently calculated (Herlihy et al., 2022, Ou et al., 2022).
• Submodularity: Guarantees from combinatorial optimization transfer directly to action selection in networked dynamics, providing $i \in V$6 optimality for greedy policies (Zhang et al., 6 Dec 2025).
• Contraction for Approximate Bellman Operators: Multi-step meta-MDP “hill-climbing Bellman” operators are provably contracting, guaranteeing value-iteration convergence under inexact greedy maximization (Zhang et al., 6 Dec 2025).
• Sample-Complexity and Communication: Communication-learning schemes provably improve sample efficiency, with regret bounds $i \in V$7 established for federated online RMABs (Tong et al., 2024).
• Sparse Communication Efficiency: Sparse graph learning is proven to match dense communication under standard coverage and entropy assumptions (Zhao et al., 2024).

6. Applications and Extensions

Networked RMABs have been applied and empirically validated across:

• Mobile health interventions with networked arms corresponding to locations; commuting/visit patterns drive positive externalities and reward coupling (Ou et al., 2022).
• Social/information networks with dynamic arm availability, modeling efficient querying of information sources under partial observability (Mehta et al., 2018).
• Epidemic control, vaccination, and public health domains modeling both direct and spillover effects on population graphs (Herlihy et al., 2022, Zhang et al., 6 Dec 2025).
• Resource allocation settings where data fidelity, privacy, and communication costs motivate communication-MDP variants and federated learning extensions (Tong et al., 2024, Zhao et al., 2024).

A plausible implication is that as networked and communication-coupled RMAB models become broader and more deeply integrated with real-world networks, future directions will need to address learning with unknown network structure, nonstationary environments, and fairness-constrained allocations at scale.