---
title: Stochastic Multiplayer Multi-Armed Bandit
url: https://www.emergentmind.com/topics/stochastic-multiplayer-multi-armed-bandit-mmab
type: topic
---

# Stochastic Multiplayer Multi-Armed Bandit

Stochastic multiplayer multi-armed bandits (MMAB) study repeated decentralized allocation of multiple agents or players to uncertain stochastic arms, with performance measured against an oracle that knows the arm means and allocates players to the best arms. In the classical decentralized formulation, \(M\) players repeatedly choose among \(K\) arms with unknown reward distributions, and simultaneous selection may induce a collision that suppresses reward; closely related lines of work study cooperative social learning with sparse gossip, no-sensing models in which a zero reward does not reveal collision, and shareable-resource models in which each arm can serve several players up to an unknown capacity [1711.02317] [1809.08151] [1910.02100] [2204.13502].

## 1. Canonical problem formulation

A standard stochastic MMAB model has \(M\ge 1\) players and \(K\ge M\) arms. Arm \(k\) generates an i.i.d. Bernoulli reward stream with mean \(\mu_k\), players act simultaneously in discrete rounds, and the classical collision model imposes “full reward occlusion”: if two or more players select the same arm, they all receive \(0\); otherwise a player receives the arm reward sample [1711.02317]. In the equivalent formulation used in collision-sensing studies, player \(j\)’s realized reward can be written as
\[
r^j(t)=X_{\pi^j(t)}(t)\bigl(1-\eta_{\pi^j(t)}(t)\bigr),
\]
where \(\eta_k(t)\) indicates whether two or more players chose arm \(k\) at round \(t\) [1809.08151].

The canonical cooperative benchmark is the total mean of the best \(M\) arms. If \(S_M^*\) denotes the set of indices of the \(M\) highest-mean arms, then the expected regret of a decentralized policy \(\rho\) over horizon \(T\) is
\[
R_T(\mu,M,\rho)=\Bigl(\sum_{k\in S_M^*}\mu_k\Bigr)T-\mathbb E\Bigl[\sum_{t\le T}\sum_{j=1}^M r^j(t)\Bigr].
\]
This regret admits a decomposition into three nonnegative terms: sub-optimal-arm selections, missing pulls of optimal arms, and collisions [1711.02317].

A distinct but related cooperative formulation replaces collision avoidance by decentralized information sharing on a common stochastic bandit instance. In that model, \(n\) agents solve the same \(K\)-armed bandit in continuous time, each agent has an independent unit-rate Poisson clock, and after \(T\) epochs of agent \(i\) its cumulative regret is
\[
R_i(T)=\sum_{t=1}^T\bigl(\mu_1-\mu_{I_i^{(t)}}\bigr),
\]
with \(\mu_1>\mu_2\ge \dots \ge \mu_K\) and gap \(\Delta=\mu_1-\mu_2>0\) [1910.02100]. This formulation emphasizes decentralized learning with communication constraints rather than collision externalities.

## 2. Observation and coordination regimes

The modern MMAB literature is organized largely by what a player can observe beyond its own realized reward. The feedback model determines both the achievable regret rate and the mechanism by which decentralized coordination is possible.

| Regime | Information available to a player | Representative consequence |
|---|---|---|
| Collision sensing | Own reward and whether a collision occurred, or a sensing variant that reveals collision after sensing the arm reward | Enables asymptotically optimal index-based algorithms such as MCTopM and lower-bound analyses for decentralized policies [1711.02317] |
| No sensing | Only the realized reward \(r^j(t)\), so zero may mean collision or a genuine zero reward | Selfish is simple and competitive empirically but lacks a general \(\log T\) guarantee; specialized algorithms can still obtain logarithmic regret in specific models [1711.02317] [1809.08151] |
| Pairwise gossip | A contacted peer, an arm identifier, and a limited number of bits | Social learning can achieve \(O(\log T)\) communications per agent while reducing per-agent regret [1910.02100] |
| Shared randomness | A public random seed fixed before play, with no communication after time \(0\) | Near-optimal \(\tilde O(\sqrt T)\) regret with zero collisions with probability at least \(1-1/T\) [2011.03896] |
| Random communication graph | Neighbors determined by an Erdős–Rényi graph, with exchange of arm and reward information | Increased connectivity accelerates belief sharing but can synchronize actions and increase collisions [1711.01628] [1712.00656] |
| Shareable-resource feedback | Exact load on an arm (SDI), a one-bit sharing indicator (SDA), public request arrivals, or public profile counts | Supports logarithmic-regret algorithms for finite capacities and stochastic sharable capacities [2204.13502] [2408.10865] |

Within collision models, the distinction between sensing and no-sensing is especially consequential. With sensing, collisions can be used as an explicit coordination primitive or even as a communication channel; without sensing, algorithms must infer occupancy or capacity indirectly from reward patterns, which substantially changes both lower-bound arguments and constructive methods [1809.08151].

## 3. Core algorithmic paradigms in the classical collision model

A first major line of work uses index policies together with decentralized orthogonalization. In the collision-sensing setting of “Multi-Player Bandits Revisited,” each player maintains arm indices such as kl-UCB values and selects among the current set of top-\(M\) arms. RandTopM performs “rank-free musical chair” randomization inside the top-\(M\) set, while MCTopM adds a sitting state under which a player remains on an arm from TopM after a collision-free success until that arm drops out of the current TopM. For RandTopM, MCTopM, or \(\rho\)Rand, the expected number of selections of any sub-optimal arm matches the lower-bound constant in front of \(\log T\), and MCTopM further controls collisions so that total regret satisfies \(R_T(\mu,M,\text{MCTopM-klUCB}) \le C_{\mu,M}\log T + o(\log T)\) [1711.02317].

A second line uses collisions themselves as a communication medium. SIC-MMAB showed that in the synchronized sensing model collisions can be deliberately enforced to transmit bits, so that decentralized players reconstruct centralized empirical information at negligible asymptotic cost. The algorithm combines Musical Chairs, sequential-hopping exploration, bit-wise send/receive primitives based on forced collisions, and successive accepts-rejects updates; under this design it achieves the same asymptotic rate as a centralized controller, thereby contradicting previously claimed decentralized lower bounds with an extra factor \(M\) [1809.08151]. DPE sharpened this approach by making exploration parsimonious: one leader explores with KL-UCB while followers exploit the current empirical top-\(M\) set, and short collision-based messages are sent only when the set changes. The resulting regret matches the Lai–Robbins lower-bound constant
\[
C(\mu)=\sum_{k>M}\frac{\mu_M-\mu_k}{\mathrm{kl}(\mu_k,\mu_M)},
\]
and the expected number of communication rounds is finite [1909.13079].

A third line minimizes coordination overhead without explicit estimation of the number of players. The trekking approach proceeds through random hopping for orthogonalization, sequential hopping for mean estimation, and a deterministic “trekking” phase in which players migrate toward better estimated arms. In the static setting, ST achieves constant regret in the horizon \(T\); in dynamic settings with arrivals and departures, the epoch-based DT algorithm achieves \(\tilde O(\sqrt{xT})\) regret and collision bounds, while the epoch-less DTS algorithm achieves \(\tilde O(K\sqrt{xT})\) without global synchronization, provided players can detect the presence of other players on an arm [1809.06040].

A recent distributed line emphasizes both group and individual regret. SynCD begins with orthogonalization and rank assignment, performs exploration with elimination rules based on synchronized confidence intervals, and triggers communication only when an Estimated Confidence Radius shrinks by a multiplicative factor. In the collision-sensing setting, this yields near-optimal group regret, optimal individual regret, and communication cost of \(O(\log\log T)\) up to polynomial factors in \(M\) and \(K\); the same work also studies a periodic asynchronous model and proves a logarithmic lower bound together with a matching \(O(\log T)\)-regret algorithm [2510.06683].

## 4. Cooperation, social learning, and strategic interaction

Cooperation can also be organized through sparse information exchange rather than collision avoidance. In the distributed social-learning model, agents communicate only through a pairwise asynchronous gossip protocol with messages of length at most \(O(\log(nK))\) bits, and the algorithm transmits only arm identifiers, not reward samples. Agents begin with distinct roughly \(K/n\)-sized blocks of arms, propagate recommendations in early phases, and then run UCB over awareness sets that expand through received recommendations. The resulting per-agent regret is
\[
R_i(T)=O\!\left(\frac{\lceil K/n\rceil+\log n}{\Delta}\log T+\frac{\log^3 n\,\log\log n}{\Delta^2}\right),
\]
while each agent communicates only \(\Theta(\log T)\) times over \(T\) epochs [1910.02100].

At the opposite extreme, communication can be completely absent after time \(0\). With shared randomness, a public random seed allows players to coordinate a combinatorial partition of the mean space into tree-labeled regions whose colorings prescribe distinct arms to distinct players. In this model, the algorithm of Bubeck, Budzinski, and Sellke guarantees that, with probability at least \(1-1/T\), no collisions occur over all \(T\) rounds, while the bandit regret is \(O(MK^{11/2}\sqrt{T\log T})\) [2011.03896]. This is a worst-case \(\tilde O(\sqrt T)\) regime rather than the instance-dependent \(\log T\) regime of collision-sensing algorithms, but it removes collisions entirely.

Networked observation introduces a different cooperative mechanism. In the heterogeneous-stochastic-interactions model, agent \(k\) observes the choices and rewards of each neighbor with probability \(p_k\), called its sociability. Each agent runs a UCB-style policy on both self-observations and observed neighbor actions, and the regret of each agent remains \(O(\log T)\). The analysis further defines a performance measure
\[
\epsilon_p^k=\frac{1}{p_k+1}\sqrt{\frac{1}{d_k\sum_{j\in\mathcal N_k}p_j}},
\]
which predicts the rank ordering of agents according to performance [1905.08731].

Communication can also be harmful when it synchronizes beliefs faster than players can deconflict actions. In decentralized models with Erdős–Rényi communication graphs, players exchange the identity of the pulled arm and the realized reward with current neighbors after each round. The empirical finding is that regret increases monotonically with connectivity, the no-communication case attains sublinear performance, and the full-communication case can produce linear regret because all players tend to choose the same arm; this motivates collision-aware randomized allocation policies for \(\alpha=1\) and cycle-coordination schemes for \(0<\alpha<1\) [1711.01628] [1712.00656].

A different branch studies strategic rather than fully cooperative interaction. In a two-player, two-arm discounted model, each player’s utility is \(u_A=\Gamma_A+\lambda\Gamma_B\) and \(u_B=\Gamma_B+\lambda\Gamma_A\), with \(\lambda=-1\) corresponding to zero-sum competition, \(\lambda=0\) to neutrality, and \(\lambda=1\) to full cooperation. In that setting, competing players explore less than a single player, cooperating players explore more than a single player, and neutral players learn from each other and receive strictly higher total rewards than they would playing alone for a range of safe-arm means \(p\in(p^*,g)\). Moreover, for \(\lambda\in\{-1,0\}\), both players eventually play the same arm forever in any Nash equilibrium, while this concentration property can fail for \(\lambda=1\) [1908.01135].

## 5. Generalizations beyond unit-capacity stationary MMAB

Several extensions relax the unit-capacity collision model. In MMAB with finite shareable resources, each arm \(i\) has an unknown capacity \(R_i\) and per-load random reward \(X_i\) with mean \(\mu_i\), and when \(n\) players pull arm \(i\) their total reward is
\[
r_i(n)=\min\{n,R_i\}X_i.
\]
Two feedback models are studied: sharing demand information (SDI), where each player learns the exact number of co-players on the chosen arm, and sharing demand awareness (SDA), where the player learns only whether there was any sharing beyond itself. Under these feedback models, DPE-SDI and SIC-SDA both achieve logarithmic regret in \(T\) [2204.13502].

A closely related model incorporates stochastic request arrivals. In the stochastic sharable-capacities setting, arm \(m\) receives \(D_{t,m}\) requests in each slot, a player assigned to arm \(m\) can serve at most one request, and the expected total reward of assigning \(n_{t,m}\) players to arm \(m\) is
\[
U_m(n_{t,m};p_m,\mu_m)=\mu_m\,\mathbb E[\min\{n_{t,m},D_{t,m}\}].
\]
The optimal static profile can be found by a greedy algorithm in \(O(KM)\) time via marginal gains, players can commit to a common known optimal profile in a constant number of rounds in expectation, and an explore-then-commit framework with an \(M\)-round consensus routine yields \(R_T=O(\log T)\) when the model parameters are unknown [2408.10865].

The no-sensing shareable-arm model is more difficult because each arm has an unknown capacity \(C_j\ge 1\), and when the number of players exceeds \(C_j\) all players on that arm receive exactly zero reward. A-CAPELLA addresses this regime with implicit collision patterns on a designated communication arm, grouped round-robin sampling, synchronized successive elimination, and capacity estimation. The paper states that A-CAPELLA is the first fully decentralized, no-sensing algorithm that learns unknown capacities \(C_j\ge 1\), coordinates via collision patterns on a shareable arm, and achieves logarithmic instance-dependent regret [2506.10127].

Other extensions change the arm-access process or the environment itself. In multi-player multi-armed walking bandits, player \(i\) only has access to a time-varying local subset \(S_i(t)\) of arms, collisions occur only when neighboring players choose the same accessible arm, and a consensus-based UCB scheme combined with Learn2Match and Learn2Rank achieves
\[
R(T)=O\bigl((K^2+NK)\log T\bigr)
\]
under a fixed connected communication graph [2212.06279]. In abruptly changing environments, arm means are piecewise constant with \(C=O(T^\nu)\) unknown breakpoints, and sliding-window algorithms RR-SW-UCB# and SW-DLP attain group regret \(O(T^{(1+\nu)/2}\ln T)\) under the collision model [1812.05165].

## 6. Lower bounds, trade-offs, and open problems

The theory of stochastic MMAB is shaped by a persistent tension between statistical efficiency and decentralized coordination. Under sensing models 1 or 2, any strongly uniformly efficient decentralized policy satisfies
\[
\liminf_{T\to\infty}\frac{R_T(\mu,M,\rho)}{\log T}\ge
M\sum_{k\notin S_M^*}\frac{\Delta_k}{\mathrm{kl}(\mu_k,\mu_M^*)},
\]
which refines an earlier lower bound of Zhao et al. by replacing \(\sum_{j=1}^M 1/\mathrm{kl}(\mu_k,\mu_j^*)\) with \(M/\mathrm{kl}(\mu_k,\mu_M^*)\) [1711.02317]. SIC-MMAB then showed that earlier decentralized lower bounds with an additional factor \(M\) were incorrect in the synchronized sensing model because they ignored the information content of collisions, and DPE subsequently matched the centralized Lai–Robbins constant with finite expected communication rounds [1809.08151] [1909.13079].

Communication complexity is a second recurring axis. In social learning, the no-communication baseline yields \(R_i(T)=\Theta((K/\Delta)\log T)\) per agent, while full interaction yields \(\Theta((K/(n\Delta))\log T + (K\log n)/(n\Delta))\) per-agent regret but requires \(T\) communication attempts and \(\Omega(T\log n)\) bits per agent. The gossip-based algorithm sits between these extremes, remaining within a \(\log n\) factor of the fully interacted optimum while using only \(O(\log T)\) communications per agent [1910.02100]. In collision-sensing decentralized MMAB, recent work pursues the same direction by driving communication from explicit \(O(\log T)\)-scale protocols toward \(O(\log\log T)\) updates [2510.06683].

No-sensing settings remain structurally delicate. The Selfish heuristic is empirically competitive and requires neither sensing nor knowledge of \(M\), but for small \(K\) one can construct absorbing configurations in which two players’ index vectors remain identical after a collision, causing collisions forever with positive probability; consequently no general \(\log T\) guarantee is known for that heuristic [1711.02317]. The no-sensing algorithm in the dynamic model of SIC-MMAB and the capacity-learning protocol of A-CAPELLA show that logarithmic regret is nevertheless attainable when occupancy or capacity can be inferred through carefully designed statistical tests and synchronization devices [1809.08151] [2506.10127].

The open problems named across the literature are consistent. They include lower-bounding the collision term in regret, extending collision-sensing algorithms to unknown or time-varying numbers of players, establishing a rigorous \(\log T\) analysis or impossibility result for no-sensing heuristics such as Selfish, and generalizing shareable-arm models to time-varying capacities, prioritized sharing, aggregate-reward feedback, partial sensing, and adversarial settings [1711.02317] [2506.10127]. A plausible implication is that the central question in stochastic MMAB is no longer only how to estimate arm means, but how to encode coordination under the exact observation constraints imposed by the model.

Source: https://www.emergentmind.com/topics/stochastic-multiplayer-multi-armed-bandit-mmab