Stochastic Multiplayer Multi-Armed Bandit

• Stochastic MMAB is a decentralized framework where multiple players select from uncertain arms and coordinate to minimize collisions and maximize rewards.
• Feedback regimes such as collision sensing, no sensing, and gossip critically influence algorithm performance and regret trade-offs.
• Algorithmic strategies including index policies, collision-based communication, and trekking enable efficient exploration with logarithmic regret.

Stochastic multiplayer multi-armed bandits (MMAB) study repeated decentralized allocation of multiple agents or players to uncertain stochastic arms, with performance measured against an oracle that knows the arm means and allocates players to the best arms. In the classical decentralized formulation, $M$ players repeatedly choose among $K$ arms with unknown reward distributions, and simultaneous selection may induce a collision that suppresses reward; closely related lines of work study cooperative social learning with sparse gossip, no-sensing models in which a zero reward does not reveal collision, and shareable-resource models in which each arm can serve several players up to an unknown capacity (Besson et al., 2017, Boursier et al., 2018, Sankararaman et al., 2019, Wang et al., 2022).

1. Canonical problem formulation

A standard stochastic MMAB model has $M\ge 1$ players and $K\ge M$ arms. Arm $k$ generates an i.i.d. Bernoulli reward stream with mean $\mu_k$, players act simultaneously in discrete rounds, and the classical collision model imposes “full reward occlusion”: if two or more players select the same arm, they all receive $0$; otherwise a player receives the arm reward sample (Besson et al., 2017). In the equivalent formulation used in collision-sensing studies, player $j$’s realized reward can be written as

$$r^j(t)=X_{\pi^j(t)}(t)\bigl(1-\eta_{\pi^j(t)}(t)\bigr),$$

where $\eta_k(t)$ indicates whether two or more players chose arm $K$0 at round $K$1 (Boursier et al., 2018).

The canonical cooperative benchmark is the total mean of the best $K$2 arms. If $K$3 denotes the set of indices of the $K$4 highest-mean arms, then the expected regret of a decentralized policy $K$5 over horizon $K$6 is

$K$7

This regret admits a decomposition into three nonnegative terms: sub-optimal-arm selections, missing pulls of optimal arms, and collisions (Besson et al., 2017).

A distinct but related cooperative formulation replaces collision avoidance by decentralized information sharing on a common stochastic bandit instance. In that model, $K$8 agents solve the same $K$9-armed bandit in continuous time, each agent has an independent unit-rate Poisson clock, and after $M\ge 1$0 epochs of agent $M\ge 1$1 its cumulative regret is

$M\ge 1$2

with $M\ge 1$3 and gap $M\ge 1$4 (Sankararaman et al., 2019). This formulation emphasizes decentralized learning with communication constraints rather than collision externalities.

2. Observation and coordination regimes

The modern MMAB literature is organized largely by what a player can observe beyond its own realized reward. The feedback model determines both the achievable regret rate and the mechanism by which decentralized coordination is possible.

Regime	Information available to a player	Representative consequence
Collision sensing	Own reward and whether a collision occurred, or a sensing variant that reveals collision after sensing the arm reward	Enables asymptotically optimal index-based algorithms such as MCTopM and lower-bound analyses for decentralized policies (Besson et al., 2017)
No sensing	Only the realized reward $M\ge 1$5, so zero may mean collision or a genuine zero reward	Selfish is simple and competitive empirically but lacks a general $M\ge 1$6 guarantee; specialized algorithms can still obtain logarithmic regret in specific models (Besson et al., 2017, Boursier et al., 2018)
Pairwise gossip	A contacted peer, an arm identifier, and a limited number of bits	Social learning can achieve $M\ge 1$7 communications per agent while reducing per-agent regret (Sankararaman et al., 2019)
Shared randomness	A public random seed fixed before play, with no communication after time $M\ge 1$8	Near-optimal $M\ge 1$9 regret with zero collisions with probability at least $K\ge M$0 (Bubeck et al., 2020)
Random communication graph	Neighbors determined by an Erdős–Rényi graph, with exchange of arm and reward information	Increased connectivity accelerates belief sharing but can synchronize actions and increase collisions (Evirgen et al., 2017, Evirgen et al., 2017)
Shareable-resource feedback	Exact load on an arm (SDI), a one-bit sharing indicator (SDA), public request arrivals, or public profile counts	Supports logarithmic-regret algorithms for finite capacities and stochastic sharable capacities (Wang et al., 2022, Xie et al., 2024)

Within collision models, the distinction between sensing and no-sensing is especially consequential. With sensing, collisions can be used as an explicit coordination primitive or even as a communication channel; without sensing, algorithms must infer occupancy or capacity indirectly from reward patterns, which substantially changes both lower-bound arguments and constructive methods (Boursier et al., 2018).

3. Core algorithmic paradigms in the classical collision model

A first major line of work uses index policies together with decentralized orthogonalization. In the collision-sensing setting of “Multi-Player Bandits Revisited,” each player maintains arm indices such as kl-UCB values and selects among the current set of top-$K\ge M$1 arms. RandTopM performs “rank-free musical chair” randomization inside the top-$K\ge M$2 set, while MCTopM adds a sitting state under which a player remains on an arm from TopM after a collision-free success until that arm drops out of the current TopM. For RandTopM, MCTopM, or $K\ge M$3Rand, the expected number of selections of any sub-optimal arm matches the lower-bound constant in front of $K\ge M$4, and MCTopM further controls collisions so that total regret satisfies $K\ge M$5 (Besson et al., 2017).

A second line uses collisions themselves as a communication medium. SIC-MMAB showed that in the synchronized sensing model collisions can be deliberately enforced to transmit bits, so that decentralized players reconstruct centralized empirical information at negligible asymptotic cost. The algorithm combines Musical Chairs, sequential-hopping exploration, bit-wise send/receive primitives based on forced collisions, and successive accepts-rejects updates; under this design it achieves the same asymptotic rate as a centralized controller, thereby contradicting previously claimed decentralized lower bounds with an extra factor $K\ge M$6 (Boursier et al., 2018). DPE sharpened this approach by making exploration parsimonious: one leader explores with KL-UCB while followers exploit the current empirical top-$K\ge M$7 set, and short collision-based messages are sent only when the set changes. The resulting regret matches the Lai–Robbins lower-bound constant

$K\ge M$8

and the expected number of communication rounds is finite (Proutiere et al., 2019).

A third line minimizes coordination overhead without explicit estimation of the number of players. The trekking approach proceeds through random hopping for orthogonalization, sequential hopping for mean estimation, and a deterministic “trekking” phase in which players migrate toward better estimated arms. In the static setting, ST achieves constant regret in the horizon $K\ge M$9; in dynamic settings with arrivals and departures, the epoch-based DT algorithm achieves $k$0 regret and collision bounds, while the epoch-less DTS algorithm achieves $k$1 without global synchronization, provided players can detect the presence of other players on an arm (Hanawal et al., 2018).

A recent distributed line emphasizes both group and individual regret. SynCD begins with orthogonalization and rank assignment, performs exploration with elimination rules based on synchronized confidence intervals, and triggers communication only when an Estimated Confidence Radius shrinks by a multiplicative factor. In the collision-sensing setting, this yields near-optimal group regret, optimal individual regret, and communication cost of $k$2 up to polynomial factors in $k$3 and $k$4; the same work also studies a periodic asynchronous model and proves a logarithmic lower bound together with a matching $k$5-regret algorithm (Zhou et al., 8 Oct 2025).

4. Cooperation, social learning, and strategic interaction

Cooperation can also be organized through sparse information exchange rather than collision avoidance. In the distributed social-learning model, agents communicate only through a pairwise asynchronous gossip protocol with messages of length at most $k$6 bits, and the algorithm transmits only arm identifiers, not reward samples. Agents begin with distinct roughly $k$7-sized blocks of arms, propagate recommendations in early phases, and then run UCB over awareness sets that expand through received recommendations. The resulting per-agent regret is

$k$8

while each agent communicates only $k$9 times over $\mu_k$0 epochs (Sankararaman et al., 2019).

At the opposite extreme, communication can be completely absent after time $\mu_k$1. With shared randomness, a public random seed allows players to coordinate a combinatorial partition of the mean space into tree-labeled regions whose colorings prescribe distinct arms to distinct players. In this model, the algorithm of Bubeck, Budzinski, and Sellke guarantees that, with probability at least $\mu_k$2, no collisions occur over all $\mu_k$3 rounds, while the bandit regret is $\mu_k$4 (Bubeck et al., 2020). This is a worst-case $\mu_k$5 regime rather than the instance-dependent $\mu_k$6 regime of collision-sensing algorithms, but it removes collisions entirely.

Networked observation introduces a different cooperative mechanism. In the heterogeneous-stochastic-interactions model, agent $\mu_k$7 observes the choices and rewards of each neighbor with probability $\mu_k$8, called its sociability. Each agent runs a UCB-style policy on both self-observations and observed neighbor actions, and the regret of each agent remains $\mu_k$9. The analysis further defines a performance measure

$0$0

which predicts the rank ordering of agents according to performance (Madhushani et al., 2019).

Communication can also be harmful when it synchronizes beliefs faster than players can deconflict actions. In decentralized models with Erdős–Rényi communication graphs, players exchange the identity of the pulled arm and the realized reward with current neighbors after each round. The empirical finding is that regret increases monotonically with connectivity, the no-communication case attains sublinear performance, and the full-communication case can produce linear regret because all players tend to choose the same arm; this motivates collision-aware randomized allocation policies for $0$1 and cycle-coordination schemes for $0$2 (Evirgen et al., 2017, Evirgen et al., 2017).

A different branch studies strategic rather than fully cooperative interaction. In a two-player, two-arm discounted model, each player’s utility is $0$3 and $0$4, with $0$5 corresponding to zero-sum competition, $0$6 to neutrality, and $0$7 to full cooperation. In that setting, competing players explore less than a single player, cooperating players explore more than a single player, and neutral players learn from each other and receive strictly higher total rewards than they would playing alone for a range of safe-arm means $0$8. Moreover, for $0$9, both players eventually play the same arm forever in any Nash equilibrium, while this concentration property can fail for $j$0 (Brânzei et al., 2019).

5. Generalizations beyond unit-capacity stationary MMAB

Several extensions relax the unit-capacity collision model. In MMAB with finite shareable resources, each arm $j$1 has an unknown capacity $j$2 and per-load random reward $j$3 with mean $j$4, and when $j$5 players pull arm $j$6 their total reward is

$j$7

Two feedback models are studied: sharing demand information (SDI), where each player learns the exact number of co-players on the chosen arm, and sharing demand awareness (SDA), where the player learns only whether there was any sharing beyond itself. Under these feedback models, DPE-SDI and SIC-SDA both achieve logarithmic regret in $j$8 (Wang et al., 2022).

A closely related model incorporates stochastic request arrivals. In the stochastic sharable-capacities setting, arm $j$9 receives $$r^j(t)=X_{\pi^j(t)}(t)\bigl(1-\eta_{\pi^j(t)}(t)\bigr),$$0 requests in each slot, a player assigned to arm $$r^j(t)=X_{\pi^j(t)}(t)\bigl(1-\eta_{\pi^j(t)}(t)\bigr),$$1 can serve at most one request, and the expected total reward of assigning $$r^j(t)=X_{\pi^j(t)}(t)\bigl(1-\eta_{\pi^j(t)}(t)\bigr),$$2 players to arm $$r^j(t)=X_{\pi^j(t)}(t)\bigl(1-\eta_{\pi^j(t)}(t)\bigr),$$3 is

$$r^j(t)=X_{\pi^j(t)}(t)\bigl(1-\eta_{\pi^j(t)}(t)\bigr),$$4

The optimal static profile can be found by a greedy algorithm in $$r^j(t)=X_{\pi^j(t)}(t)\bigl(1-\eta_{\pi^j(t)}(t)\bigr),$$5 time via marginal gains, players can commit to a common known optimal profile in a constant number of rounds in expectation, and an explore-then-commit framework with an $$r^j(t)=X_{\pi^j(t)}(t)\bigl(1-\eta_{\pi^j(t)}(t)\bigr),$$6-round consensus routine yields $$r^j(t)=X_{\pi^j(t)}(t)\bigl(1-\eta_{\pi^j(t)}(t)\bigr),$$7 when the model parameters are unknown (Xie et al., 2024).

The no-sensing shareable-arm model is more difficult because each arm has an unknown capacity $$r^j(t)=X_{\pi^j(t)}(t)\bigl(1-\eta_{\pi^j(t)}(t)\bigr),$$8, and when the number of players exceeds $$r^j(t)=X_{\pi^j(t)}(t)\bigl(1-\eta_{\pi^j(t)}(t)\bigr),$$9 all players on that arm receive exactly zero reward. A-CAPELLA addresses this regime with implicit collision patterns on a designated communication arm, grouped round-robin sampling, synchronized successive elimination, and capacity estimation. The paper states that A-CAPELLA is the first fully decentralized, no-sensing algorithm that learns unknown capacities $\eta_k(t)$0, coordinates via collision patterns on a shareable arm, and achieves logarithmic instance-dependent regret (Hu et al., 11 Jun 2025).

Other extensions change the arm-access process or the environment itself. In multi-player multi-armed walking bandits, player $\eta_k(t)$1 only has access to a time-varying local subset $\eta_k(t)$2 of arms, collisions occur only when neighboring players choose the same accessible arm, and a consensus-based UCB scheme combined with Learn2Match and Learn2Rank achieves

$\eta_k(t)$3

under a fixed connected communication graph (Xiong et al., 2022). In abruptly changing environments, arm means are piecewise constant with $\eta_k(t)$4 unknown breakpoints, and sliding-window algorithms RR-SW-UCB# and SW-DLP attain group regret $\eta_k(t)$5 under the collision model (Wei et al., 2018).

6. Lower bounds, trade-offs, and open problems

The theory of stochastic MMAB is shaped by a persistent tension between statistical efficiency and decentralized coordination. Under sensing models 1 or 2, any strongly uniformly efficient decentralized policy satisfies

$\eta_k(t)$6

which refines an earlier lower bound of Zhao et al. by replacing $\eta_k(t)$7 with $\eta_k(t)$8 (Besson et al., 2017). SIC-MMAB then showed that earlier decentralized lower bounds with an additional factor $\eta_k(t)$9 were incorrect in the synchronized sensing model because they ignored the information content of collisions, and DPE subsequently matched the centralized Lai–Robbins constant with finite expected communication rounds (Boursier et al., 2018, Proutiere et al., 2019).

Communication complexity is a second recurring axis. In social learning, the no-communication baseline yields $K$00 per agent, while full interaction yields $K$01 per-agent regret but requires $K$02 communication attempts and $K$03 bits per agent. The gossip-based algorithm sits between these extremes, remaining within a $K$04 factor of the fully interacted optimum while using only $K$05 communications per agent (Sankararaman et al., 2019). In collision-sensing decentralized MMAB, recent work pursues the same direction by driving communication from explicit $K$06-scale protocols toward $K$07 updates (Zhou et al., 8 Oct 2025).

No-sensing settings remain structurally delicate. The Selfish heuristic is empirically competitive and requires neither sensing nor knowledge of $K$08, but for small $K$09 one can construct absorbing configurations in which two players’ index vectors remain identical after a collision, causing collisions forever with positive probability; consequently no general $K$10 guarantee is known for that heuristic (Besson et al., 2017). The no-sensing algorithm in the dynamic model of SIC-MMAB and the capacity-learning protocol of A-CAPELLA show that logarithmic regret is nevertheless attainable when occupancy or capacity can be inferred through carefully designed statistical tests and synchronization devices (Boursier et al., 2018, Hu et al., 11 Jun 2025).

The open problems named across the literature are consistent. They include lower-bounding the collision term in regret, extending collision-sensing algorithms to unknown or time-varying numbers of players, establishing a rigorous $K$11 analysis or impossibility result for no-sensing heuristics such as Selfish, and generalizing shareable-arm models to time-varying capacities, prioritized sharing, aggregate-reward feedback, partial sensing, and adversarial settings (Besson et al., 2017, Hu et al., 11 Jun 2025). A plausible implication is that the central question in stochastic MMAB is no longer only how to estimate arm means, but how to encode coordination under the exact observation constraints imposed by the model.