Multi-Player Multi-Armed Bandits

• MP-MAB are a class of problems that model decentralized, sequential arm selection among multiple players with collision avoidance and limited communication.
• The framework categorizes feedback into regimes like collision-sensing, no-sensing, and collision-dependent rewards, enabling implicit communication and coordination.
• Algorithmic paradigms such as random learning, index-based stabilization, and leader-follower schemes achieve near-optimal logarithmic regret even in heterogeneous and non-stationary environments.

Searching arXiv for relevant MP-MAB papers to ground the article with current references. Multi-Player Multi-Armed Bandits (MP-MAB), also called MMAB in several papers, study decentralized sequential allocation of a common set of arms by multiple players under interference, partial observability, and limited or absent communication. In the canonical formulation, $M$ players repeatedly choose among $K$ arms, and the performance criterion is regret relative to a centralized collision-free allocation of the best $M$ arms; however, the literature now includes collision sensing and no-sensing regimes, heterogeneous player-arm rewards, collision-dependent reward distributions, shareable finite-capacity arms, selfish or adversarial participants, abruptly changing environments, and asynchronous activation without a global clock (Kalathil et al., 2012, Besson et al., 2017, Shi et al., 2021, Pacchiano et al., 2021).

1. Formal models and regret benchmarks

A standard homogeneous MP-MAB instance has $K$ arms, $M\le K$ players, and stochastic rewards with unknown means. In the collision-eliminated model, if multiple players choose the same arm, they collide and receive no reward; in the collision-sensing setting, the colliding players are informed that a collision occurred, whereas in the no-sensing setting they observe only the realized reward (Besson et al., 2017, Proutiere et al., 2019, Shi et al., 2020). A representative regret benchmark is

$$R^\pi(T)=T\sum_{k=1}^M \mu_k-\sum_{t=1}^T\sum_{i=1}^M \mathbb{E}\big[\mu_{k_i^\pi(t)}\big],$$

or, equivalently in the homogeneous stochastic case,

$$\mathbb{E}[R(T)] = T\sum_{k=1}^{M}\mu(k)-\mathbb{E}_{\mu}\!\left[\sum_{t=1}^{T}\sum_{m=1}^{M} r_t^{(m)}\right].$$

The benchmark is the reward of always assigning the top-$M$ arms without collisions (Proutiere et al., 2019, Zhou et al., 8 Oct 2025).

A closely related formalization used in the cooperative no-sensing setting writes the pseudo-regret as

$$\mathcal{R}_T = T \max_{\mathbf{a}\in\{0,1\}^K:\sum_i a(i)=M}\langle \mathbf{a},\boldsymbol{\mu}\rangle -\sum_{t=1}^T\sum_{p=1}^M \mu_{i_t^p},$$

with arm ordering $\mu_{\sigma_1}\ge \cdots \ge \mu_{\sigma_K}$ and critical gap $K$0 (Pacchiano et al., 2021). This formulation makes explicit that the difficulty is not merely identifying the best individual arm, but separating the top $K$1 arms from the rest while coordinating decentralized play.

The literature also contains heterogeneous models in which arm $K$2 has player-dependent mean $K$3. In that case the natural benchmark is a maximum-weight matching

$K$4

or, in older decentralized formulations, the optimal bipartite assignment $K$5 (Kalathil et al., 2012, Shi et al., 2021). This shifts the problem from identifying a common top-$K$6 arm set to learning a collision-free matching.

A recurrent source of confusion is to treat zero-reward collisions as the only MP-MAB interaction model. The surveyed literature does not support that simplification. Collision-dependent rewards, resource sharing, and stochastic sharable capacities all define nonzero or load-dependent interaction models that are not reducible to collision elimination (Shi et al., 2021, Wang et al., 2022, Xie et al., 2024).

2. Feedback regimes and communication through collisions

A major organizing principle in MP-MAB is the feedback model. “Multi-Player Bandits Revisited” distinguishes three regimes: simultaneous sensing and collision, sensing then collision, and no sensing (Besson et al., 2017). The no-sensing case is strictly harder because a zero reward does not identify whether the cause was arm quality or a collision (Shi et al., 2020). In the collision-dependent reward model, the ambiguity is two-sided: bit $K$7 is transmitted through distribution $K$8 and bit $K$9 through distribution $M$0, and the supports may overlap (Shi et al., 2021).

This difficulty turned collisions from a nuisance into an implicit communication primitive. In collision-sensing algorithms such as DPE, the leader communicates only when the current best-arm set changes, and communication phases have fixed finite duration; the expected number of communication phases is finite, so communication does not affect the asymptotic $M$1 regret (Proutiere et al., 2019). In no-sensing settings, by contrast, collision-based messages must be decoded from reward samples. EC-SIC models the communication channel as a Z-channel with capacity

$M$2

where $M$3, and uses error-correction coding for fixed-length quantized means (Shi et al., 2020). EC3 generalizes this idea to collision-dependent rewards and explicitly frames the problem as reliable communication over a noisy channel, with random coding theorem guarantee

$M$4

and code length

$M$5

This makes the communication cost itself an information-theoretic design variable (Shi et al., 2021).

The cooperative no-sensing analysis with unknown collision reward introduces a different decoding mechanism. There, communication begins only after collision-free round-robin exploration has produced empirical confidence bands and a disconnected confidence graph. For player $M$6, the witness lower bound

$M$7

supports a threshold test $M$8 on empirically strong arms, and the analysis shows that all receivers recover each bit correctly with high probability on the good event $M$9 (Pacchiano et al., 2021). This suggests that in no-sensing MP-MAB, the key obstacle is not communication per se, but synchronization and decoding under weak side information.

Communication efficiency itself has become a first-class objective. SynCD communicates only when an Estimated Confidence Radius shrinks by a factor $K$0, sends differences between successive quantized estimates rather than full statistics, and obtains $K$1 communication cost together with near-optimal group and individual regret (Zhou et al., 8 Oct 2025).

3. Coordination mechanisms and representative algorithmic paradigms

The earliest decentralized line emphasized index policies plus explicit distributed assignment. The dUCB$K$2 algorithm uses UCB-style indices, exponentially spaced decision frames, and a distributed auction-based bipartite matching subroutine, achieving near-$K$3 expected regret in decentralized heterogeneous settings with costly communication (Kalathil et al., 2012). This line established the assignment-based view of MP-MAB.

A second paradigm uses orthogonalization and “freeze after coordination.” Musical Chairs first performs random learning, estimates the number of players from collision frequency, then repeatedly samples among the top estimated arms until each player finds a collision-free arm and sticks to it (Rosenski et al., 2015). Trekking refines this philosophy by separating learning a ranking of arms from trekking toward and locking onto the best available arm; in the static case, ST has constant regret, and in the dynamic case it yields sub-linear regret with high probability, while DTS removes the need for global time synchronization when arm-occupancy sensing is available (Hanawal et al., 2018).

A third paradigm is index-based local top-$K$4 stabilization. RandTopM and MCTopM maintain local confidence indices and a top-$K$5 set; once a player obtains a collision-free arm in TopM, MCTopM fixes that arm and thereby reduces both collisions and arm switches (Besson et al., 2017). This line is distinct from epochal orthogonalization because coordination persists through local confidence updates rather than through a single lock-in event.

Leader-follower algorithms form another major strand. DPE uses a random self-terminating initialization to orthogonalize players, determine $K$6, assign ranks, and designate a leader; only the leader performs KL-UCB exploration, while followers greedily cover the current best empirical arms. Communication occurs only when the best-arm set changes, and the communicated payload is only which arm left and which arm entered (Proutiere et al., 2019). BEACON extends this philosophy to heterogeneous MP-MAB through batched exploration, a CUCB-style optimistic matching oracle, and Adaptive Differential Communication (ADC), which transmits differences

$K$7

rather than full statistics (Shi et al., 2021).

Elimination-based recursion appears in the cooperative no-sensing literature. In the unknown-collision-reward setting, players first explore in a collision-free round-robin schedule, construct an arm connectivity graph from overlapping confidence intervals, communicate the top connected component through collisions on a high-mean arm, and recurse on the induced subproblems (Pacchiano et al., 2021). This replaces the traditional “synchronize then commit” structure by repeated statistically justified partitioning.

4. Regret theory and optimality benchmarks

The canonical asymptotic lower bound in collision-sensing homogeneous MP-MAB is

$K$8

which is also the classical centralized benchmark for uniformly good policies (Proutiere et al., 2019). In sensing-based decentralized models, “Multi-Player Bandits Revisited” proves the stronger decentralized lower bound on suboptimal pulls

$K$9

leading to a regret lower bound larger by a factor $M\le K$0 than the centralized one (Besson et al., 2017).

DPE closes this gap in the collision-sensing homogeneous case: $M\le K$1 Its initialization contributes only constant expected regret, and the expected number of communication rounds is finite (Proutiere et al., 2019). This is the clearest exact asymptotic optimality statement in the surveyed collision-sensing literature.

Several later works preserve logarithmic regret under harder observation models. EC-SIC approaches the centralized stochastic MP-MAB regret in the no-collision-information setting via Z-channel coding (Shi et al., 2020). EC3 proves logarithmic regret in the no-sensing collision-dependent reward model and exhibits the explicit tradeoff between code length and decoding error through the random coding exponent $M\le K$2 (Shi et al., 2021). The cooperative unknown-collision-reward analysis gives the first instance-dependent logarithmic regret guarantee in that regime, with a simplified bound of the form

$M\le K$3

with probability at least $M\le K$4 (Pacchiano et al., 2021).

For heterogeneous MP-MAB, BEACON gives

$M\le K$5

and extends to nonlinear system rewards under monotonicity and bounded smoothness assumptions (Shi et al., 2021). SynCD adds a fairness-oriented criterion, the individual regret

$M\le K$6

and proves that the leading term is smaller than the group regret scale by a factor $M\le K$7 (Zhou et al., 8 Oct 2025).

A plausible implication is that MP-MAB regret theory now separates into at least three layers: information-theoretic identification of good arms or matchings, communication complexity under partial feedback, and fairness or role asymmetry among players.

5. Heterogeneous, non-stationary, and asynchronous extensions

Heterogeneity changes both the oracle and the algorithmic toolkit. In decentralized heterogeneous MP-MAB, player $M\le K$8 pulling arm $M\le K$9 yields mean $$R^\pi(T)=T\sum_{k=1}^M \mu_k-\sum_{t=1}^T\sum_{i=1}^M \mathbb{E}\big[\mu_{k_i^\pi(t)}\big],$$0, and the optimal joint action is a maximum-weight matching rather than a common top-$$R^\pi(T)=T\sum_{k=1}^M \mu_k-\sum_{t=1}^T\sum_{i=1}^M \mathbb{E}\big[\mu_{k_i^\pi(t)}\big],$$1 arm set (Kalathil et al., 2012, Shi et al., 2021). BEACON operationalizes this with a CUCB-style oracle

$$R^\pi(T)=T\sum_{k=1}^M \mu_k-\sum_{t=1}^T\sum_{i=1}^M \mathbb{E}\big[\mu_{k_i^\pi(t)}\big],$$2

batched exploration of duration $$R^\pi(T)=T\sum_{k=1}^M \mu_k-\sum_{t=1}^T\sum_{i=1}^M \mathbb{E}\big[\mu_{k_i^\pi(t)}\big],$$3, and ADC-based communication (Shi et al., 2021). The same architecture extends to nonlinear rewards $$R^\pi(T)=T\sum_{k=1}^M \mu_k-\sum_{t=1}^T\sum_{i=1}^M \mathbb{E}\big[\mu_{k_i^\pi(t)}\big],$$4, including proportional fairness, minimal reward, thresholded reward, video-quality utility, and top-$$R^\pi(T)=T\sum_{k=1}^M \mu_k-\sum_{t=1}^T\sum_{i=1}^M \mathbb{E}\big[\mu_{k_i^\pi(t)}\big],$$5 utility (Shi et al., 2021).

Non-stationarity introduces time-varying optimal arm sets. In abruptly changing environments with $$R^\pi(T)=T\sum_{k=1}^M \mu_k-\sum_{t=1}^T\sum_{i=1}^M \mathbb{E}\big[\mu_{k_i^\pi(t)}\big],$$6, RR-SW-UCB# and SW-DLP use window width

$$R^\pi(T)=T\sum_{k=1}^M \mu_k-\sum_{t=1}^T\sum_{i=1}^M \mathbb{E}\big[\mu_{k_i^\pi(t)}\big],$$7

and achieve sublinear expected cumulative group regret of order $$R^\pi(T)=T\sum_{k=1}^M \mu_k-\sum_{t=1}^T\sum_{i=1}^M \mathbb{E}\big[\mu_{k_i^\pi(t)}\big],$$8 (Wei et al., 2018). Dynamic Musical Chairs and dynamic trekking address entering and leaving players through epochs or an epoch-less sensing-based design, with $$R^\pi(T)=T\sum_{k=1}^M \mu_k-\sum_{t=1}^T\sum_{i=1}^M \mathbb{E}\big[\mu_{k_i^\pi(t)}\big],$$9 regret in the dynamic case (Rosenski et al., 2015, Hanawal et al., 2018).

Fully asynchronous decentralized MP-MAB removes the global clock itself. ACE maintains, for each player, a set $$\mathbb{E}[R(T)] = T\sum_{k=1}^{M}\mu(k)-\mathbb{E}_{\mu}\!\left[\sum_{t=1}^{T}\sum_{m=1}^{M} r_t^{(m)}\right].$$0 of arms currently believed to be occupied and alternates between exploration and exploitation. Occupied or released arms are detected through queue-based tests with

$$\mathbb{E}[R(T)] = T\sum_{k=1}^{M}\mu(k)-\mathbb{E}_{\mu}\!\left[\sum_{t=1}^{T}\sum_{m=1}^{M} r_t^{(m)}\right].$$1

and the resulting regret is

$$\mathbb{E}[R(T)] = T\sum_{k=1}^{M}\mu(k)-\mathbb{E}_{\mu}\!\left[\sum_{t=1}^{T}\sum_{m=1}^{M} r_t^{(m)}\right].$$2

under arbitrary arrivals and departures (Fan et al., 30 Sep 2025). This is a substantive departure from synchronization-based protocols: coordination is inferred from local collision histories rather than from shared time.

6. Shareable resources, strategic behavior, and robustness

Several recent models replace winner-takes-all collisions by load-dependent sharing. In MMAB-SA, arm $$\mathbb{E}[R(T)] = T\sum_{k=1}^{M}\mu(k)-\mathbb{E}_{\mu}\!\left[\sum_{t=1}^{T}\sum_{m=1}^{M} r_t^{(m)}\right].$$3 has unknown per-load reward $$\mathbb{E}[R(T)] = T\sum_{k=1}^{M}\mu(k)-\mathbb{E}_{\mu}\!\left[\sum_{t=1}^{T}\sum_{m=1}^{M} r_t^{(m)}\right].$$4 with mean $$\mathbb{E}[R(T)] = T\sum_{k=1}^{M}\mu(k)-\mathbb{E}_{\mu}\!\left[\sum_{t=1}^{T}\sum_{m=1}^{M} r_t^{(m)}\right].$$5 and unknown capacity $$\mathbb{E}[R(T)] = T\sum_{k=1}^{M}\mu(k)-\mathbb{E}_{\mu}\!\left[\sum_{t=1}^{T}\sum_{m=1}^{M} r_t^{(m)}\right].$$6, and the total reward from $$\mathbb{E}[R(T)] = T\sum_{k=1}^{M}\mu(k)-\mathbb{E}_{\mu}\!\left[\sum_{t=1}^{T}\sum_{m=1}^{M} r_t^{(m)}\right].$$7 players on that arm is

$$\mathbb{E}[R(T)] = T\sum_{k=1}^{M}\mu(k)-\mathbb{E}_{\mu}\!\left[\sum_{t=1}^{T}\sum_{m=1}^{M} r_t^{(m)}\right].$$8

Two feedback models are studied: Sharing Demand Information (SDI), which reveals $$\mathbb{E}[R(T)] = T\sum_{k=1}^{M}\mu(k)-\mathbb{E}_{\mu}\!\left[\sum_{t=1}^{T}\sum_{m=1}^{M} r_t^{(m)}\right].$$9, and Sharing Demand Awareness (SDA), which reveals only $M$0. The corresponding decentralized algorithms, DPE-SDI and SIC-SDA, have logarithmic regret (Wang et al., 2022). A related distributed-selection model with stochastic sharable arm capacities uses request arrivals $M$1, arm-pulling profiles $M$2, a greedy offline optimizer with complexity $M$3, a distributed commitment algorithm with constant expected rounds, and an ETC online algorithm with $M$4 regret (Xie et al., 2024).

Strategic MP-MAB introduces equilibrium considerations. “Selfish Robustness and Equilibria in Multi-Player Bandits” studies $M$5-Nash equilibrium and $M$6-stability under collisions. It gives the first logarithmic-regret algorithm robust to selfish players in the statistic sensing setting, constructs Grim Trigger-type robust communication and punishment in the full-sensing homogeneous and $M$7-heterogeneous cases, and proves impossibility results in the no-sensing case and under arbitrary heterogeneity when regret is measured against the optimal assignment (2002.01197). A different strategic model assumes averaging allocation: $M$8 which yields a singleton congestion game with unique equilibrium occupancy vector

$M$9

The SMAA algorithm achieves $$\mathcal{R}_T = T \max_{\mathbf{a}\in\{0,1\}^K:\sum_i a(i)=M}\langle \mathbf{a},\boldsymbol{\mu}\rangle -\sum_{t=1}^T\sum_{p=1}^M \mu_{i_t^p},$$0 regret for each player when all players follow the algorithm and the equilibria are even strong Nash equilibria (Xu et al., 2023).

A second robustness line treats adversarial attacks rather than selfish deviation. In heterogeneous MP-MAB with attacks that mimic collisions, a common policy for all players uses collision-free exploration, a payoff-based matching phase with one-bit communication, and achieves

$$\mathcal{R}_T = T \max_{\mathbf{a}\in\{0,1\}^K:\sum_i a(i)=M}\langle \mathbf{a},\boldsymbol{\mu}\rangle -\sum_{t=1}^T\sum_{p=1}^M \mu_{i_t^p},$$1

regret, where $$\mathcal{R}_T = T \max_{\mathbf{a}\in\{0,1\}^K:\sum_i a(i)=M}\langle \mathbf{a},\boldsymbol{\mu}\rangle -\sum_{t=1}^T\sum_{p=1}^M \mu_{i_t^p},$$2 is the total number of time units for which there was an adversarial attack on at least one arm (Magesh et al., 21 Jan 2025). In no-sensing adversarial MP-MAB, A2C2 organizes algorithms by local or global attackability and uses forced-collision communication, error-correction coding, and randomized synchronization to obtain asymptotic attackability-dependent sublinear regret (Shi et al., 2020).

Taken together, these extensions show that MP-MAB is no longer a single model class defined by zero-reward collisions and obedient cooperative players. It is a broader theory of decentralized online allocation under shared uncertainty, where the interaction law may be collision-eliminated, collision-dependent, capacity-limited, or strategically manipulated, and where the central technical question is how much coordination can be recovered from local feedback alone.