Decentralized Multi-Player Q-Learning in Episodic Markov Decision Processes with Information Asymmetry

Abstract: We study decentralized multi-player reinforcement learning in episodic tabular Markov decision processes (MDPs) under three forms of information asymmetry: (A) unobserved actions with common rewards, (B) observed actions with independent rewards, and (C) unobserved actions with independent rewards. Players cannot communicate during learning but may agree on a protocol a priori. For Problems A and B we propose \texttt{mQ-learning} and \texttt{mQ-learning-intervals}, achieving $$\tilde{O}(\sqrt{H⁴ S A_{\text{joint}}\, T})$$ regret, where $H$ is the horizon, $S$ the state count, $T = KH$ the total steps, and $$A_{\text{joint}} = \prod_{i=1}^M |\mathcal{A}<em>i|$$ the joint action space across $M$ players. For Problem C we give \texttt{mEXC} and \texttt{mEXC-Bellman}, two-phase explore-then-commit algorithms with regret $$\tilde{O}(H (S A</em>{\text{joint}})^{1/3} T<sup>{2/3})$$. Against the centralized joint-action benchmark, decentralized learning under information asymmetry matches the single-agent Q-learning rate of \cite{jin2018q} up to logarithmic factors. Because $A_{\text{joint}}$ grows exponentially in $M$, the bounds are most meaningful for small $M$ or small per-player action sets.

• The paper introduces mQ-learning algorithms that match the centralized joint-action regret rate of approximately Õ(√(H⁴SA_jointT)) when players share rewards or observe actions.
• Deterministic tie-breaking, confidence intervals, and deviations as implicit signals let players coordinate adaptive exploration without communication, despite independent reward observations.
• Under full information asymmetry, mEXC and mEXC-Bellman use explore-then-commit schedules and achieve Õ(H(SA_joint)¹⁄³T²⁄³) regret, leaving optimal rates and scalable extensions open.

This paper studies decentralized, communication-free reinforcement learning for $M$ players in episodic tabular MDPs under three information-asymmetry models: (A) unobserved actions with common rewards, (B) observed actions with independent rewards, and (C) unobserved actions with independent rewards. The central contribution is a set of algorithms—mQ-learning, mQ-learning-intervals, mEXC, and mEXC-Bellman—that achieve regret matching the centralized joint-action benchmark of Jin et al.'s UCB Q-learning (Saiki et al., 2018) up to logarithmic factors for Problems A and B, and an explore-then-commit rate for Problem C.

Model and problem setup

The setting is a layered episodic tabular MDP $(\mathcal{S}, \mathcal{A}, H, \mathbb{P}, r)$ with $S$ states, horizon $H$, $K$ episodes, $T = KH$ total steps, and joint action space $$\mathcal{A} = \mathcal{A}_1 \times \cdots \times \mathcal{A}_M$$ of size $A_{\text{joint}} = \prod_i |\mathcal{A}_i|$. Rewards are 1-subgaussian with unknown means $\mu_{\bm{a}}$, transitions are unknown, and per-player regret is defined against the optimal joint policy: $R_T = \sum_k [V_1^*(x_1^k) - V_1^{\pi_k}(x_1^k)]$. Players may agree on a deterministic protocol before learning but cannot communicate during it.

The three asymmetry models differ in what each player observes:

• Problem A: each player sees states, common rewards, and only its own action.
• Problem B: each player sees all actions but receives an independent i.i.d. copy of the reward.
• Problem C: each player sees only its own action and its own reward; this strictly generalizes both A and B.

Implicit coordination via deterministic tie-breaking

The key mechanism, imported from cooperative multi-player bandits (Chang et al., 2021), is that if all players run the same deterministic algorithm on shared information, they independently reach identical decisions. The paper formalizes this through a lexicographic order on joint-action tuples (viewing $(\mathcal{S}, \mathcal{A}, H, \mathbb{P}, r)$0 as a base-$(\mathcal{S}, \mathcal{A}, H, \mathbb{P}, r)$1 number) used to break ties deterministically.

For Problem A, mQ-learning is exactly Jin et al.'s UCB Q-learning applied to the joint action space, with lexicographic tie-breaking. The synchronization argument is a one-line invariant: since rewards and next states are common to all players, and updates depend only on these plus visit counts, identical initializations ($(\mathcal{S}, \mathcal{A}, H, \mathbb{P}, r)$2) imply identical Q-tables at every step. No probabilistic argument is needed; agreement is structural rather than statistical. The resulting guarantee is $(\mathcal{S}, \mathcal{A}, H, \mathbb{P}, r)$3 regret with probability $(\mathcal{S}, \mathcal{A}, H, \mathbb{P}, r)$4, matching the single-agent rate against the joint-action benchmark. The proof follows the standard optimism-plus-recursion template: a learning-rate weight decomposition (Lemma on weight properties), a recursion decomposing $(\mathcal{S}, \mathcal{A}, H, \mathbb{P}, r)$5 into initialization bias, propagated error, and transition noise, optimism via Azuma–Hoeffding concentration, and pigeonhole bounding of bonus sums.

For Problem B, independent rewards destroy table identity, so mQ-learning-intervals maintains upper and lower confidence bounds at each state-action pair and a "desired set" of plausible optima. Because actions are observable, a unilateral deviation from the agreed candidate acts as a one-bit implicit signal: the deviator has certified the candidate suboptimal via its confidence interval, and all players eliminate it. A notable technical point is that interval width after $(\mathcal{S}, \mathcal{A}, H, \mathbb{P}, r)$6 visits equals $(\mathcal{S}, \mathcal{A}, H, \mathbb{P}, r)$7, a deterministic function of $(\mathcal{S}, \mathcal{A}, H, \mathbb{P}, r)$8 identical across players—the intervals are merely shifted by independent reward noise. Optimism must be union-bounded over $(\mathcal{S}, \mathcal{A}, H, \mathbb{P}, r)$9 players, contributing a $S$0 factor absorbed into the logarithm. The regret is again $S$1; the analysis adds a "slack" term from playing non-greedy desired-set actions, bounded by the interval width, which decays at the same rate as the exploration bonus and therefore does not change the leading order.

Explore-then-commit for full asymmetry

Problem C removes both coordination channels: deviations cannot be observed, and reward streams are not shared. The paper proposes two-phase algorithms. During exploration, players play the least-visited joint action with lexicographic tie-breaking—a rule depending only on visit counts, which are common because exploration is joint. At a predetermined episode $S$2 (computable from common knowledge), all players switch simultaneously to greedy play under their own learned tables. mEXC uses incremental Q-learning updates during exploration; mEXC-Bellman uses plug-in empirical Bellman estimates with empirical transition counts, which can yield tighter estimates at the cost of storing transition counts.

The regret is $S$3: exploration costs $S$4, commit-phase suboptimality is bounded by the post-exploration value-function error $S$5, and balancing the two terms yields the rate. The hard switch (rather than $S$6-greedy) is necessary because $S$7-greedy would require shared randomness when actions are unobserved. The $S$8 rate is the standard penalty of explore-then-commit when the suboptimality gap is unknown; whether $S$9 is achievable for Problem C is explicitly left open.

Summary of guarantees

Problem	Algorithm	Regret (high probability)
A (unobserved actions, common rewards)	mQ-learning	$H$0
B (observed actions, independent rewards)	mQ-learning-intervals	$H$1
C (full asymmetry)	mEXC / mEXC-Bellman	$H$2

A useful reading offered by the authors is that observability of either actions or rewards suffices to encode a low-bandwidth inter-player signal adequate for adaptive UCB-style exploration; removing both channels leaves only coarse shared visit-count information, forcing a non-adaptive schedule.

Scope and limitations

The paper is candid about several constraints. First, all bounds scale with $H$3, which grows exponentially in $H$4. This dependence is not attributable to asymmetry—even a centralized learner pays $H$5 in the tabular setting—but it means the results are most meaningful for small $H$6 or small per-player action sets. Concretely, the authors estimate that with $H$7, three actions per player, $H$8, $H$9, per-episode suboptimality drops below $K$0 around $K$1 episodes, whereas $K$2 requires $K$3. Notably, the bounds contain no separate polynomial factor in $K$4: repartitioning a fixed joint action space among more players does not change regret. Second, no lower bounds specific to each asymmetry model are provided, so it remains unverified whether the $K$5 rates for Problems A and B or the $K$6 rate for Problem C are optimal under asymmetry. Third, the analysis is confined to tabular MDPs; extensions to factored structure, mean-field couplings, or linear function approximation are identified as necessary for scalability but not addressed. Fourth, the Problem C commit-phase argument assumes that after sufficient exploration, all players' concentrated Q-tables agree on the same greedy action—an assumption justified by optimism plus tie-breaking but dependent on the uniform closeness of every player's table to $K$7.

Conclusion

The paper extends the information-asymmetry framework of cooperative multi-player bandits to episodic MDPs and shows that pre-agreed deterministic protocols—lexicographic tie-breaking, deviation-as-signal elimination, and shared exploration schedules—recover the centralized joint-action Q-learning rate when either actions or rewards are observable, degrading to an explore-then-commit $K$8 rate under full asymmetry. The main open questions left by the work are the optimal regret rate for Problem C, instance-dependent or asymmetry-specific lower bounds, and regret–communication tradeoffs in decentralized multi-player MDPs.