Unsynchronized Decentralized Q-Learning Overview

• Unsynchronized decentralized Q-learning is a multi-agent reinforcement learning approach where agents update Q-values independently without a shared global revision clock, leading to non-stationary dynamics.
• It employs local temporal-difference updates with asynchronous phase-based revisions and inertia mechanisms to mitigate interference and stabilize policy evolution.
• Empirical studies demonstrate practical gains such as improved throughput in wireless MAC systems and collision-free VLC TDMA scheduling, elucidating trade-offs in convergence behavior.

Searching arXiv for recent and foundational papers on unsynchronized decentralized Q-learning and closely related variants. Unsynchronized decentralized Q-learning denotes a class of multi-agent reinforcement-learning dynamics in which agents update action-value estimates, and often policies, without a common global revision clock and without a centralized coordinator. Its central technical difficulty is non-stationarity: from any one agent’s viewpoint, the environment changes because other agents are simultaneously adapting. In the literature, this family appears in discounted stochastic games, Markov potential games, zero-sum Markov games, Dec-POMDPs, wireless MAC optimization, visible-light communication scheduling, and decentralized mean-field systems (Yongacoglu et al., 2023, Arslan et al., 2015, Maheshwari et al., 2022, Sayin et al., 2021, Makvandi et al., 2023, Subramanian et al., 2021).

1. Scope, formulations, and information structure

The term unsynchronized refers to the absence of a shared policy-revision timetable or global update barrier. In the stochastic-game formulation, each agent $i$ chooses its own increasing sequence of exploration-phase times

$0 = t^i_0 < t^i_1 < t^i_2 < \cdots,$

so policy changes by other agents may occur inside agent $i$’s current phase (Yongacoglu et al., 2023). In the earlier decentralized stochastic-team and game framework, each agent likewise has private phase lengths $T_k^i$ and begins phase $k$ at time $t_k^i$, with no need for the schedules to line up across agents (Arslan et al., 2015). In alternating-update cooperative learning, agents act jointly in the environment but only the selected agent updates its Q-function at a given global step; the update schedule can be implemented by a cyclic or hashed assignment, or by each agent checking a local time mod $n$ rule (Su et al., 2022).

The term decentralized is equally context-dependent. In the local-action-learner model for discounted stochastic games, players observe the global state and their own costs, but not other players’ actions or models (Yongacoglu et al., 2023). In radically uncoupled zero-sum Markov games, each agent observes only the current state, its own action, its own immediate reward, and the next state; it does not observe the opponent’s actions or payoffs and may even be unaware of the zero-sum structure (Sayin et al., 2021). In wireless resource allocation, the formulation can be fully stateless: each wireless network learns only from experienced throughput, with no observation of neighboring actions, no global state, and no state exchange (Wilhelmi et al., 2017). In Dec-RL for IRSA MAC optimization, each node observes only its own buffer history $H_t^i$, not other nodes’ buffers or the uncontrolled load $G$ (Nisioti et al., 2018). In decentralized mean-field games, each agent uses a local mean-field estimate $\mu_t^i$ rather than the true population distribution $0 = t^i_0 < t^i_1 < t^i_2 < \cdots,$0 (Subramanian et al., 2021).

These formulations therefore share an architectural property rather than a single observation model: learning is local, coordination is absent or minimal, and the resulting environment is non-stationary from each agent’s perspective.

2. Canonical update rules and asynchronous scheduling patterns

A common computational core is the one-step temporal-difference Q-update

$0 = t^i_0 < t^i_1 < t^i_2 < \cdots,$1

which appears in tabular VLC TDMA learning, stateless wireless resource allocation, cooperative alternate Q-learning, decentralized mean-field Q-learning, and related variants (Makvandi et al., 2023, Wilhelmi et al., 2017, Su et al., 2022, Subramanian et al., 2021). In cost-minimization stochastic games, the max operator is replaced by a min operator, yielding the constant-step-size update

$0 = t^i_0 < t^i_1 < t^i_2 < \cdots,$2

for the visited component (Yongacoglu et al., 2023).

The major distinction across unsynchronized decentralized methods lies not in the local TD step, but in how policy evolution is slowed, structured, or regularized.

In asynchronous decentralized Q-learning for stochastic games, each agent fixes a baseline policy $0 = t^i_0 < t^i_1 < t^i_2 < \cdots,$3 during its own phase, mixes exploitation with uniform exploration of probability $0 = t^i_0 < t^i_1 < t^i_2 < \cdots,$4, and revises the baseline only at $0 = t^i_0 < t^i_1 < t^i_2 < \cdots,$5. Policy revision is further regularized by inertia: with probability $0 = t^i_0 < t^i_1 < t^i_2 < \cdots,$6, the agent keeps its current baseline policy; otherwise it switches to a $0 = t^i_0 < t^i_1 < t^i_2 < \cdots,$7-greedy policy with respect to its current $0 = t^i_0 < t^i_1 < t^i_2 < \cdots,$8 (Yongacoglu et al., 2023). The earlier Arslan–Yüksel algorithm has the same broad logic: small exploration inside a phase, near-best-reply computation at phase boundaries, and inertial switching across phases (Arslan et al., 2015).

In Markov potential games, the scheduling is actor-critic-like. Q-estimates $0 = t^i_0 < t^i_1 < t^i_2 < \cdots,$9 are updated only on the visited component using a faster stepsize $i$0, while mixed policies $i$1 are updated more slowly through a best-response direction using $i$2, with $i$3 (Maheshwari et al., 2022). In zero-sum Markov games, the fast process is the local Q-estimate $i$4, while the slow process is the value estimate $i$5; action selection uses a smoothed best response, such as softmax under entropy regularization (Sayin et al., 2021).

Other variants impose asynchrony through turn-taking rather than phase persistence. MA2QL updates only one agent’s Q-function when that agent’s local turn arrives, while all agents continue to act in the environment (Su et al., 2022). MacDec-MADDRQN and Parallel-MacDec-MADDRQN handle asynchronous macro-actions of variable duration: each robot independently executes a macro-action until termination, emits a macro-observation $i$6, and stores decentralized tuples $i$7, while centralized joint tuples are aligned at the earliest macro-action termination $i$8 (Xiao et al., 2019).

A representative cross-section is summarized below.

Variant	Distinguishing mechanism	Reported setting
Asynchronous decentralized Q-learning	Independent exploration phases, inertia, constant $i$9	Discounted stochastic games (Yongacoglu et al., 2023)
MA2QL	Alternating agent turns, $T_k^i$0 local Q-updates per turn	Fully decentralized cooperative MARL (Su et al., 2022)
QD-learning	Consensus + innovation over a sparse graph	Distributed multi-agent MDPs (Kar et al., 2012)
VLC decentralized TDMA	Slot selection by local Q-learning after one synchronization frame	VLC IoT networks (Makvandi et al., 2023)

The significance of these scheduling choices is that they create a separation between rapid value adaptation and slower policy movement, or otherwise reduce simultaneous interference among learners.

3. Convergence mechanisms and equilibrium guarantees

The most explicit unsynchronized analysis is the two-timescale “persistence” framework. Its central claim is that constant learning rates are critical for relaxing synchronization assumptions: with diminishing $T_k^i$1, old data are overweighted and Q-estimates can stick to outdated environments, whereas constant $T_k^i$2 geometrically discounts stale data and enables rapid relearning when others shift policies (Yongacoglu et al., 2023). Under finite discounted stochastic games, nonempty stationary deterministic equilibria $T_k^i$3, weak acyclicity, sufficiently small $T_k^i$4, and sufficiently long phase lengths $T_k^i$5, the paper states that for any $T_k^i$6,

$T_k^i$7

so the joint baseline policy stays in a deterministic equilibrium with high probability (Yongacoglu et al., 2023).

The earlier decentralized Q-learning theory for stochastic teams and games establishes almost-sure convergence under weak acyclicity under strict best replies, small exploration, tolerances below the minimum Q-value gaps, and sufficiently long phases. The proof strategy embeds the decentralized learning dynamics into a best-reply process with inertia once within-phase Q-estimates are sufficiently accurate (Arslan et al., 2015). This is a foundational formulation of decentralized equilibrium learning in stochastic dynamic games.

For Markov potential games, the convergence route is different. Under irreducibility and full-support conditions, bounded payoffs, potential structure, decreasing stepsizes with $T_k^i$8, square-summability conditions, and $T_k^i$9, the fast-timescale Q-estimates converge almost surely to the true one-step-deviation Q-functions $k$0, and the slow-timescale policy process converges almost surely to the set of stationary Nash equilibria. The potential $k$1 acts as a strict Lyapunov function for the policy differential inclusion (Maheshwari et al., 2022).

In zero-sum Markov games, convergence depends on the temperature schedule $k$2. Under appropriate asynchronous stepsizes and connectivity assumptions, Regime A with $k$3 yields convergence to an $k$4-Nash neighborhood, while Regime B with $k$5 yields exact Nash convergence. The same dynamics is also rational in the sense that if the opponent’s strategy converges to a fixed interior policy, the learner converges to the corresponding best-response value (Sayin et al., 2021).

Other guarantees rely on different structural devices. QD-learning combines local TD innovations with inter-agent consensus over a sparse and possibly stochastic communication graph. Under weak connectivity on average, bounded moments of local costs, and mixed timescales with $k$6, each agent’s table $k$7 converges almost surely to the centralized optimum $k$8 (Kar et al., 2012). Best Possible Q-Learning replaces the ordinary Bellman operator with a “best possible” operator that optimizes over deterministic collapse kernels. The exact operator is a $k$9-contraction, and the simplified monotonic operator converges almost surely to $t_k^i$0; the resulting greedy decentralized policy is optimal for the original centralized cooperative MDP (Jiang et al., 2023).

Taken together, these analyses show that unsynchronized decentralized Q-learning is not associated with a single theorem. Convergence is obtained by imposing persistence, two-timescale separation, potential structure, zero-sum geometry, weak acyclicity, consensus, or operator-level monotonicity.

4. Representative instantiations across communication and control systems

A particularly concrete instance is decentralized TDMA in VLC IoT networks. Time is partitioned into transmission frames of 16 equal slots indexed $t_k^i$1, followed by a reserved finish-up interval; in the hardware experiments, frame duration is $t_k^i$2 and slot duration is approximately $t_k^i$3. There is no global clock or central coordinator. When a new node enters, it broadcasts a fixed synchronization frame, and all receiving nodes reset their local frame timers to that instant; no further control messages or beaconing occur (Makvandi et al., 2023). Each node has three states $t_k^i$4, 16 actions corresponding to slot indices, rewards $t_k^i$5, $t_k^i$6, and $t_k^i$7 for success, collision with retries left, and packet drop after retry exhaustion, and uses $t_k^i$8, $t_k^i$9, and $n$0-greedy exploration. Convergence is declared if a slot choice occurs at least $n$1 times in the past $n$2 iterations; after convergence, each node commits to the slot forever (Makvandi et al., 2023).

In wireless resource allocation without shared state, each wireless network chooses from $n$3 actions, where $n$4 channels and transmit power lies in $n$5. The reward is normalized throughput

$n$6

where $n$7 and $n$8. The Q-table collapses to a vector $n$9 because the problem is stateless, and each agent updates asynchronously in a random order without synchronization barriers (Wilhelmi et al., 2017).

For decentralized MAC optimization in IRSA, the problem is cast as a Dec-POMDP in which each agent observes only buffer occupancy and maintains a history $H_t^i$0 of length $H_t^i$1. The action is the number of packet replicas $H_t^i$2, the discount factor is set to $H_t^i$3, and the learning rate is polynomially decaying as $H_t^i$4. A notable acceleration device is virtual experience: histories with the same transformed difference process $H_t^i$5 are batch-updated because the collision-resolution dynamics depend on recent differences $H_t^i$6, not on absolute buffer levels (Nisioti et al., 2018).

In large-scale decentralized mean-field learning, each agent updates

$H_t^i$7

while also refining a local mean-field estimate $H_t^i$8. No centralized clock or coordination is required (Subramanian et al., 2021).

These instantiations differ substantially in state representation, action structure, and observability, but each uses local Q-learning to extract stable behavior from decentralized interaction.

5. Empirical behavior, performance, and observed trade-offs

In the VLC TDMA system, the empirical average reward for $H_t^i$9 nodes and 16 slots rises from negative values to a plateau at $G$0 to $G$1 around iteration 220, corresponding to a convergence time of approximately $G$2. Under 180-byte packets, the learned TDMA schedule yields up to $G$3 higher goodput and up to $G$4 lower average delay than IEEE 802.15.7 CSMA/CA under identical load, and the post-convergence schedule is collision-free (Makvandi et al., 2023).

The stateless wireless resource-allocation study exposes a different empirical regime. Aggregate throughput improves to approximately $G$5 of the centralized optimum, with the best reported average aggregate throughput approximately $G$6, compared with a centralized optimum of $G$7, at $G$8. Yet individual network throughputs oscillate heavily over time, and the paper explicitly attributes the variability to the adversarial setting in which the most played actions provide alternating good and poor performance depending on neighboring decisions. It further notes that the standard theoretical convergence guarantees of Q-learning, which require a fixed stochastic environment, do not hold in this setting (Wilhelmi et al., 2017).

The IRSA Dec-RL study reports that the decentralized algorithm matches baseline IRSA for $G$9 and outperforms it for $\mu_t^i$0 by up to $\mu_t^i$1–$\mu_t^i$2. The baseline exhibits a classical waterfall drop near $\mu_t^i$3, whereas Dec-RL IRSA degrades more gracefully. Virtual experience reduces the optimal number of learning iterations from $\mu_t^i$4 to approximately $\mu_t^i$5, an approximately $\mu_t^i$6 reduction in $\mu_t^i$7-convergence time for $\mu_t^i$8 (Nisioti et al., 2018).

In cooperative MARL, MA2QL converges in approximately 100 steps on a tabular cooperative stochastic game, whereas IQL requires approximately 2000 steps. The same paper reports better asymptotic and stability properties in Multi-Agent Particle Environments, consistent gains in Multi-Agent MuJoCo, and improved win rates on harder SMAC maps (Su et al., 2022). BQL reports convergence to $\mu_t^i$9 of the optimal return on random infinite-horizon stochastic grid-games and stronger performance than IQL, HIQL, I2Q, and MA2QL on MPE, Multi-Agent MuJoCo, SMAC, and Google Research Football (Jiang et al., 2023). In decentralized mean-field games, DMFG-QL and DMFG-AC are reported to outperform IL, MFQ, and MFAC in mixed and competitive tasks, while in a ride-sharing problem DMFG-QL yields service-rate gains of $0 = t^i_0 < t^i_1 < t^i_2 < \cdots,$00–$0 = t^i_0 < t^i_1 < t^i_2 < \cdots,$01 relative to centralized NeurADP or optimization across varying fleet sizes, capacities, and wait-time constraints (Subramanian et al., 2021). In macro-action robot learning, real robots executing decentralized policies learned via Parallel-MacDec-MADDRQN achieve the correct ordered delivery in approximately $0 = t^i_0 < t^i_1 < t^i_2 < \cdots,$02 of trials, with end-to-end task times within $0 = t^i_0 < t^i_1 < t^i_2 < \cdots,$03 of simulated execution times (Xiao et al., 2019).

The empirical record therefore contains both stable scheduling outcomes and persistent oscillatory regimes. The contrast is not accidental: the studies differ in whether the underlying interaction admits a collision-free schedule, a weakly acyclic best-reply path, a potential function, a zero-sum equilibrium structure, or instead a persistently adversarial moving-payoff landscape.

6. Conceptual boundaries, misconceptions, and open directions

A frequent misconception is that unsynchronized means the absence of any temporal organization. In fact, many successful formulations rely on explicit local structuring: exploration phases $0 = t^i_0 < t^i_1 < t^i_2 < \cdots,$04, long inter-update intervals $0 = t^i_0 < t^i_1 < t^i_2 < \cdots,$05, inertia parameters $0 = t^i_0 < t^i_1 < t^i_2 < \cdots,$06, actor-critic timescale separation, or turn assignments $0 = t^i_0 < t^i_1 < t^i_2 < \cdots,$07 (Yongacoglu et al., 2023, Maheshwari et al., 2022, Su et al., 2022). What is removed is the requirement of a shared global revision schedule, not the use of local clocks or sparse revision rules.

A second misconception is that decentralized always means no communication whatsoever. The communication assumptions vary sharply. QD-learning requires exchange of Q-values over a sparse communication graph and assumes weak connectivity on average (Kar et al., 2012). The VLC TDMA system has no coordinator and no ongoing beaconing, yet it still uses one synchronization frame when a new node enters (Makvandi et al., 2023). MacDec-MADDRQN learns decentralized execution policies, but during training each decentralized Q-net uses a centralized joint Q-net for target construction (Xiao et al., 2019). By contrast, radically uncoupled zero-sum learning and stateless wireless learning are explicitly designed so that agents do not observe opponents’ actions or payoffs (Sayin et al., 2021, Wilhelmi et al., 2017).

A third misconception is that unsynchronized decentralized Q-learning has a universal convergence guarantee. The literature instead provides guarantees under distinct structural assumptions: weak acyclicity in stochastic games (Arslan et al., 2015), persistence with constant stepsizes and long phases in discounted stochastic games (Yongacoglu et al., 2023), potential structure in Markov potential games (Maheshwari et al., 2022), zero-sum structure with smoothed best responses (Sayin et al., 2021), consensus connectivity in distributed MDPs (Kar et al., 2012), or operator-level monotonicity in cooperative MDPs (Jiang et al., 2023). Where these structures are absent, empirical gains may still occur, but stable convergence need not (Wilhelmi et al., 2017).

This suggests several research directions. A plausible implication is that extending theory beyond weakly acyclic, potential, zero-sum, or monotone-operator settings remains central. A plausible implication is also that the gap between tabular or stylized guarantees and neural, replay-based, macro-action, or partially observed implementations remains significant, because strong theory is often attached to compact discrete models while large-scale empirical success is reported in function-approximation settings (Su et al., 2022, Jiang et al., 2023, Xiao et al., 2019). More broadly, the existing record indicates that unsynchronized decentralized Q-learning is best understood not as one algorithm, but as a design principle for stabilizing local value learning under decentralized adaptation.