Independent Multi-Agent Q-Learning

• Independent Multi-Agent Q-Learning is a decentralized reinforcement learning paradigm where agents independently update their Q-functions without explicit coordination.
• This approach is applied in multi-agent systems such as pursuit-evasion tasks, emphasizing challenges like non-stationarity and phase transitions in policy stability.
• Variants like MA2QL and BQL improve convergence by mitigating kernel drift and enhancing synchronization, offering practical solutions for scalable decentralized learning.

Independent Multi-Agent Q-Learning (IMQL) is a foundational paradigm in multi-agent reinforcement learning (MARL) characterized by decentralized, local learning: each agent independently learns its own action-value function via Q-learning, making no explicit attempt to coordinate with other agents or to model their behavior. IMQL offers significant scalability and deployment simplicity but faces intrinsic challenges stemming from environment non-stationarity and the resulting instability in learned policies due to the simultaneous learning dynamics of interacting agents.

1. Core Algorithmic Structure

In IMQL, each agent $i$ maintains its own Q-function $Q_i(s, a_i)$ and treats the non-controlled agents as part of the environment. The Q-update for agent $i$ at time $t$ takes the form

$$Q_i(s_t, a^i_t) \leftarrow Q_i(s_t, a^i_t) + \alpha \left[ r_t + \gamma \max_{a'_i} Q_i(s_{t+1}, a'_i) - Q_i(s_t, a^i_t) \right]$$

where:

• $s_t$ is the observed (possibly global) state,
• $a^i_t$ is agent $i$'s action,
• $r_t$ is the reward (shared in fully cooperative scenarios),
• $\alpha$ is the learning rate,
• $Q_i(s, a_i)$0 is the discount factor.

Each agent's effective transition kernel is

$Q_i(s, a_i)$1

which evolves dynamically as others' policies $Q_i(s, a_i)$2 change.

IMQL is widely used as a minimalist baseline for intricate decentralized MARL dynamics, and provides the starting point for various improvements aimed at stabilizing learning and fostering coordination (Yamaguchi, 28 Nov 2025, Su et al., 2022, Luo et al., 2024).

2. Emergent Phase Structure and Instabilities

Systematic studies of IMQL reveal that its macroscopic behavior, specifically the emergence and collapse of coordination, exhibits phase-like structure as a function of environmental scale ($Q_i(s, a_i)$3) and agent density ($Q_i(s, a_i)$4) (Yamaguchi, 28 Nov 2025). The progression is captured by two summary statistics:

• Cooperative Success Rate (CSR): Fraction of episodes in which all agents succeed in the task.
• Stability Index ($Q_i(s, a_i)$5): Defined as the log-variance of agents' temporal-difference errors, reflecting update stability.

Sweeping ($Q_i(s, a_i)$6, $Q_i(s, a_i)$7) traces a phase map with three regimes:

1. Coordinated Phase: Low $Q_i(s, a_i)$8, small $Q_i(s, a_i)$9. High CSR ($i$0), low $i$1—stable collective behavior.
2. Fragile Transition (Instability Ridge): Intermediate parameter region. CSR fluctuates; $i$2 and kernel drift $i$3 peak, marking susceptibility to coordination collapse.
3. Jammed/Disordered Phase: High $i$4, large $i$5. CSR approaches zero, $i$6 falls but TD update noise dominates.

The central driver of instability is kernel drift: as agents update their policies independently, the effective dynamics faced by each agent change non-stationarily. Quantitatively,

$i$7

Kernel drift peaks sharply along a double Instability Ridge, demarcating the boundary between coordinated and fragile/jammed phases (Yamaguchi, 28 Nov 2025).

3. Synchronization and Temporal Alignment

Coordination stability in IMQL is contingent not only on environment structure but also on the synchronization of learning updates across agents. Define the synchronization index

$i$8

which captures temporal alignment of agents' behaviors. Coordination is sustained when $i$9. In the fragile regime, $t$0 leads to oscillatory breakdown and recovery; when $t$1, synchronization is lost irreversibly and the system transitions to the disordered phase. Removing agent identifiers (sourcing full symmetry) collapses kernel drift, erasing the three-phase structure and yielding flat, high CSR across all densities (Yamaguchi, 28 Nov 2025).

4. Practical Algorithmic Variants and Convergence

Several algorithmic modifications have been proposed to mitigate the non-stationarity challenge and recover stronger convergence guarantees:

• Alternate Q-Learning (MA2QL): Serializes agent updates such that only one agent updates its Q-function at a time while others' policies are fixed, restoring the stationary subproblem and establishing Nash equilibrium convergence in the tabular setting. MA2QL demonstrates empirical stability and outperforms simultaneous IMQL in stochastic, sparse-reward, and continuous control benchmarks (Su et al., 2022).
• Best Possible Q-Learning (BQL): Each agent maintains a "max-projected" Q-function over all possible transition kernels induced by other agents, provably converging to the globally optimal cooperative policy in fully decentralized settings. This is achieved via a monotone max-projection over sampled kernels, requiring no explicit model of other agents (Jiang et al., 2023).
• Generalized Individual Q-Learning: Interpolates between pure IMQL and belief-based (smoothed fictitious play) updates by incorporating partial observations of other agents' actions, enabling provable acceleration to quantal-response equilibria in polymatrix games (Donmez et al., 2024).

Convergence of IMQL under non-trivial coupling between agents is nontrivial: theory confirms finite-sample convergence to a ball around the global optimum, governed by the “dependence level” $t$2 of the true environment relative to its best separable (decoupled) approximation. Rate-optimal sample complexity $t$3 is achieved up to a gap $t$4 (Jin et al., 2024).

Variant	Key Property	Convergence Guarantee
IMQL	Parallel, fully independent updates	Approximate, gap scales with dependence $t$5 (Jin et al., 2024)
MA2QL	Cyclic, one-agent-at-a-time update	Nash eq. in tabular; stronger stability (Su et al., 2022)
BQL	Max over possible environment kernels	Global optimum in cooperative tasks (Jiang et al., 2023)

5. Empirical Regimes and Performance

Empirical studies reveal that IMQL is not universally suboptimal. In tabular cooperative pursuit-evasion games with explicit embodiment constraints (speed, stamina), fully independent Q-learning can outperform centralized alternatives, retaining adaptability critical for coordination under heterogeneous roles and constraints (Atif et al., 24 Jan 2026). However, IMQL tends to stall or converge to sub-optimal Nash equilibria in complex, tightly coupled environments or in the presence of sparsity in reward and severe credit assignment issues—phenomena mitigated by algorithmic interventions like MA2QL, BQL, and auxiliary critics (Yamaguchi, 28 Nov 2025, Su et al., 2022, Avalos et al., 2021).

6. Coordination, Credit Assignment, and Design Guidelines

IMQL exhibits a phase-transition-like behavior: emergent coordination in large-scale decentralized MARL is governed by the interplay between environment structure (scale, density), kernel drift, and synchronization. Interventions to foster coordination include:

• Minimizing kernel drift (e.g., reducing per-agent learning rates, sharing features)
• Enhancing update synchronization (policy swapping, shared schedules)
• Aggressively controlling agent density or effective coupling (task decomposition, spatial partitioning)

Conversely, to avoid coordination collapse, it is crucial to ensure $t$6 at all times (Yamaguchi, 28 Nov 2025). Algorithms like Local Advantage Networks (LAN) further leverage dueling architectures and lightweight central critics at training time to mitigate non-stationarity and stabilize advantage estimation, achieving high scalability and superior credit assignment (Avalos et al., 2021).

7. Limitations, Open Problems, and Future Directions

IMQL’s limitations stem from its neglect of interactive non-stationarity and lack of explicit coordination, resulting in performance gaps in tightly coupled settings. While practical variants (MA2QL, BQL) circumvent these issues under certain conditions, challenges remain:

• In environments with strong dependence between agents, irreducible optimality gaps persist.
• Extensions to partial observability, non-Markovian settings, and adversarial policy drift require further theoretical development (Jin et al., 2024, Donmez et al., 2024).
• The high-dimensional function approximation regime is not fully understood; empirical strategies (experience replay, target networks, dueling heads) are vital but not universally sufficient (Luo et al., 2024).

Current research suggests that IMQL, when interpreted through the lens of phase structure and distributional interaction, provides insights into emergent coordination and informs the design of robust, scalable MARL algorithms (Yamaguchi, 28 Nov 2025, Su et al., 2022, Jiang et al., 2023).