ConcaveQ: Concave Mixer for Multi-Agent RL

• ConcaveQ is a deep multi-agent reinforcement learning framework that leverages a concave neural mixer for non-monotonic value function factorization.
• It employs an iterative coordinate ascent algorithm and an input-concave network architecture to effectively capture complex inter-agent dependencies.
• Empirical evaluations in predator–prey and StarCraft II settings show that ConcaveQ achieves faster convergence and higher win rates compared to monotonic baselines.

ConcaveQ is a non-monotonic value function factorization framework for deep multi-agent reinforcement learning (MARL), formulated to address the representational limitations inherent in monotonic value function decomposition. By parameterizing the mixing function as a neural network that is concave (but not monotonic) in its per-agent utilities, ConcaveQ achieves greater expressivity and facilitates efficient action selection in cooperative multi-agent tasks. Empirical evaluation demonstrates that ConcaveQ consistently outperforms state-of-the-art monotonic and mixed-monotonic baselines in challenging coordination domains, including multi-agent predator-prey and StarCraft II micromanagement (Li et al., 2023).

1. Theoretical Foundations and Motivation

Multi-agent value function factorization is central to scalable MARL, enabling the decomposition of a joint action-value function (joint Q) into per-agent utilities aggregated by a mixing function. Classical approaches, such as QMIX, enforce a monotonicity constraint on the mixing function, which guarantees the Individual-Global-Maximum (IGM) property—that agents’ decentralized greedy actions align with the global optimum. However, this monotonicity severely restricts the representational flexibility of the value factorization, rendering monotonic approaches incapable of capturing non-monotonic inter-agent dependencies commonly present in tightly coupled environments.

ConcaveQ relaxes the monotonicity constraint. The key observation is that concave (but non-monotonic) mixing functions retain several desirable properties: they permit efficient maximization of the joint action-value via coordinate ascent and guarantee a unique global maximizer in the joint action space. This approach enables the representation of a much richer set of inter-agent cooperation patterns, while still supporting effective decentralized policies.

2. Formal Model and Concave Mixer Architecture

For MARL environments with $n$ agents, let $Q_i(\tau_i, a_i)$ denote agent $i$’s local action-value, based on its action-observation history $\tau_i$ and action $a_i$. The centralized joint action-value is defined as:

$$Q_{\rm tot}(\boldsymbol\tau, \boldsymbol a) \equiv f_{\rm mix}(Q_1(\tau_1, a_1), \ldots, Q_n(\tau_n, a_n)),$$

with $\boldsymbol\tau = (\tau_1, \ldots, \tau_n)$ and $\boldsymbol a = (a_1, \ldots, a_n)$. ConcaveQ parameterizes $f_{\rm mix}$ as a $k$-layer ($Q_i(\tau_i, a_i)$0 in the default implementation) input-concave network:

$\boldsymbol a = (a_1, \ldots, a_n)$6

Key architectural constraints to ensure concavity (Theorem 3.3 in (Li et al., 2023)):

• All weight matrices $Q_i(\tau_i, a_i)$1 must be elementwise nonnegative.
• Each activation $Q_i(\tau_i, a_i)$2 must be convex and nondecreasing (e.g., ReLU).

Given these constraints, the network’s final output $Q_i(\tau_i, a_i)$3 is concave in $Q_i(\tau_i, a_i)$4 by induction: each $Q_i(\tau_i, a_i)$5 is convex in $Q_i(\tau_i, a_i)$6, and the final layer negates a convex function.

The table below summarizes the role of key components:

Component	Role	Constraints
Per-agent utility $Q_i(\tau_i, a_i)$7	Local action-value estimate	None (two-layer MLP with ReLU, typical)
Concave mixer $Q_i(\tau_i, a_i)$8	Aggregates $Q_i(\tau_i, a_i)$9 for $i$0	Concave, input-concave net structure
Auxiliary joint $i$1	Unrestricted joint action value	Standard feedforward net (no constraint)

3. Training Objective and Loss Functions

ConcaveQ employs a multi-term objective incorporating:

• A concave mixer $i$2,
• An auxiliary, unconstrained joint action value estimator $i$3,
• A factorized soft-actor-critic policy $i$4.

The total loss is given by:

$i$5

where:

• $i$6
• $i$7
• $i$8

where $i$9 if $\tau_i$0, else $\tau_i$1, as in WQMIX. $\tau_i$2 is the TD target, and $\tau_i$3 is obtained by maximizing $\tau_i$4 using the iterative coordinate-ascent scheme described below.

4. Iterative Joint Action Maximization

Due to its non-monotonic, concave $\tau_i$5, ConcaveQ cannot leverage decentralized greedy maximization of $\tau_i$6 for $\tau_i$7. Instead, an iterative coordinate-ascent algorithm is deployed during training:

1. Initialize $\tau_i$8 by greedy maximization of each $\tau_i$9.
2. For each agent $a_i$0: For each action $a_i$1:
   • Let $a_i$2 be $a_i$3 with agent $a_i$4’s action replaced by $a_i$5.
   • If $a_i$6, update $a_i$7, $a_i$8.
3. Return $a_i$9.

Concavity ensures that coordinate ascent converges to the unique global optimum in $$Q_{\rm tot}(\boldsymbol\tau, \boldsymbol a) \equiv f_{\rm mix}(Q_1(\tau_1, a_1), \ldots, Q_n(\tau_n, a_n)),$$0 steps, where $$Q_{\rm tot}(\boldsymbol\tau, \boldsymbol a) \equiv f_{\rm mix}(Q_1(\tau_1, a_1), \ldots, Q_n(\tau_n, a_n)),$$1 is the action set size (Li et al., 2023).

5. Algorithmic Workflow and Architectural Details

• Initialization: Networks for per-agent $$Q_{\rm tot}(\boldsymbol\tau, \boldsymbol a) \equiv f_{\rm mix}(Q_1(\tau_1, a_1), \ldots, Q_n(\tau_n, a_n)),$$2, concave mixer, unrestricted $$Q_{\rm tot}(\boldsymbol\tau, \boldsymbol a) \equiv f_{\rm mix}(Q_1(\tau_1, a_1), \ldots, Q_n(\tau_n, a_n)),$$3, and local policies $$Q_{\rm tot}(\boldsymbol\tau, \boldsymbol a) \equiv f_{\rm mix}(Q_1(\tau_1, a_1), \ldots, Q_n(\tau_n, a_n)),$$4 are initialized. Target networks and replay buffer $$Q_{\rm tot}(\boldsymbol\tau, \boldsymbol a) \equiv f_{\rm mix}(Q_1(\tau_1, a_1), \ldots, Q_n(\tau_n, a_n)),$$5 are set up.
• Centralized training: For each episode, agents act according to policy $$Q_{\rm tot}(\boldsymbol\tau, \boldsymbol a) \equiv f_{\rm mix}(Q_1(\tau_1, a_1), \ldots, Q_n(\tau_n, a_n)),$$6, transitions are stored in $$Q_{\rm tot}(\boldsymbol\tau, \boldsymbol a) \equiv f_{\rm mix}(Q_1(\tau_1, a_1), \ldots, Q_n(\tau_n, a_n)),$$7. Training steps sample mini-batches, perform iterative (joint) action maximization for target computation, and update networks per the multi-term loss.
• Decentralized execution: At test time, each agent executes actions greedily according to their local policy $$Q_{\rm tot}(\boldsymbol\tau, \boldsymbol a) \equiv f_{\rm mix}(Q_1(\tau_1, a_1), \ldots, Q_n(\tau_n, a_n)),$$8, requiring no centralized coordinator or mixing function.

Architecturally:

• Per-agent $$Q_{\rm tot}(\boldsymbol\tau, \boldsymbol a) \equiv f_{\rm mix}(Q_1(\tau_1, a_1), \ldots, Q_n(\tau_n, a_n)),$$9 networks are two-layer MLPs with ReLU (exact details not specified, but “as in QMIX”).
• The concave mixer is a 4-layer input-concave net with ReLU; weight matrices enforce nonnegativity except for the first, and a hypernetwork based on the full state $\boldsymbol\tau = (\tau_1, \ldots, \tau_n)$0 parameterizes weights, using absolute-value nonlinearity.
• Policy networks $\boldsymbol\tau = (\tau_1, \ldots, \tau_n)$1, while not specified in detail, can be implemented as two-layer MLPs with softmax output. An entropy parameter $\boldsymbol\tau = (\tau_1, \ldots, \tau_n)$2 is learned.
• Auxiliary joint $\boldsymbol\tau = (\tau_1, \ldots, \tau_n)$3 is a standard two-layer MLP.

6. Hyperparameters and Training Heuristics

Key hyperparameters (all explicitly specified):

• Learning rate: $\boldsymbol\tau = (\tau_1, \ldots, \tau_n)$4 (all networks)
• Batch size: 128
• Replay buffer size: 10,000
• $\boldsymbol\tau = (\tau_1, \ldots, \tau_n)$5-greedy exploration: annealed from 0.995 to 0.05 over 100,000 steps
• Target network update: every 200 episodes
• Soft-actor-critic temperature: initialized to $\boldsymbol\tau = (\tau_1, \ldots, \tau_n)$6, learning rate $\boldsymbol\tau = (\tau_1, \ldots, \tau_n)$7
• WQMIX-style weight: $\boldsymbol\tau = (\tau_1, \ldots, \tau_n)$8 if $\boldsymbol\tau = (\tau_1, \ldots, \tau_n)$9, else $\boldsymbol a = (a_1, \ldots, a_n)$0
• TD-$\boldsymbol a = (a_1, \ldots, a_n)$1: $\boldsymbol a = (a_1, \ldots, a_n)$2 (only if used)

No architecture-specific layer sizes for $\boldsymbol a = (a_1, \ldots, a_n)$3 or policy networks are specified.

7. Empirical Evaluation and Ablation

ConcaveQ is benchmarked in two classes of MARL environments:

• Predator–Prey (10×10 grid, 8 vs. 8 agents): With local 5×5 partial observation and variable penalty $\boldsymbol a = (a_1, \ldots, a_n)$4 for uncoordinated capture, ConcaveQ achieves parity or outperforms QMIX, WQMIX, QPLEX, RESQ, PAC, FOP, especially as $\boldsymbol a = (a_1, \ldots, a_n)$5 becomes more negative (requiring non-monotonic coordination).
• StarCraft II Micromanagement (SMAC): Evaluated on hard and super-hard maps at "Insane AI" difficulty (e.g., 3s_vs_5z, 5m_vs_6m, 27m_vs_30m, 6h_vs_8z, corridor, MMM2), ConcaveQ demonstrates faster convergence and higher final test win rates than all aforementioned monotonic and mixed-monotonic MARL methods. Gains are most pronounced on highly non-monotonic tasks (e.g., 6h_vs_8z).

Ablation experiments on 3s_vs_5z confirm the necessity of each innovation, with performance degraded by removing the concave mixer, iterative action selection, or soft policy network—removal of all leads to learning collapse (Li et al., 2023).

---

ConcaveQ introduces a principled, tractable refinement to value function factorization in deep MARL, enabling non-monotonic coordination through a concave neural mixer, and establishing new state-of-the-art empirical performance in benchmark cooperative tasks.