Learning to Contest: Decentralized Robust Fairness in Cooperative MARL via Cross-Attention

Abstract: Fair cooperative multi-agent RL (MARL) teams maximizing egalitarian welfare are exploitable: a single selfish agent free-rides on the surplus fair agents forgo to raise the worst-off. A centralized need-based allocator removes it, but only by taking allocation out of agents' hands; whether decentralized policies can be robust was left open. We show this futility is an artifact of all-or-nothing contention. Under graded contention (a contested resource delivers $1-c$, wasting $c$), we prove that for any $c<1$ a worst-off cooperator that contests a free-rider strictly improves on yielding, so decentralized leverage exists (Prop. 1). Realizing it is a coordination problem under uncertainty: the number of free-riders is unknown and variable, so any fixed rule is dominated. We introduce CAN, a permutation-equivariant cross-attention policy over agents' observed behaviour that infers the number of free-riders and responds proportionally: turn-taking when none, contesting just enough when some. Trained against an adversarial league (PSRO), CAN keeps best-response exploitability low ($ρ\approx1.2$-$1.5$, vs. $ρ=N$ unprotected) across the contention range, wasting almost nothing at $D=0$ (efficiency $\approx1.0$) and retaining most of it at $D\geq1$ (efficiency 0.83-0.96), approaching the centralized oracle on both axes, no central allocator. Fair-MARL learners fail on complementary axes (GGF/FEN yield and are exploitable, SOTO all-contests and wastes), while CAN is both. On two further games we find clear scope, not blanket generality: CAN stays efficient and Pareto-dominates the fair learners, but its robustness holds only in proportion to the contest leverage: strong on a multi-server game, partial when it weakens, absent under winner-take-all (Prop. 1 fails). We also report its fragilities: weak leverage and zero-shot transfer to larger teams degrade it at high contention.

• The paper introduces CAN, a permutation-equivariant cross-attention policy trained against learned exploiters to help cooperative agents infer when to contest free-riders without centralized control.
• The method achieves 0.98 efficiency with no defectors and a best-response free-ride factor of 1.2–1.5, outperforming GGF, FEN, and wasteful all-contest strategies across contention levels.
• The results show robustness depends on contest leverage: CAN generalizes across several environments when defectors can be denied a positive share, but degrades under weak leverage and at larger team sizes.

Motivation and problem statement

Fair cooperative multi-agent reinforcement learning (MARL) methods such as FEN and SOTO maximize Schur-concave welfare objectives—typically the Generalized Gini Welfare (GGF)—over per-agent returns, so that no agent is starved. These methods implicitly assume that every agent cooperates in being fair. When one agent is self-interested, the fair team is exploitable: welfare-oriented agents sacrifice their own utility to raise the worst-off, and a defector free-rides on exactly that sacrifice. Prior work formalized this vulnerability under strictly rivalrous (all-or-nothing) contention, where a cooperator that contests a free-rider merely causes a collision and gains nothing; the proposed remedy was a centralized need-based allocator, which removes the exploit by taking allocation out of the agents' hands. Whether decentralized policies could be robust was left open. This paper answers that question affirmatively, with explicitly stated caveats.

Graded contention restores decentralized leverage

The paper's first contribution is a game-theoretic observation: the futility of policy-level defense is an artifact of all-or-nothing contention. The authors introduce a graded-contention game in which $N$ agents act over $T$ steps, each choosing Claim or Yield on a unit resource. A sole claimer wins the unit; $m \ge 2$ claimers split a fraction $1-c$ (wasting $c$); if nobody claims, the resource routes losslessly to the worst-off agent. The parameter $c$ interpolates from lossless sharing ($c=0$) to the all-or-nothing model ($c=1$).

Proposition 1 proves that for any $c < 1$, a worst-off cooperator who contests a lone claiming free-rider receives $(1-c)/2 > 0$ instead of $T$0, while the free-rider's take falls from $T$1 to $T$2. Contesting therefore strictly dominates yielding for the worst-off agent, and it is also objective-aligned: it raises the cooperator mean and lowers their spread under the training welfare $T$3. All-or-nothing contention is recovered as the limiting case $T$4. This result matters because it converts an impossibility into a bounded condition: decentralized leverage exists precisely when the contention rule leaves a contester a positive share.

Leverage alone is not a policy. Because the number of free-riders $T$5 is unknown and varies per episode (including $T$6), any fixed rule fails: always-contest pays the waste $T$7 even when nobody defects, while always-yield collapses when someone does. The residual problem is inference-and-coordination—estimate $T$8 from observed behaviour and contest just enough.

Method: cross-attention networks trained against a league

CAN (Cross-Attention Networks) addresses this with a shared, permutation-equivariant single-head self-attention block over per-agent tokens of six publicly observable features: normalized utility, deviation from team mean, gap to the minimum, a worst-off indicator, running claim-rate, and time progress. Each agent concatenates its token with its attention context and outputs Claim/Yield logits. Permutation-equivariance makes the policy both fully decentralized and $T$9-agnostic, enabling zero-shot evaluation at unseen team sizes. Cooperators are trained with REINFORCE to maximize the masked mean-minus-std welfare over cooperators, with the per-episode defector count drawn from $m \ge 2$0.

Three adversarial training schemes are compared: single co-training, population co-training ($m \ge 2$1 co-evolving defectors), and league training via PSRO, which alternates retraining cooperators against the frozen pool of all past best-response exploiters and adding fresh ones. Exploitation is measured by a bounded free-ride factor $m \ge 2$2, where $m \ge 2$3 is exactly fair and $m \ge 2$4 means a lone defector takes everything; every trained policy is audited by a freshly trained best-response defector, a stronger test than a fixed always-claim adversary.

Head-to-head results: only CAN occupies the good corner

On the graded game ($m \ge 2$5, $m \ge 2$6, 5 seeds), the paper reports a clean two-axis separation between efficiency at $m \ge 2$7 and best-response robustness:

Method	$m \ge 2$8 efficiency	Best-response $m \ge 2$9	Outcome
GGF	1.00	6.0 (= $1-c$0)	efficient, exploitable
FEN	0.68–0.98	3.0–6.0	unstable, mostly exploitable
SOTO	$1-c$1	1.0	robust, wasteful
CAN (league)	0.98	1.2–1.5	both
Centralized oracle	1.00	1.0	upper bound

Every fair learner collapses onto one of two scripted extremes. GGF learns to yield—efficient but maximally exploitable, with a lone defector taking everything at every $1-c$2. SOTO learns to all-contest—robust but wasteful, delivering exactly $1-c$3 (down to $1-c$4 at $1-c$5) because its annealing schedule settles in a fair-by-symmetry but resource-destroying equilibrium. FEN is seed-unstable, landing in either corner. CAN is the only method attaining both axes simultaneously, clustered beside the centralized oracle. Notably, its robustness is essentially free: with a best-response defector present, league-trained CAN keeps $1-c$6 efficiency at $1-c$7–$1-c$8 across the contention range—at $1-c$9 it delivers $c$0 versus the all-contest equilibrium's $c$1—because it contests selectively, mostly via turn-taking rather than collisions.

Robustness across contention and stress tests

League training keeps $c$2 low across the entire contention range ($c$3–$c$4, pooled over independent gen-6 runs), versus $c$5–$c$6 for vanilla co-training against a static adversary. Counterintuitively, CAN is most robust at the highest contention ($c$7 at $c$8), consistent with Proposition 1: higher waste makes a single contest deny the free-rider more. A stress test with $c$9 and coordinated defector coalitions shows larger coalitions are easier ($c$0 for $c$1); the worst case remains a lone free-rider ($c$2–$c$3).

Two ablations isolate what carries the result. First, the choice among adversarial schemes (single, population, league) is statistically indistinguishable—all land near $c$4–$c$5—so the headline robustness reflects training against a learned, adapting exploiter at all, not the league specifically. Second, an architecture ablation shows pooling is not enough: mean-pool and deep-sets aggregators are markedly more exploitable ($c$6 vs. $c$7). The bidirectional GRU is competitive on the mean ($c$8) and better at low $c$9, so the authors credit behaviour-conditioning plus adaptive aggregation rather than attention per se; cross-attention earns its place through stability at high contention ($c=0$0 at $c=0$1 vs. the GRU's erratic $c=0$2). They also note honestly that under a lighter training budget the CAN–GRU ranking reverses, underscoring that ablations must use the deployed budget.

Scope: leverage governs generality

Across two further environments—a multi-server congestion game and a stochastic-value "stakes" game—the efficiency axis carries over uniformly: CAN maintains $c=0$3 efficiency of $c=0$4–$c=0$5 everywhere and Pareto-dominates GGF and FEN throughout, while SOTO's robustness costs catastrophic efficiency ($c=0$6–$c=0$7, or $c=0$8 at stakes $c=0$9). Robustness, however, tracks how much leverage Proposition 1 actually delivers:

• Strong leverage (base): robust at every $c=1$0, most robust at $c=1$1.
• Capped leverage (congestion): robust, but bounded near the structural ceiling $c=1$2.
• Weakening leverage (stakes): partial—contesting a jackpot at $c=1$3 returns only $c=1$4, degrading to $c=1$5 while still staying efficient ($c=1$6).
• Absent leverage (Matthew rule): the richest claimer wins, a worst-off contester captures nothing, Proposition 1 fails by construction, and CAN is exploited ($c=1$7).

The method works as far as the leverage it is built on, and no further.

Limitations

The paper is explicit about fragility. Zero-shot transfer of the $c=1$8 policy to teams of 12 and 24 holds at low contention ($c=1$9 at $c < 1$0, $c < 1$1) but degrades sharply at high contention ($c < 1$2 at $c < 1$3, $c < 1$4); a size curriculum does not fix this—it helps only at $c < 1$5 and leaves the high-contention gap unchanged, apparently by diluting per-size training budget. Single-adversary co-training is seed-sensitive at low contention, requiring league training for consistency at extra cost. The environments are controlled abstractions with a single shared cooperator policy; spatial and continuous settings remain untested. Finally, the centralized oracle remains an upper bound that CAN approaches but does not reach.

Conclusion

This paper reframes decentralized robust fairness in cooperative MARL as a tractable, measurable target bounded by a clear condition. Graded contention makes contesting a free-rider strictly worthwhile for the worst-off agent (Proposition 1), and a permutation-equivariant cross-attention policy trained against learned exploiters converts that leverage into practice: low best-response exploitability ($c < 1$6–$c < 1$7) across the contention range, near-oracle efficiency when no free-rider is present, and retained efficiency even under attack—all without a central allocator. The authors' central normative claim is that fair-MARL methods should be reported not only on the fairness they achieve under full cooperation but on the fairness they retain under self-interest. The scope is stated plainly rather than overstated: efficiency generalizes broadly, robustness holds in proportion to the contest leverage a game provides, and crowd-scale robustness at high waste remains open.