NeurIPS 2024 Concordia Contest

• The NeurIPS 2024 Concordia Contest is a simulation challenge that tests LLM-based agents in complex, text-mediated multi-agent scenarios such as negotiation, coordination, and norm enforcement.
• The contest employs the Concordia simulation environment, where agents interact with a central Game Master via free-form text to navigate partially-observable, mixed-motive social dilemmas.
• Empirical findings reveal substantial generalization deficits in current LLM-based agents, emphasizing the need for enhanced memory, theory-of-mind capabilities, and advanced persuasion modules.

The NeurIPS 2024 Concordia Contest focused on evaluating the generalization capabilities of LLM-based agents in zero-shot, mixed-motive multi-agent scenarios. Leveraging the Concordia simulation environment, the contest systematically assessed agents’ ability to achieve mutual gains across social dilemmas, negotiation, norm enforcement, coordination under uncertainty, and complex collective action—contexts characterized by partially observable information, multiple stakeholders, and conflicting interests. Unlike conventional benchmarks that focus on single-agent reasoning or static cooperation games, the Concordia Contest introduced dynamic, text-mediated interaction within a unified API, with empirical analysis highlighting the persistent challenges facing current LLM-based social agents (Smith et al., 3 Dec 2025, Vezhnevets et al., 2023).

1. Concordia Simulation Environment

Concordia is a natural language, multi-agent simulation platform in which agents and a central “Game Master” interact entirely via free-form textual descriptions. Each scenario is formalized as a Partially-Observable Stochastic Game without Rewards (POSG\R), where agents receive observations, submit intents, and receive state updates—without direct access to the underlying transition or observation dynamics. The “Game Master” resolves interactions according to latent rules, computes payoffs, and enforces hidden scenario constraints.

Five configurable scenario “substrates” were developed, each intended to capture distinct aspects of mixed-motive social interaction:

• Negotiation (e.g., haggling, state formation with constituency satisfaction)
• Coordination under uncertainty (e.g., pub selection with hidden closures)
• Persuasion and norm enforcement (e.g., Reality Show with alternating communication/action phases)
• Collective action (e.g., labor strikes balancing group and individual incentives)
• Social network dynamics

All observations, proposals, and commitments proceed via open-ended text, ensuring that agents must cope with ambiguity, non-standard dialog, and context-specific norms (Smith et al., 3 Dec 2025, Vezhnevets et al., 2023).

2. Contest Structure, Task, and Evaluation Protocol

Agents were evaluated in a zero-shot “veil of ignorance” protocol:

• During development, agents were exposed to public scenario generations and background strategies.
• In the evaluation phase, agents faced 14 held-out scenarios (seven in Resident mode, seven in Visitor mode) with previously unseen background agents and partner strategy mixes.
• Each agent played both self-play and cross-play matches.

Performance aggregation combined Elo ratings with voting-based methods (Iterative Maximal Lotteries, Copeland, Ranked Pairs) to mitigate noise and non-transitivity in outcomes.

The principal formal metrics include:

• Mutual Gain (MG):

$$\mathrm{MG} = \frac{R_{\rm agent} + R_{\rm partner}}{2}$$

where $R_{\rm agent}$ and $R_{\rm partner}$ denote final payoffs for the focal agent and coplayer.

• Relative performance gap over baseline $B$ (e.g. myopic/rational agent):

$G = \frac{\mathrm{MG} - B}{B}$

• Normalized scenario score:

$s(G,\pi) \in [0,1]$

via affine rescaling of raw returns between theoretical minimum and maximum for scenario $G = \langle E,\rho,m\rangle$ (substrate, partner strategy, mode).

Scenario parameters (player count, payoff matrices, network topology) were randomized per evaluation. Partners’ strategies encompassed naive altruism, stubborn defection, conditional cooperation, and retaliation (Smith et al., 3 Dec 2025).

3. Agent Design and Implementation

To enforce comparability, all agent teams deployed an instruction-tuned 9B parameter Gemma 2 LLM as the core model. Agent policies were structured as:

$\pi = \mathrm{LLM}(f(o))$

where $f$ is a scaffolding function that handles agent memory, logic, computations, and summarizes prior history for LLM input.

Architectural variations among the 25 finalist agents included:

• Negotiation modules:
  • “Theory-of-Mind” inference (e.g., Synthetic_tom) to estimate partner goals/payoff functions;
  • Concession scheduling driven by loss-aversion or dynamic target-setting.
• Persuasion modules:
  • Chain-of-Thought prompting for multi-step argumentation;
  • Memory tracking for reputation and commitment management.
• Norm enforcement modules:
  • Sanction suggestion and social threat detection in mixed-motive games;
  • “Veil-of-Ignorance” conformity mechanisms for Visitor mode.

Memory, action selection, and logic were scaffolded via modular “components” in the Concordia library. Agents stored long-term memory as a set of event strings and maintained working memory as a dynamic summary of history and internal state (Vezhnevets et al., 2023).

4. Empirical Findings and Analysis

Aggregate contest performance revealed significant generalization deficits for current LLM-based agents:

• The overall average normalized scenario score was:

$\overline{s} = 0.426 \pm 0.005 \quad(\mathrm{SE})$

• Scenario-specific averages varied:
  • Coordination-only settings (no explicit conflict): $s > 0.8$
  • Persuasion, norm enforcement, and complex negotiation: $s < 0.2$ (mean for state formation ≈ 0.30, with high variance)
• Hierarchical beta-regression demonstrated that requirements for complex persuasion, convention following, negotiation, or norm discouragement depressed expected performance by 10–20 points.

The evaluation phase Elo ranking of top agents was as follows:

Rank	Agent	Elo
1	in2ai_megamind	1588
2	fluffy_fluffyagent_v16	1577
3	SSCT_super_agent	1566
4	taehun_cgcal	1564
5	hgyun_loss_aversion_v3	1562

Cross-play confirmed taehun_cgcal as the overall winner by all aggregation methods.

Measurement Layout models decomposed agent abilities across latent factors (Persuasion, Coordination, Norm Adherence, Calculation, etc.), with persuasion accounting for the greatest inter-agent variance—the primary axis distinguishing high versus low Elo agents (Smith et al., 3 Dec 2025).

5. Capability Gaps, Failure Modes, and Representative Examples

Observed capability gaps included:

• Inability to construct multi-step persuasive arguments (Reality Show);
• Ineffective sanctioning to restore cooperation (Labor Collective Action);
• Failures in following majority conventions in Visitor mode— agents frequently persisted in idiosyncratic routines, depressing group payoff;
• Brittleness in complex negotiations, notably in State Formation;
• Repeatedly attempting infeasible coordination under hidden information (e.g., choosing closed pubs).

Representative generalization failures:

• Agents attempted to coordinate at a closed pub due to poor inference and memory utilization.
• Loss-aversion agent persisted in strikers, missing group wage maximization when strike thresholds had already been satisfied (Smith et al., 3 Dec 2025).

6. Methodological Framework and Benchmarks

The Concordia library provides a modular infrastructure for Generative Agent-Based Models (GABM) informed by LLMs and explicit associative memory. Contestants build agents via component stacks mediating memory, reasoning, and planning, interacting with a “Game Master” that simulates the environment. The framework supports the following abstractions (Vezhnevets et al., 2023):

• Agent abstraction: Long-term memory ($M$), working memory ($Z$), action sampling from $a_t \sim p(\cdot|f^a(Z_t))$.
• Game Master abstraction: Grounded variables ($V$), event sampling $(e_t \sim p(\cdot|f^e(Z_{t}^{GM}), a_t))$, state updates, and observation emission $(o_{t+1}^i)$.
• Action language: Agents produce free-form textual intents, which the GM interprets, verifies, and executes.
• Memory management: Explicit component-level scoring and memory retrieval by embedding similarity; optional penalization by age.

Benchmark scenarios included Digital Assistant Challenge, Social Dilemma, and Multi-Scale Economy, with metrics for task-completion, communication efficiency, and inter-agent coherence. Additional qualitative assessment (plausibility, narrative consistency) employed small-scale human annotation and contradiction checks.

7. Conclusions and Future Directions

Current best-in-class LLM-based agents demonstrate partial success at zero-shot negotiation and basic coordination but fall short in tasks requiring robust persuasion, dynamic norm enforcement, and flexible convention following. Persistent failure modes reveal the need for:

• Enhanced memory and richer Theory-of-Mind reasoning for persistent, cross-context belief tracking;
• Dedicated persuasion modules transcending raw LLM outputs (explicit multi-turn argument planning);
• Multi-modal grounding to integrate non-verbal contextual signals;
• Adaptive curricula that stress-test agents with procedurally generated partner variation and dynamic social dilemmas.

The NeurIPS 2024 Concordia Contest establishes a scalable, interpretable, and quantitatively rigorous methodology for measuring general cooperative intelligence in LLM agents, offering a reference suite for continuing advances in socially intelligent language agent research (Smith et al., 3 Dec 2025, Vezhnevets et al., 2023).