Agent Island: A Saturation- and Contamination-Resistant Benchmark from Multiagent Games

Abstract: Static capabilities benchmarks suffer from saturation and contamination, making it difficult to track capabilities progress over time. We introduce Agent Island, a multiplayer simulation environment in which language-model agents compete in a game of interagent cooperation, conflict, and persuasion. The environment yields a dynamic benchmark designed to mitigate both saturation and contamination; new models can always outperform the current leading player in this winner-take-all game, and agents compete against other adaptive agents rather than face a fixed task set. We rank players with a Bayesian Plackett-Luce model, allowing us to quantify uncertainty in player skill. In 999 games involving 49 unique models, openai/gpt-5.5 dominates its peers with a posterior mean skill of 5.64, compared with 3.10 for the second-ranked model, openai/gpt-5.2, and 2.86 for the third-ranked model, openai/gpt-5.3-codex. We release the game logs as a dataset for analyses of model behavior. As an example, we investigate same-provider preference in final-round votes and find that models are 8.3 p.p. more likely to support a same-provider finalist than finalists from other providers. This preference is not uniform across providers: among separately estimated providers, the effect is strongest for OpenAI models and weakest for Anthropic models.

• The paper introduces Agent Island, a dynamic benchmark that uses multiagent game structures to resist saturation and training-set contamination.
• It employs a Bayesian Plackett–Luce model to estimate latent skill parameters, demonstrating significant performance gaps among leading LLMs.
• The research uncovers emergent social biases, such as same-provider preference in jury voting, influencing model rankings and strategy.

Agent Island: A Dynamic and Robust Multiagent Benchmark for LLMs

Motivation and Benchmark Design

The stagnation of static AI benchmarks, due to both saturation and contamination, has become a significant challenge in LLM evaluation. Once a fixed benchmark is widely solved, observed progress decouples from actual capability improvements. Furthermore, as benchmarks mature, the risk that their tasks have been encountered during model training—rendering performance measurements unreliable—increases markedly. The "Agent Island" benchmark is introduced to address these concerns by creating a dynamic, multiagent game-based environment explicitly engineered to be resistant to both saturation and training-set contamination.

Agent Island draws inspiration from strategic competition formats, especially the "Survivor" game structure, in which agents must balance cooperation, persuasion, and elimination strategies across multiple rounds. Key to Agent Island's architecture is that all interactions and outcomes emerge from active competition between language-model agents, not from a static task set. This ensures continuous adversarial pressure and adaptation, as well as inherent novelty across iterations.

Game Structure and Data Generation

Each Agent Island simulation involves seven anonymized AI agents, sampled without replacement from a pool of 49 unique LLMs. Gameplay proceeds through five elimination rounds and a final jury round, implementing the following core phases in each round:

• Private Discussions: Each player selects another for a private sidebar of alternating messages, facilitating alliance formation and information exchange.
• Public Pitches: All players pitch their case to avoid elimination.
• Secret Ballots and Elimination: Private elimination votes are cast, with the lowest-voted player removed.
• Final Jury Voting: In the finale, the five eliminated agents constitute the jury that decides the victor among the last standing players.

Each run is thoroughly logged, producing a rich structured dataset: round-by-round events, private communication, voting rationales, and, where supported by provider APIs, internal model traces. The public dataset (999 games comprising 49 models) is curated and released for broader research use, with full specifications and replication artifacts provided for reproducibility.

Skill Estimation via Bayesian Plackett–Luce Scoring

To rank models within this winner-take-all, multiplayer context, a Bayesian Plackett–Luce model is used. Each model has an inferred latent skill parameter $\lambda_i$, estimated from the set of games it participates in and whether it wins. Given game set $\mathcal{L}$ and player set $\mathcal{I}_\ell$ for game $\ell$, the model assumes the win probability for agent $i$ as $$\lambda_i / \sum_{j \in \mathcal{I}_\ell} \lambda_j$$. Model skill posteriors are computed via a Gibbs sampler, yielding both means and credible intervals.

Figure 1: Posterior skill $\lambda_i$ per model, ranked by posterior mean. Thick lines are $50\%$ intervals; thin lines are $95\%$; prior mean at $\lambda = 1$.

The reward structure is highly nonlinear; only the winner receives a reward, making the environment strongly competitive and significantly slowing down skill identification relative to settings with more gradual reward distribution.

Empirical Results and Comparative Model Skill

Analysis of the 999-game dataset yields several robust findings. OpenAI's GPT-5.5 emerges as the dominant model, with a posterior mean skill of 5.64, and is separated by a large gap from both OpenAI GPT-5.2 (mean 3.10) and GPT-5.3-Codex (mean 2.86). The remainder of the model pool is more tightly clustered around the prior mean ($\mathcal{L}$0). Notably, matchups involving GPT-5.5 yield extremely high posterior epistemic confidence in its superiority.

Figure 2: Pairwise head-to-head statistics for the five focal models. Left: Cliff's $\mathcal{L}$1 shows posterior confidence in skill order; right: Plackett–Luce head-to-head win rate predicts empirical win frequencies.

• Cliff’s $\mathcal{L}$2 for GPT-5.5 vs. GPT-5.2 is +0.996, indicating near-certain posterior ordering.
• Head-to-head win rate for GPT-5.5 vs. GPT-5.2 is 0.644, further quantifying the advantage.
• At the lower end, models such as deepseek/deepseek-r1-0528 and moonshotai/kimi-k2.5 perform significantly worse, barely winning matchups against any of the top-3 models.

The skill distribution, as visualized, highlights the benchmark’s capacity to distinguish fine-grained differences across a diverse and cohort-sampled model set and resists both ceiling effects and simple memorization strategies.

Emergent Social Phenomena: Same-Provider Preference

Beyond traditional skill estimation, the Agent Island dataset enables the study of emergent social behaviors among LLMs. One notable finding is a statistically significant "same-provider preference" in jury-phase voting. When a finalist and a voting juror originate from the same model provider, the juror is, on average, 8.3 percentage points more likely to support their in-group peer, controlling for other finalist popularity effects (95% CI: [4.7, 11.8] p=0.000). However, this effect strongly varies by provider, being most pronounced for OpenAI and smallest for Anthropic.

Figure 3: Same-provider voting preference. Left: pooled provider effect (+8.3 pp); right: disaggregated by finalist provider, with OpenAI showing the strongest effect.

These findings challenge a core assumption in the skill scoring model: that agent skill is invariant to the composition of the player pool. Inter-agent bias—predicated simply on shared provider identity—offers evidence for more complex, context-sensitive skill distributions and player dependency effects.

Implications, Limitations, and Future Directions

Practically, Agent Island represents a substantial advance in competitive multiagent LLM benchmarking, offering ongoing adaptability and resistance to contamination and saturation. Its continuous, multiplayer format allows divergent models to emerge as dominant over time, and new algorithms can demonstrate improvement even after the current leaderplateaus. The availability of granular interaction logs is valuable for behavioral, sociotechnical, and game-theoretic analysis of LLMs.

Several limitations are acknowledged:

• The environment is low-stakes, lacking explicit external rewards beyond game victory, which may underrepresent competitive or deceptive behavior that might manifest in higher-stakes deployments.
• The current scoring system does not yet account for observed matchup effects, particularly those arising from same-provider bias.
• As models converge in ultimate skill, the environment may still eventually saturate via win-rate oscillations at the upper-ceiling.

Future research directions include explicit modeling of matchup effects, detection and defense against advanced tactics such as prompt injection, investigation of emergent coalition structures, and deeper analysis of provider- or model-specific play styles. Exploration of higher-stakes dynamics and extensions to environments involving real-world resources or decision authority are of particular interest for risk analysis and AGI safety research.

Conclusion

Agent Island operationalizes a robust dynamic benchmark for LLMs in a multiagent, adversarial context. Its structural features mitigate the central weaknesses of static benchmarks and introduce new paradigms for measuring emergent strategic, cooperative, and persuasive competencies in autonomous language agents. The results reveal large gaps in competitive performance among leading LLMs and illuminate sociotechnical dynamics such as in-group preference not previously observed in fully autonomous agent environments. The public release of both game logs and benchmarking infrastructure positions Agent Island as a key resource for ongoing research on AI evaluation, multiagent systems, and behavioral analysis in artificial agents.