- The paper introduces a progress-based evaluation framework that leverages per-turn game state and machine-learned estimators to assess strategic decision-making in Civilization V.
- It compares multiple LLM strategists across varied game configurations against a baseline rule-based system using metrics like ELO deltas and ROC-AUC up to 0.865.
- The framework elucidates nuanced behavioral differences, highlighting trade-offs in strategic specialization, adaptation, and agentic pivoting in multi-agent environments.
CivBench: A Progress-Based Framework for Evaluating LLM Strategic Decision-Making in Civilization V
Motivation and Context
Comprehensive evaluation of LLM-powered agents in open-ended, competitive, and longitudinal decision-making tasks remains unresolved. Existing benchmarks for LLM strategists either focus on isolated task execution, utilize highly simplified games, or reduce evaluation to coarse, terminal outcomes, failing to probe multi-agent, long-horizon, strategic optimization. Civilization V was selected due to its depth, strategic diversity, and the presence of quantifiable, multi-agent victory objectives, but terminal win/loss signals are too sparse for robust measurement. CivBench introduces a dense, progress-based evaluation methodology that extends the design space of agentic benchmarks.
CivBench Design: Dense Progress-Based Evaluation
CivBench builds on the open infrastructure Vox Deorum, which interfaces LLM strategists with the Civilization V engine while leveraging the Vox Populi mod's enhanced core AI for tactical execution. LLMs control macro-strategic dimensions (victory path, technology, policy adoption, flavor parameters, diplomatic posture), while delegation to VPAI ensures that the core focus remains on strategic, rather than tactical, reasoning—a key distinction recognizing LLMs' current limitations in sub-turn micromanagement.
The central algorithmic contribution is the construction of robust, machine-learned estimators that use per-turn game state to predict terminal victory probabilities. These are validated according to predictive (outcome accuracy, with ROC-AUC up to 0.865), construct (alignment with plausible strategic factors), and convergent (agreement across estimator classes) validity. The estimator’s outputs allow for turn-level assessment of within-game competitive standing, enabling not just aggregate skill ranking but the diagnosis of intermediate strategic trajectories.
Experimental Study: Benchmarking LLM Strategists
The empirical setup covers 307 games with 7 LLM classes and 12 matchup configurations. Two agentic setups—Simple (direct game state) and Briefed (strategic reports condensed by fixed, weaker LLM briefers)—probe the impact of observation mediation and LLM+pipeline interaction. A "Null" ablation (suppressed strategic logic, random choice) and VPAI-only baseline contextualize LLM performance.
Seven estimator models, including logistic regression, XGBoost, several MLP variants, and attention-based architectures, are evaluated for signal fidelity and generalization. The top performing estimator (AttentionMLP) maintains strong discriminative power across all victory conditions, not just Domination, directly addressing known deficiencies in using in-game score as a progress proxy.
Core Findings
1. Evaluation Signal Quality
Progress-based estimators substantively improve on terminal-score proxies, yielding denser, more behaviorally meaningful assessment of strategic progress. Model agreement is high for relative in-game standings (mean pairwise Spearman ρ ≈ 0.92). Feature analysis highlights that science, policy, city growth, score, and influence are decisive; direct military state variables contribute less value, confirming that their impact is largely absorbed by aggregated scores or is not adequately captured by turn-snapshot models.
2. LLM Capability Comparison
No LLM strategist consistently surpasses the expert-crafted VPAI rule-based system in aggregate ELO, but several match its performance under certain configurations and victory types. ELO deltas illustrate configuration- and model-specific effects: e.g., Kimi-K2.5 and Qwen-3.5 gain 67 and 75 points, respectively, under the Briefed setup (relative to Simple), whereas Sonnet-4.5 suffers a -99 drop. Briefing improves culture victory specialization generally, but effects on other strategic dimensions are nuanced and non-uniform.
3. Strategic Profiling: Beyond Aggregate Skill
CivBench’s logging of per-turn strategic decisions enables fine-grained profiling:
- Victory preferences: Strong model-induced tendencies, e.g., Sonnet-4.5 over-commits to science (up to 77.6% time allocation post-briefing), while others (Minimax, GPT-OSS-120B) display more dispersion.
- Commitment: Some LLMs hyper-commit (Sonnet-4.5 to science, Minimax to culture), often exceeding VPAI’s focus levels.
- Adaptation/Pivoting: Most LLMs pivot strategies reactively, not proactively—pivots cluster at low predicted win probability, while VPAI pivots frequently and is more responsive. There is a notable misalignment between the preferred and strength-maximizing strategies (e.g., Minimax-M2.5-Simple allocates heavily to culture but peaks in diplomatic ELO).
Implications
Practical
CivBench establishes an unsaturated, longitudinal, competitive benchmark suitable for evaluating large-scale, agentic LLM systems. By generating dense, per-turn capability signals, it reduces experimental sample complexity and supports adaptive, incomplete pairing schedules, crucial for fast-evolving model landscapes.
The interface design—a strategic LLM on top of a tactical executor with flavor-level parameter passing—mirrors real-world hierarchical human-AI decision systems, exposing both the strengths and lingering limitations of LLMs in translating intent into optimal, adaptive, strategic action amid stochastic, multilateral, adversarial contexts.
Theoretical
CivBench reveals pronounced knowing-doing gaps, agentic setup-model interaction effects, and diverging capability profiles that are invisible to win/loss metrics. Standard score-based proxies are inadequate, especially for non-military victory paths. Dense longitudinal evaluation is necessary for understanding not just average performance, but also behavioral diversity, initiative, and adaptability.
The reactivity in pivot patterns and strong specialization calls into question the degree to which current LLM strategists are capable of nuanced, counterfactual-aware, multi-agent adaptation. This exposes axes for future research in agentic self-reflection, continual learning, and more sophisticated hierarchical or meta-strategy modeling.
Future Directions
Suggested extensions include deploying sequence-aware estimators for better capturing campaign momentum in conflict, integrating cross-game learning signals (warranting dynamic rating methods such as OpenSkill), and leveraging CivBench for human-AI and multi-LLM interaction studies. The field would also benefit from controlled ablations isolating the effects of varying tactical modules, information access, and explicit communication pipelines between LLMs and their "subagent" briefers.
Developing more robust, interpretable models for strategic adaptation and error diagnosis could further elucidate the knowing-doing gap and inform approaches to close it—e.g., with intervention or meta-cognitive prompting, or by targeting adaptive, feedback-driven strategic corrections.
Conclusion
CivBench delivers a validated, dense, progress-based framework for evaluating LLM strategic decision-making in a complex, multi-agent 4X environment ("Civilization V"), enabling both quantitative comparisons and diagnostic profiling. The benchmark exposes persistent performance gaps relative to rule-based systems and, crucially, uncovers qualitative behavioral differences across models and configurations. As LLMs increasingly serve as agentic components in autonomously operated systems, the methods and insights established by CivBench define key directions for future capability assessment and agent development (2604.07733).