CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI

Abstract: We present CivBench, an open-source benchmark for evaluating LLM agents in long-horizon, tool-mediated environments through the Model Context Protocol (MCP). A single episode spans 300+ turns and produces thousands of tool calls over a large action space, requiring sustained planning, state monitoring, and execution under partial observability. The environment exposes 76 MCP tools and a narration layer that converts visual game state into structured text. We use CivBench to characterise agent behaviour across four model families in 23 admissible runs. The sample is a pilot, not a model ranking: aggregate outcomes do not reliably discriminate models at this scale. Instead, we introduce two interface-level metrics that the environment makes measurable: Proactive Monitoring Rate (PMR), capturing whether agents actively query latent strategic state, and RAG@10, capturing whether commitments stated in structured planning reflections are executed within ten subsequent turns. Across runs we observe two consistent patterns under a shared playbook protocol. Agents under-monitor strategically relevant state that is available but requires explicit querying: despite playbook guidance to query victory progress every 20 turns, agents do so only every 30 to 75 turns, and in 7 of 20 detectable defeats they failed to query within the 20 turn warning window before game end. Agents also frequently fail to execute near-term commitments stated in their own planning reflections (RAG@10 between 48.2% and 65.8% across models). Both patterns arise despite tool access and explicit guidance, and we interpret them as deviations under instruction rather than absences of capability. We release the environment, scenarios, logs, metrics, and analysis pipeline at https://github.com/lmwilki/civ6-mcp

• The paper assesses tool-mediated agents in the Civilization VI game by establishing CIVBench to evaluate long-horizon decision-making and strategic monitoring with two key measures: Proactive Monitoring Rate (PMR) and RAG@10 between language-model agents.
• Agents using CivBench tools typically prioritize local actions over global monitoring, and frequently fail to execute short-term commitments
• Final game scores do not reliably distinguish model performances, underlining the importance of monitoring interaction traces.

Benchmark motivation and scope

CivBench addresses a specific deficiency in long-horizon agent evaluation: aggregate task success does not reveal whether an agent retrieved the information required for a decision, maintained strategic state over time, or executed commitments made earlier in an episode. The benchmark places language-model agents in Civilization VI, where a full game typically exceeds 300 turns and requires interdependent decisions concerning expansion, production, research, diplomacy, military operations, religion, spatial planning, and victory progression. The environment therefore combines long temporal horizons, partial observability, a large action space, delayed consequences, and competing objectives.

The paper’s central methodological claim is that an agent’s interaction trace can expose behavioural failures that final outcomes compress. CivBench distinguishes between information that is available through the interface and information that has actually been retrieved by the agent. This permits direct measurement of attention allocation rather than inferring it from game score. The authors consequently frame the work as a benchmark and measurement protocol, not as a comparative model leaderboard. Across 23 admissible runs involving four model families, the study investigates two interface-level behaviours: proactive monitoring of latent strategic state and execution of near-term commitments stated in planning reflections (2609.02459).

MCP environment and controlled observability

CivBench connects Civilization VI to language-model agents through the Model Context Protocol. The server exposes 76 tools spanning state queries, unit actions, city management, diplomacy, governance, religion, resources, research, and game lifecycle operations. A typical turn requires approximately 5–15 tool calls, producing thousands of calls during a complete game. Unlike a conventional Gymnasium interface, the agent must select both the action and the observation mechanism through which relevant state enters its context.

The narration layer is the principal interface contribution. It converts visual and game-engine state into structured textual observations while preserving active retrieval as a constraint. The layer contains 29 narration functions covering overview statistics, units, cities, map structure, combat, production, technology, diplomacy, trade, governance, religion, espionage, World Congress, victory progress, Great People, resources, and notifications. The resulting interface provides semantically useful descriptions without automatically exposing all available information.

Figure 1: MCP architecture connecting the language-model agent, tool server, narration layer, and Civilization VI.

This design separates representational adequacy from retrieval behaviour. For example, victory progress is available through a dedicated query, but no victory information enters the agent’s context unless that query is made. The benchmark can therefore distinguish a failure to reason over known state from a failure to retrieve state in the first place. The distinction is important because the paper’s strongest findings concern information that was both available and explicitly recommended by the playbook.

The environment uses fixed scenarios, seeds, a versioned harness, a shared playbook, and complete MCP transcripts. The primary scenarios are Ground Control, a baseline Pangaea setting; Snowflake, a military-focused small-map scenario; and Cry Havoc, an adversarial stress test excluded from the reported results. The admissible dataset contains 19 Ground Control runs and four Snowflake runs. A structured diary is required at each turn, with tactical, strategic, tooling, planning, and hypothesis fields. This diary stabilizes interaction over long episodes and supplies explicit commitments for subsequent execution analysis.

Evaluation design and behavioural metrics

The paper reports conventional outcome measures alongside two trace-level metrics. Normalised score is defined as the agent’s raw score divided by the winning agent’s raw score at game end. The authors explicitly treat this quantity as descriptive: Civilization VI score aggregates heterogeneous systems and is strongly affected by opponent strength, game duration, and scenario conditions.

The first trace-level metric is Proactive Monitoring Rate (PMR):

$$\mathrm{PMR} = \frac{\text{strategic monitoring calls}} {\text{all non-infrastructure tool calls}}.$$

Monitoring calls include queries such as victory progress and diplomacy, whereas infrastructure calls such as end_turn are excluded. PMR is not assumed to be monotonically beneficial. High monitoring can be wasteful, and low monitoring can be appropriate in a locally constrained state. Its purpose is to quantify how much of the agent’s tool budget is allocated to maintaining global strategic awareness.

The second metric is RAG@10, the execution rate for concrete commitments made in planning reflections over the next ten turns. Executed commitments receive full credit, partially executed commitments receive half credit, and unexecuted commitments receive no credit:

$\mathrm{RAG@10} = \frac{Y+0.5P}{Y+P+N}.$

RAG@10 measures execution fidelity rather than planning quality. A commitment can be strategically poor but successfully executed, or strategically sound but invalidated by new information. The fixed ten-turn horizon is intended to capture near-term follow-through while limiting ambiguity caused by long-range changes in game state.

The commitment extraction pipeline uses Claude Haiku 4.5 and a fixed rubric. On a stratified sample of 50 commitments, agreement with independent author labels was 92.0%, with Cohen’s $\kappa=0.879$. This supports the reliability of the operationalization, although the metric remains dependent on model-assisted interpretation of diary text.

Aggregate outcomes and the limits of model ranking

Aggregate results do not support stable model comparisons. There are only three victories in 23 admissible runs, all on Ground Control. Claude Opus 4.6 records two Ground Control victories, Gemini 3.1 Pro records one, and GPT-5.4 and Kimi-K2.5 record none in the reported runs. No model wins on Snowflake. Fisher’s exact test does not establish model-level discrimination for win/loss outcomes ($p=0.488$).

Normalised score is similarly non-discriminative, with a Kruskal–Wallis statistic of $H=1.90$ and $p=0.594$. An exploratory intraclass-correlation analysis reinforces this result: among 13 candidate metrics, only exploration at turn 100 shows clear discriminative structure ($\mathrm{ICC}=0.717$, $p=0.0065$). Final score, city counts, science, culture, gold, and military measures have near-zero or negative ICC estimates. Within-model variation is therefore comparable to, or larger than, between-model variation in this pilot.

Figure 2: Outcome heatmap summarizing sparse wins and losses across models and scenarios.

The implication is methodological rather than merely statistical. A full-game score aggregates hundreds of decisions and can conceal the mechanisms that produced divergence. The paper accordingly shifts analysis from leaderboard-style outcomes to tool selection, information retrieval, and commitment execution. This is a reasonable response to the dataset size, although it means that the empirical claims are about behavioural patterns under a protocol rather than comparative model capability.

Tool allocation and strategic under-monitoring

Across models, tool-call budgets are dominated by local actions and ordinary state queries. Strategic monitoring is consistently a small fraction of interaction, despite the availability of dedicated tools and explicit playbook instructions to perform global checks every 20 turns.

Figure 3: Rolling tool-call composition showing the dominance of local actions and ordinary state queries over strategic monitoring.

Aggregate PMR ranges from 0.96% to 2.13% of non-infrastructure calls. Victory-progress queries are especially rare, accounting for only 0.05%–0.29% of calls, or approximately 3.7–10.0 queries per game. The observed interval between queries is approximately 30–75 turns, substantially longer than the prescribed 20-turn checkpoint.

Figure 4: Proactive Monitoring Rate by model and monitoring subcategory, with victory-progress retrieval particularly sparse.

The authors call this pattern the “sensorium effect”: human players passively receive many visual and interface signals, whereas the agent receives only information it explicitly requests. The result is not a simple tooling deficit. The tools exist, the state is representable, and the playbook identifies the relevant queries. The behavioural failure lies in allocating insufficient interaction budget to retrieving globally relevant state.

The missed-warning analysis connects PMR to concrete outcomes. Among 20 detectable defeats, all 20 contained a rival victory threat that could have been exposed at least 20 turns before game end. Agents queried victory progress during this warning interval in only 13 cases and failed to do so in seven. GPT-5.4 missed four of eight such warnings, Claude Opus 4.6 missed three of six, and Gemini 3.1 Pro missed one of five. Kimi-K2.5 had one detectable defeat and queried within the warning window.

This association should not be interpreted as a causal estimate: the sample is small, the games are not independent in a strong statistical sense, and a query does not guarantee effective strategic response. Nevertheless, the result establishes that critical information can remain outside the agent’s context despite explicit guidance, and that this omission sometimes coincides with avoidable late-game losses. PMR also does not increase systematically toward the endgame, when global monitoring might be most valuable. The observed behaviour is therefore consistent with a persistent allocation policy rather than a transient early-game limitation.

The reflection–action gap

The second principal finding concerns the relationship between stated plans and subsequent tool calls. Diary entries frequently contain concrete near-term commitments such as founding a city, constructing campuses, checking victory progress, or completing a particular military action. These commitments are then compared against the next ten turns of interaction.

RAG@10 ranges from 48.2% to 65.8% across the three model families with sufficient commitments for analysis. Thus, between approximately one-third and one-half of commitments are either only partially executed or not executed within the evaluation window. Bootstrap confidence intervals overlap across the tested families, so the paper does not claim reliable differences between models.

Figure 5: RAG@10 decomposition into executed, partially executed, and unexecuted commitments over the following ten turns.

The result is significant because the commitments are self-generated and elicited immediately before subsequent action. It is therefore not a comparison between an external plan and an agent’s behaviour, but between an agent’s own structured reflection and its later interaction trace. The authors interpret this as an execution-fidelity problem under instruction, not evidence that the models lack strategic knowledge. A model may articulate an appropriate plan while failing to preserve it as an actionable priority when local state queries and immediate actions compete for context.

The diary itself creates an important confound. Because the playbook requires explicit planning fields, RAG@10 measures adherence to prompted commitments rather than spontaneous planning. Conversely, without structured reflections, intent would be substantially harder to infer from action sequences. The metric is therefore valuable precisely because it sacrifices ecological purity for interpretability.

The paper’s examples also show why a ten-turn horizon matters. A commitment such as “found second city” or “check victory progress” is sufficiently concrete to evaluate immediately, whereas a broad statement such as “improve the economy” would be difficult to map onto individual actions. RAG@10 should consequently be understood as a metric for operationalizing short-horizon plan persistence, not as a general measure of strategic coherence.

Trajectory-level evidence

Supplementary trajectories provide context for the weak aggregate discrimination. On Ground Control, model score trajectories are relatively close through approximately turn 50 but separate during the midgame. At turn 50, mean raw scores are approximately 96.6 for Claude, 112.2 for Gemini, and 91.8 for GPT. By turn 150, Claude and Gemini reach approximately 394 and 416 respectively, compared with approximately 320 for GPT. At turn 200, Claude reaches approximately 547 while GPT is approximately 418. These differences indicate meaningful divergence in developmental trajectories even though final normalised score does not reliably distinguish the model families.

Figure 6: Raw score trajectories on Ground Control, with 95% confidence bands.

The trajectory decomposition attributes much of the separation to science output. Culture trajectories cross repeatedly and show weaker between-model structure, while cumulative gold and faith are strongly affected by game length. This pattern is compatible with the compounding mechanics of Civilization VI, in which technology unlocks can increase subsequent production and yield efficiency. It also illustrates why final score and normalised score can be misleading: a run’s apparent standing depends on the winning opponent and termination point, not only on the agent’s developmental trajectory.

Expansion provides another trajectory-level signal. All model groups fall below the playbook benchmark of three cities at turn 50: Claude and GPT average 2.83, Gemini 2.67, and Kimi 2.00. At turn 100, Claude and GPT average 4.67 cities, Gemini 4.00, and Kimi 3.00. These results are descriptive and unevenly sampled, but they support the paper’s broader observation that agents tend to prioritize locally salient actions over delayed-payoff investments such as expansion.

Figure 7: Mean city counts at turn checkpoints, showing persistent shortfalls relative to the playbook’s expansion benchmarks.

Limitations and open questions

The principal limitation is statistical power. The 23 admissible runs are distributed unevenly across four model families, including a single exploratory Kimi-K2.5 run. The data can support detection of large, repeated behavioural patterns but not stable model rankings or precise estimates of effect size. Snowflake contributes only four runs, and all observed victories occur on Ground Control; generalization across maps, difficulties, victory constraints, and opponent configurations is therefore unresolved.

The shared playbook is both a control mechanism and a confound. It standardizes turn structure, highlights strategic checkpoints, and elicits commitments, but the resulting behaviour reflects model reasoning jointly with instruction following and scaffolding. In particular, the claims about under-monitoring are claims about failure under explicit guidance, not unassisted behaviour. The pre-harness comparison cannot solve this problem: only 21% of pre-harness runs reached a natural conclusion, making those runs inadmissible rather than a valid baseline.

No random, scripted, or human baseline is included. Such baselines would clarify whether the observed PMR and RAG values are low relative to competent play, and whether the metrics distinguish useful monitoring from indiscriminate querying. The benchmark also depends on Civilization VI’s commercial licensing, game-version stability, FireTuner’s single-connection limitation, provider-side model updates, and substantial per-run cost of approximately $31–229. Full episodes require two to eight hours on local consumer hardware.

Finally, both central metrics are protocol-dependent. PMR depends on the selected monitoring-tool taxonomy and does not by itself establish that more monitoring improves outcomes. RAG@10 depends on LLM-assisted commitment extraction and on a fixed ten-turn horizon. The paper leaves open whether removing the diary, adding persistent commitment tracking, enforcing monitoring schedules, or surfacing selected state automatically would improve execution and game performance. Those interventions are necessary to determine whether the measured behaviours are properties of the models, the interface, the scaffolding, or their interaction.

Conclusion

CivBench provides a long-horizon MCP environment in which agent behaviour is observable at the level of tool selection, state retrieval, and plan execution. Its main contribution is not the sparse outcome comparison across four model families, but the separation of aggregate success from the interaction mechanisms that produce it.

Across 23 admissible runs, agents devote little tool budget to strategic monitoring and fail to retrieve victory information during a substantial fraction of detectable warning windows. They also execute only 48.2%–65.8% of concrete commitments within ten turns. These findings are appropriately framed as pilot observations under a shared playbook, but they demonstrate that availability of information and ability to state a plan do not ensure retrieval or execution. The released environment, transcripts, metrics, and analysis pipeline provide a basis for testing whether persistent memory, enforced monitoring, alternative narration policies, or different reflection protocols alter these two measurable failure modes (2609.02459).