FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents

Abstract: LLM agents now execute bounded tasks reliably. Whether they can sustain effective decision-making over long horizons, where actions have cumulative consequences and the environment responds to their choices, remains largely unmeasured. FM-Bench (Football Management Benchmark) measures this. An LLM agent runs a football club for 20 in-game years through 26 tools and roughly 340 to 400 decision stops. It drafts a squad on the same budget as every rival, trades players, negotiates contracts, invests in facilities and youth, sets lineups, and answers to a board that can fire it, while a deterministic engine accumulates every year into one final score with no LLM judge or human rater. The solo track plays each of 15 frontier models against a frozen scripted world, and the Arena places the same models plus a scripted anchor in one shared 20-year world; to our knowledge, the first head-to-head evaluation at this scale. We measure six behavioral capabilities behind the score. Across three seeds, all 15 models complete every horizon while the blind scripted baselines die out in most of theirs, and claude-fable-5 tops the solo board on mean score and the Arena, where the title nonetheless rotates among ten models. Neither scale, price, nor vendor predicts the order; the order settles only late in the horizon, and the best first-play human lands only at the bottom of the model board. What separates the models is managerial behavior rather than computation. Higher-scoring models reduce slow-payoff investment near the end, keep cash invested rather than idle, and open renewals well before the deadline, while token spend predicts nothing. No model learns the market's hidden prices from hundreds of rejected bids, and self-managed memory fails in two opposite modes: an archive that only grows or a plan rewritten every season. Code is available at https://github.com/Analogy-AI/fm-bench.

• The paper introduces FM-Bench, a deterministic benchmark in which language-model agents manage football clubs for 20 years across hidden information, delayed consequences, adaptive markets, and board pressure without LLM or human judging.
• The benchmark finds claude-fable-5 leading both tracks with a 90.94 solo score and 76.26 Arena score, while model scale, vendor, price, and token use fail to predict performance.
• The results show that successful long-horizon agents need endgame awareness, proactive renewals, disciplined cash allocation, price discovery, and effective memory curation, while rankings can change substantially over time and competition limits runaway advantages.

FM-Bench (2608.18423) is a benchmark for evaluating language-model agents on long-horizon management under competition. An agent runs a football club for 20 in-game years, roughly 340–400 decision stops and ~1,760 API turns at the median, through a 26-tool interface, with every quantity graded by a deterministic engine into a single composite score. No LLM judge or human rater participates anywhere in grading.

Task structure and the four demands

The benchmark defines management by four demands, each instantiated as a concrete mechanism with a calibration lever rather than prompt-level description:

• Hidden information: player ability, potential, and traits are permanently hidden; scouting returns bands with a per-scout bias, so valuation requires calibrated statistical inference.
• Cumulative consequences: facility and youth investment pay off years later; insolvency and board-confidence collapse compound yet remain recoverable via capped revival (up to three restarts discounted by $0.8^k$).
• Counter-adaptive market: rejected bids raise hidden asks, repeat-pair dealings incur markups, negotiation spam triggers cooldowns.
• Multi-objective pressure: a board jointly judges results and financial discipline against targets scaled to squad strength, and can fire the manager.

Each decision stop opens a fresh conversation; the only cross-stop state the agent controls is a self-authored notebook, making memory curation an explicit measured capability rather than a harness choice. Scoring aggregates log-compressed honors, net-worth value added (with anti-hoarding discounts), and squad value; the paper proves no fire-to-cap arbitrage, channel monotonicity, fair-price transfer neutrality on the VA channel, and diminishing returns, so the scale cannot saturate or be gamed by early settlement.

Two tracks

The solo track plays each of 15 flagship models (10 closed, 5 open-weight) against 15 tiered scripted clubs over three seeds. The Arena places all 15 models plus a scripted anchor in one shared 20-year world — claimed to be the first head-to-head evaluation of this many frontier models at this horizon. An oracle policy that reads true hidden state but acts through the same interface fixes a soft ceiling at 95.54.

Headline results

All 15 models complete the horizon across seeds, while blind scripted anchors die out in 7 of 9 runs; the disciplined heuristic scores only 17.05 solo. claude-fable-5 tops both tracks: 90.94 ± 5.20 solo (~95% of the oracle) and 76.26 in the Arena. Notably:

Seat	Solo score	Tokens
claude-fable-5	90.94 ± 5.20	24M
kimi-k2.6	88.49 ± 0.15	87M
gpt-5.6-terra	86.66 ± 1.20	28M
claude-haiku-4.5	36.90 ± 22.73	86M

Neither model scale, price, vendor, nor token spend orders the ranking ($r_s = -0.19$ for tokens, $p = 0.50$). The ordering settles late: rank correlation with final order climbs from 0.19 at year 5 to 0.78 at year 15, and deepseek-v4-pro led at years 5 and 10 yet finished 12th. In the Arena, the league title rotates among ten different models while the four best solo runs hold dynasties — isolating the counter-adaptive market as the mechanism that prevents rich-get-richer compounding.

Six first-play humans ran the same track: four were fired, and the best survivor (74.64) lands near the bottom of the model board, placing frontier agents between untrained humans and the oracle. Human self-reports independently named all four demands as the hard parts, which the authors use as construct-validity evidence.

Behavioral decomposition

The scalar score is decomposed into six behavioral capabilities with Spearman correlations stable in sign on every seed:

• Endgame awareness ($r_s = -0.58$): strong models cut slow-payoff facility/academy investment as year 20 approaches; gemini-3.5-flash instead ramps up and is still building facilities in year 19.
• Credit assignment ($r_s = -0.50$): idle-cash ratio separates winners (~80%) from bottom models (claude-haiku-4.5 at 196%); total investment is uncorrelated.
• Proactive control ($r_s = +0.45$): renewal lead time; the winner opens negotiations a median of 18 months before expiry.
• Price discovery: no model learns hidden market prices despite hundreds of rejections — median 30 offers per completed signing versus the oracle's 1.0.
• Memory curation: notebooks fail in two opposite regimes, append-only archives (gpt-5.6-sol, similarity 0.91) and wholesale rewrites (claude-sonnet-5, 0.20); the winner holds a stable strategy skeleton at 0.39.
• Compute efficiency: token spend spans sevenfold and predicts nothing under any accounting.

The capability matrix shows the winner as a generalist (no axis below 0.79), mid-table models pairing one strength with one decisive gap, and the bottom failing on several axes at once. Arena case studies add a knowing–doing gap: claude-opus-4.8 wrote the correct deploy-idle-cash policy in its notebook yet finished holding ~2,100M in reserves.

Limitations

The authors concede that three seeds bound the solo board without inferential power for pairwise claims, the Arena is a single seed-7 world with no error bars, and behavioral metrics are correlational ($n=15$, with warning exposure demonstrably confounded); causal probes such as notebook ablation remain future work. Construct validity is limited to one domain, and transfer of rankings to other long-horizon settings is untested. Two failed metrics (warning exposure, youth harvest) had unstable per-seed signs and were dropped, and a TF-IDF-based memory metric awaits semantic-embedding replication.

Conclusion

FM-Bench demonstrates that current frontier agents can sustain competent management over horizons where short benchmarks would have ranked a different set of models, while exposing specific, reproducible behavioral failures — endgame misallocation, idle cash, last-minute renewals, unlearned prices, and broken memory curation. The open questions it leaves include whether human–agent collaboration beats either alone on this track, whether multi-Arena aggregation changes orderings, and whether any model can learn hidden market prices from experience within a single run.