- The paper introduces RTSGameBench, a comprehensive benchmark for assessing strategic reasoning in VLMs in large-scale, multi-agent RTS environments.
- It employs a multi-agent framework featuring full-game evaluations and diagnostic mini-games to test competencies like resource management, spatial reasoning, and adversarial planning.
- Empirical results show that while top models excel in simple scenarios, they struggle with coordination and robustness under escalating scale and adversarial complexity.
RTSGameBench: Comprehensive RTS Benchmark for Vision-LLM Strategic Reasoning
Motivation and Benchmark Design
RTSGameBench addresses persistent deficiencies in strategic reasoning by vision-LLMs (VLMs) observed in complex agentic environments. Modern VLMs have demonstrated subpar performance in tasks requiring robust anticipation and influence over other agents under uncertainty, particularly in multi-agent competitive/cooperative settings. Existing benchmarks, such as those built on StarCraft II (SC2), impose limitations in evaluation scope, scenario diversity, and systematic competency diagnosis. RTSGameBench leverages the expanded scale and diversity of Beyond All Reason (BAR), an open-source RTS platform with a unit and battlefield scale that far exceeds prior testbeds. BAR's structural properties—2,000 unit cap per player, 32,000 total units, 100 player maximum, and 64x map area relative to SC2—facilitate rigorous evaluation of strategic reasoning involving large-scale coordination, adversarial planning, and resource allocation.
RTSGameBench is organized around three pillars: (1) Full Game Evaluation spanning various matchup structures (1v1, symmetric teams, asymmetric teams, multi-polar free-for-all), (2) Diagnostic Mini-Games targeting individual strategic competencies mapped to an RTS AI taxonomy (resource management, spatial & temporal reasoning, opponent modeling, collaboration, adversarial planning), and (3) Self-Evolving Game Generation Framework, built as an extensible multi-agent pipeline capable of converting arbitrary free-form queries into new diagnostic mini-games, with iterative rubric refinement and database reuse for scenario generation.
Environment and Agent Architecture
BAR provides an ideal substrate for scalable RTS evaluation due to its streaming economy, explicit build dependencies, and automation of routine unit micro. The benchmark defines a standardized observe-decide-act loop with multimodal observation: visual channels (global minimap, local tactical views) and structured textual extraction via engine wrappers. The action space encompasses building construction, unit production, and movement, all projected onto a normalized coordinate grid. Partial observability is enforced via fog-of-war where relevant.
RTSGameAgent, the baseline agent accompanying the benchmark, enables tractable VLM operation in large-scale scenarios. Its architecture integrates FSM-driven group management for scalable unit coordination (decomposing per-unit micro into squad-level finite-state abstraction) and LLM-based agentic memory, distinguishing short-term transient event logs from long-term situational summaries to preserve cross-step context. Each decision iteration retrieves memory relevant to the current tactical situation, feeding both to the VLM policy for strategic planning across four atomic action categories.
Evaluation Procedures and Metrics
RTSGameBench's evaluation comprises both full-game matchups and controlled mini-games designed for precise competency isolation. Full-game evaluations probe individual versus team-based strategic demands, including coordination under asymmetry and multi-polar threat prioritization. Mini-games enforce strict scenario boundaries—fixed unit composition, deterministic attack schedules, team structure, or static fortifications—to expose specific functional weaknesses in models.
Performance is quantified using win rate (WR), game time for wins/losses (GTw/GTL), damage efficiency (DE), and rank score (RS) for non-binary outcomes, as well as auxiliary metrics such as average objective completion time (AT) in planning tasks. Scaling analyses vary map size and unit count to interrogate model robustness under increased strategic complexity.
Empirical Results and Analyses
RTSGameBench's experiments profile eleven proprietary and open-source VLMs, with Gemini-3-Flash and GPT-5.2 leading overall performance. In full games, Gemini achieves WR 0.87 in Duel and RS 0.66 FFA, but performance degrades in team contexts (WR drops to 0.50 in 2v2, 0.20 in 3v4), revealing pronounced deficiencies in allied coordination and sustained multi-agent engagement. Open-source models (Qwen3-VL-235B-T, LLaMA4, Mistral-Large) exhibit minimal win rates except in scenarios with trivial demands or saturated task objectives.
Mini-game evaluations show competency-specific gaps: resource management (TCP) near-saturation (four models WR 1.00) with lower-than-human completion times; spatial-temporal reasoning (MFD) exposes the largest gap (best VLM DE 1.82 versus human DE 3.46); opponent modeling (FS-F) reveals adversarial anticipation weakness; collaboration (FS-T) remains unsolved for most models without explicit communication channels; adversarial planning (SP) yields inconsistent transfer from production planning, with top models unable to generalize.
Component ablations underscore that FSM-based group management is indispensable for large-scale tactical cohesion, while agentic memory is critical for tasks requiring context retention across temporal spans. Vision input is crucial for spatial reasoning and defense, less so for adversarial or economic tasks dominated by textual cues. Task scaling analysis demonstrates uniform performance deterioration as unit count or map size increases. Models not only lose effectiveness but also suffer increased game duration, highlighting compounding challenges in maintaining strategic closure or operational tempo.
Benchmark Extensibility via Self-Evolving Framework
RTSGameBench's self-evolving scenario generation is mediated by a multi-agent pipeline—designer, developer, analyst, project manager—operating through orchestrated dialog, iterative rubric-based validation, simulation feedback, and database artifact reuse. The framework translates arbitrary natural language queries into executable mini-games, iteratively updating rubric definitions (e.g., pathability criteria in unit placement) to capture recurring simulation failures, thus improving diagnostic robustness without constraining researchers to static scenario suites. Empirical assessment shows that the pipeline outperforms end-to-end baselines in playability, generation time, and human preference; self-evolution via rubric refinement and artifact reuse accelerates quality and coverage growth, enabling the benchmark to remain perpetually extensible.
Implications, Theoretical and Practical
RTSGameBench demonstrates that current VLM architectures remain fundamentally limited in large-scale agentic reasoning, with sharp performance decline as coordination demands, uncertainty, and operational scale grow. Diagnostic isolation of competencies shows that integration of visual and textual modalities, persistent memory architectures, and hierarchical group abstractions are baseline requirements but insufficient to solve multi-agent strategic reasoning. Benchmark extensibility via self-evolving scenario generation allows for “infinite” coverage, facilitating generalization and transfer studies as models improve.
Practically, RTSGameBench presents a reproducible and extensible testbed for the AI community, suitable for probing long-horizon decision-making, coordination protocol learning, proactive adversarial reasoning, and model generalization under partial observability. Theoretically, it foregrounds future directions involving persistent memory, emergent coalition formation, decentralized coordination mechanisms, and adaptive rubric specification. As VLM architectures evolve, RTSGameBench's scenario diversity and diagnostic rigor will remain critical to characterizing emergent strategic competencies and deficiencies in agentic AI.
Conclusion
RTSGameBench leverages the large-scale, multi-agent complexity of BAR to establish a holistic, extensible, and diagnostically precise benchmark for strategic reasoning by vision-LLMs. Systematic empirical analysis reveals that state-of-the-art models, even with group abstraction and persistent memory, struggle to maintain coordination, adapt to adversarial dynamics, and generalize with scale. The self-evolving generation pipeline ensures perpetual expansion and refinement, positioning RTSGameBench as a foundational resource for both benchmarking and advancing strategic reasoning research in agentic multimodal AI (2606.18950).
