TEXTSCII-ALL: Multi-Race Text-Based SC2 Benchmark

• TEXTSCII-ALL is a text-based StarCraft II benchmark that supports all three races and nine matchups for testing long-horizon strategic reasoning.
• It employs a race-specific symbolic action space and a hierarchical multi-agent (HIMA) framework to mitigate invalid actions and streamline multi-step planning.
• Experimental findings reveal varied win rates across difficulty levels and a significant reduction in LLM call overhead, emphasizing efficient macro-strategy evaluation.

Searching arXiv for the primary paper and closely related benchmarks to ground the article.
arXiv search query: 2508.06042
arXiv search query: "TextStarCraft II"
TEXTSCII-ALL is an expanded text-based StarCraft II evaluation environment introduced alongside the Hierarchical Imitation Multi-Agent framework, HIMA, as a race-complete benchmark for LLM-based strategic agents. It is built on top of TextStarCraft II, uses a text interface rather than raw SC2 click- or key-level control, and covers all three races—Protoss, Terran, and Zerg—across all nine player–opponent race matchups. In the paper that introduces it, TEXTSCII-ALL is presented both as a benchmark environment and as an evaluation protocol for long-horizon strategic reasoning under resource constraints, partial observability, and race-specific tech trees [2508.06042].

1. Definition, motivation, and benchmark scope

TEXTSCII-ALL is defined as a StarCraft II testbed with a text interface designed for LLM-based agents, and as a race-complete extension of TextStarCraft II, which originally supported only one matchup, Protoss player versus Zerg opponent. Its stated purpose is to remove the coverage gap of earlier text-based SC2 benchmarks, notably TextStarCraft II and SwarmBrain, each of which fixes a single matchup. By extending evaluation to all race combinations, TEXTSCII-ALL requires an agent to handle different tech trees, unit types, and race-specific economy and abilities, rather than demonstrating competence only in a single scripted niche [2508.06042].

The benchmark includes all three races and their nine possible match-ups. Mirror matchups are explicitly included. All experiments are run on a single competitive 1v1 ladder map, Ancient Cistern LE. The environment is full-game rather than scenario-based: it spans early game, mid game, and late game, and does not define fixed templates such as rush-only or macro-only settings. Strategic diversity instead arises from professional replays used for training, race-specific tech trees, and HIMA’s multi-agent specializations.

From the benchmark-design perspective, TEXTSCII-ALL is organized along three axes: player race, opponent race, and built-in AI difficulty level. The paper uses difficulty levels 4–10, corresponding to Hard through Cheat Insane. For the main built-in AI evaluation, each difficulty–matchup configuration is evaluated with 50 games; some ablations use 20 games per difficulty; and head-to-head agent-versus-agent comparisons use 10 games per pairing.

Scope element	Coverage
Races	Protoss, Terran, Zerg
Matchups	9 player–opponent race combinations
Built-in AI levels	Lv.4–Lv.10

The inclusion of all nine matchup families is central to the benchmark’s rationale. This suggests that TEXTSCII-ALL is meant to test general strategic competence across SC2 race diversity rather than optimization for a single race pairing.

2. Interface, symbolic action space, and formal task structure

TEXTSCII-ALL inherits the text-based interaction paradigm of TextStarCraft II. Agents do not issue low-level SC2 controls. Instead, they receive a structured textual summary of the game state and respond with high-level text-encoded actions drawn from a race-specific action vocabulary. The state shown to agents includes player-side information only: current used and maximum supply, unit details by unit type, building details, technology details, and ongoing commands. Prompt examples in the appendix show this information in a structured textual format optimized for LLM use [2508.06042].

The action space is explicitly enumerated and race-specific: Protoss has 58 actions, Zerg 61, and Terran 62. These actions fall into three categories: unit production, building construction, and technology development. The environment also implements race-specific macro abilities—Chrono Boost for Protoss, Inject Larva for Zerg, and Call Mule for Terran—as well as a small set of general commands with rule-based logic, including attack and scout. Agents generate sequences of these symbolic actions as text, and the environment parses them for execution through python-sc2.

Race	Number of actions
Protoss	58
Zerg	61
Terran	62

Execution is constrained by explicit prerequisite checks. Before executing a selected action, TEXTSCII-ALL checks resource availability, tech-tree constraints, and building availability. Invalid actions are skipped or fail without effect. This is important because the paper emphasizes that naïve LLM baselines frequently propose invalid actions, whereas HIMA attempts to reduce such failures through imitation and hierarchy.

The environment is described as behaving as a Partially Observable Markov Decision Process. The underlying state space is the full SC2 game state; the observation space consists of textual summaries of player-visible state and selected feedback; and the action space is the race-specific symbolic command set. The observation function is described as a deterministic text summarizer over visible player state, and the implicit episodic reward is binary: 1 for a win and 0 otherwise. Intermediate rewards are not used. In training, the state (S_t) is explicitly limited to the player’s information only and does not contain enemy-related data. In evaluation, imitation agents continue to use player-state prompts, while the Strategic Planner additionally receives outcome feedback and observed opponent information embedded into the text input.

3. Episodes, temporal protocol, and evaluation methodology

Each episode in TEXTSCII-ALL is a full 1v1 game beginning from the standard SC2 ladder start on Ancient Cistern LE, with one starting base, initial workers, and race-specific initial buildings. Termination follows standard SC2 outcomes: victory when the opponent’s buildings are destroyed, defeat when the player’s buildings are destroyed, or timeout. In practice, the experiments use about 20 minutes as a reference time window, and long stalemates are not the focus [2508.06042].

The temporal decision protocol differs substantially between HIMA and prior baselines. TextStarCraft, HEP, and EpicStar query the LLM at short fixed intervals, such as every in-game second, producing single-step actions. HIMA instead has specialized imitation agents propose multi-step action sequences over a fixed time window of (\Delta = 3) in-game minutes. An imitation agent outputs a structured sequence (A_{t:t+\Delta}), and the Strategic Planner selects, merges, and possibly truncates these proposals. The environment continues to run in real time while these plans are being generated and executed.

The primary evaluation metric is win rate, defined for a given race matchup and difficulty level as the percentage of wins over total games. The paper reports per-matchup, per-difficulty win rates rather than collapsing the full benchmark into a single scalar. It also supports additional metrics inherited from TextStarCraft II—Average Population Utilization, Resource Utilization Ratio, Production-Block Ratio, and Technology Rate—but treats them as secondary because they correlate less reliably with actual winning. The paper therefore centers claims of performance on win rate.

TEXTSCII-ALL does not impose an explicit APM cap. Instead, it restricts action frequency implicitly through the race-specific high-level action vocabulary and the LLM query frequency. As reported in the paper, this biases evaluation toward macro-strategy and high-level planning rather than raw micro APM. A plausible implication is that the benchmark is especially informative for studying strategic reasoning under abstraction, but less informative for testing fine-grained control skill.

4. Role in HIMA and hierarchical strategic control

TEXTSCII-ALL serves as the operational environment in which HIMA is defined and evaluated. At each observation step, HIMA receives a textual prompt encoding the current player state, with the Strategic Planner additionally receiving environmental feedback and possibly observed opponent information. For each race, HIMA uses (k=3) specialized imitation agents, denoted ({\pi^{(r)}_1, \pi^{(r)}_2, \pi^{(r)}_3}), where (r \in {\text{Protoss, Zerg, Terran}}). Each imitation agent maps a text state to a multi-step action plan together with Tactical Rationale and Strategic Objective [2508.06042].

The Strategic Planner is an LLM policy that aggregates these proposals. The paper states that it uses Nominal Group Technique for advisor strategy resolution together with temporal Chain-of-Thought. The final output is a unified action sequence over the same (\Delta = 3)-minute horizon. TEXTSCII-ALL then parses this symbolic plan, executes actions over time, and skips invalid steps according to its internal resource and tech checks. When critical events occur, such as the appearance of a large enemy army or repeated plan-execution failures, the feedback system triggers replanning by the Strategic Planner.

Conceptually, this establishes a hierarchical policy over the TEXTSCII-ALL state and action abstractions. Level 1 policies generate candidate sequences; the Strategic Planner performs selection and merging; and the environment supplies the shared state representation, action semantics, and game dynamics against which these policies are evaluated. This hierarchical decomposition is not a property of the benchmark alone, but TEXTSCII-ALL is the concrete substrate that makes the decomposition executable and measurable.

The environment is also tightly coupled to the expert demonstration pipeline used to train the imitation agents. The paper uses SC2EGSet professional replays with 39.9k training instances for Protoss, 39.8k for Zerg, and 30.8k for Terran. Raw SC2 actions are mapped into the TEXTSCII-ALL action space, player-only states are extracted, future action windows of (\Delta = 3) minutes are collected, and GPT-4o-mini is used to generate Tactical Rationale over immediate, short-term, and long-term scopes. The imitation agents are then trained by supervised fine-tuning with a standard behavioral cloning loss over multi-step plans.

5. Comparative results, ablations, and efficiency

TEXTSCII-ALL is used as a common evaluation bed for HIMA, single-agent ablations, and prior LLM-based SC2 systems where direct comparison is possible. The benchmark enables comparison with TextStarCraft II, EpicStar, and HEP on Protoss versus Zerg, and with SwarmBrain on Zerg versus Terran, while HIMA itself is evaluated across all nine matchups [2508.06042].

In Protoss versus Zerg against built-in AI, the paper reports the following win rates. TextStarCraft reaches up to 75% at Lv.5 and 8% at Lv.6. EpicStar reports 67% at Lv.5 and 30% at Lv.6. HEP reports 100% at Lv.4, 75% at Lv.5 and Lv.6, and 25% at Lv.7. HIMA reports 100% at Lv.4, 92% at Lv.5, 84% at Lv.6, 82% at Lv.7, 68% at Lv.8, 20% at Lv.9, and 16% at Lv.10. In Zerg versus Terran, SwarmBrain reports 100% at Lv.4 and 76% at Lv.5, while HIMA reports 100% at Lv.4 and 84% at Lv.5, before dropping to 12% at Lv.6 and 8% at Lv.7. The paper also notes strong HIMA results up to Lv.7–8 in several Protoss matchups and weaker performance when Terran is the player race, which it associates with higher micro demands.

In agent-versus-agent mode on the same map and action space, all tested pairings use HIMA as Protoss. The reported outcomes are 100% win against SwarmBrain as Zerg, 100% win against TextStarCraft as Protoss, and 100% win against HEP as Protoss. EpicStar was not evaluated in this setting because released code was unavailable.

Ablation experiments on Protoss versus Zerg compare four configurations: Single-Agent trained on the TextStarCraft dataset, Single-Agent trained on the new human-demonstration dataset, Single-Agent with imitation learning plus Strategic Planner, and Multi-Agent with imitation learning plus Strategic Planner, अर्थात HIMA. The reported trend is that better data improves low-difficulty performance, adding the Strategic Planner helps at mid difficulties, and the full multi-agent plus Strategic Planner configuration maintains high performance at higher difficulties up to Lv.10.

The paper also reports a marked reduction in LLM call overhead over approximately 20-minute games. TextStarCraft uses 636 calls and about 7,695 seconds of total LLM time; HEP uses 468 calls and about 4,867 seconds; HIMA uses about 11 calls and about 247 seconds. Although each HIMA call is longer because multiple imitation agents and the Strategic Planner are involved, the much lower query frequency reduces total wall-clock LLM time substantially.

System	LLM calls / total LLM time
TextStarCraft	636 / ~7,695 s
HEP	468 / ~4,867 s
HIMA	~11 / ~247 s

6. Limitations, reproducibility, and future extensions

The paper identifies several limitations of TEXTSCII-ALL as currently instantiated. First, the environment is macro-level and uses a high-level action space with limited micro control. Terran performs worst in the benchmark, and the paper notes that Terran versus Terran drops to 0% win rate by Lv.6. The authors therefore suggest that richer micro abstractions or more fine-grained control may be needed in future versions. Second, all experiments are conducted on a single map, Ancient Cistern LE, so map diversity is not tested. Third, the training paradigm is largely player-centered: imitation agents use only player-side information, while opponent information is introduced mainly at deployment through the Strategic Planner. Fourth, the text abstraction omits spatial detail, terrain, and fine-grained tactical behaviors such as kiting and positioning. Fifth, the secondary metrics APU, RUR, PBR, and TR can be misleading relative to winning [2508.06042].

These limitations define the interpretive boundaries of reported results. The benchmark evaluates strategic and macro reasoning in SC2 under a text abstraction, not full SC2 mastery. This suggests that high performance in TEXTSCII-ALL should be read as evidence of competence in long-horizon strategic planning and symbolic action sequencing, rather than as evidence of comprehensive real-time mechanical skill.

The implementation is released as part of the HIMA project at https://github.com/snumprlab/hima. The system is built atop Blizzard’s StarCraft II, Patch 5.0.14.93333, and the python-sc2 library. Reproducibility details reported in the paper include the use of Ancient Cistern LE, built-in AI levels 4–10, inference temperature 0.7 for both imitation agents and the Strategic Planner, race-specific action lists in code, and rule-based implementations of Chrono Boost, Inject Larva, Call Mule, attack, and scout. Episodes run to completion under standard SC2 victory conditions.

The paper also reports hardware and runtime details. The imitation agents are fine-tuned on four NVIDIA H100 80GB GPUs, with training taking about six hours for the full multi-agent setup using LoRA adapters. At inference time, three imitation agents, each a 1.5B Qwen-2 model, run concurrently on a single NVIDIA A6000 40GB GPU. The Strategic Planner typically uses GPT-4o-mini via API, though open-source planners based on Qwen and Llama are also evaluated. Future directions implied by the paper include adding micro-oriented controls, incorporating more maps and possibly scenario-based tasks, extending the interface toward richer opponent modeling under fog-of-war, and establishing TEXTSCII-ALL as a standard benchmark for multi-agent LLM strategic reasoning.