SC2Arena Benchmarking in StarCraft II

• SC2Arena is a StarCraft II evaluation framework that supports full game episodes, all three races, and intricate low-level action controls for nuanced agent benchmarking.
• It offers two distinct formulations: one with a text-based interface for LLMs and another with a gym-style MARL environment for adversarial algorithm testing.
• The benchmark leverages detailed metrics, hierarchical planning, and combat approximations to evaluate strategic planning, self-correction, and overall agent performance.

Searching arXiv for SC2Arena and related StarCraft II benchmark papers. SC2Arena denotes a StarCraft II evaluation environment whose most explicit published formulation is a benchmark for LLMs in complex decision-making, introduced together with the StarEvolve framework. In that formulation, SC2Arena is designed to preserve full-game strategic depth by supporting all three playable races, a text-based observation interface, the complete low-level action space, and agent-vs-agent evaluation over full-length episodes (Shen et al., 14 Aug 2025). A second, distinct usage appears in the multi-agent reinforcement learning literature, where the StarCraft II battle arena (SC2BA) environment is “often referred to as ‘SC2Arena’ in code”; that environment emphasizes algorithm-vs-algorithm adversarial benchmarking under a gym-style interface and paired or mixed adversary modes (Li et al., 18 Dec 2025). The term therefore refers not to a single canonical software artifact across the literature, but to closely related StarCraft II arena-style benchmarking environments with different primary target communities and experimental protocols.

1. Definition and Scope

In the LLM benchmarking setting, SC2Arena was created because existing StarCraft II benchmarks for language-model agents “simplify or omit large portions of the original game,” including by supporting “only micro-management scenarios or high-level abstractions,” often “only one or two races, partial action sets, no agent-vs-agent,” and being “limited to fixed, short contexts (build-order subroutines)” (Shen et al., 14 Aug 2025). SC2Arena addresses these omissions by “supporting full-length game episodes (20–30 minutes, hundreds of actions/minute),” “exposing the complete low-level action space ($\sim 300$ discrete commands),” “enabling all three asymmetric races (Terran, Protoss, Zerg),” and “providing agent-vs-agent self-play and ranking mechanisms” (Shen et al., 14 Aug 2025).

In the MARL setting, SC2BA is built on StarCraft II “to replace the fixed built-in AI opponents in SMAC with evolving MARL-controlled teams,” and the accompanying summary states that it is “often referred to as ‘SC2Arena’ in code” (Li et al., 18 Dec 2025). Its emphasis differs from the LLM benchmark: it is organized around fairness, usability, and customizability for inter-algorithm adversarial evaluation, with matched combat forces, mirrored maps, a discrete combat-oriented action space, and dual-team training/testing through APyMARL (Li et al., 18 Dec 2025).

This suggests that “SC2Arena” functions as an overloaded label spanning at least two research lineages: one centered on text-conditioned generalist agents, and another centered on adversarial MARL evaluation. A plausible implication is that technical discussions of SC2Arena require disambiguation by paper, task family, and interface.

2. LLM-Oriented SC2Arena: Environment Design

The SC2Arena benchmark introduced with StarEvolve uses a text-based interface in which the environment produces an observation and the LLM returns a JSON list of low-level actions, which are then parsed and executed (Shen et al., 14 Aug 2025). The benchmark “fully supports all playable races, low-level action spaces, and optimizes text-based observations to tackle spatial reasoning challenges” (Shen et al., 14 Aug 2025). The supported races are Terran, Zerg, and Protoss, each with race-specific mechanics such as “repairable buildings, add-ons,” “creep spread, larva injections,” and “warp-in mechanics, Chrono Boost” respectively (Shen et al., 14 Aug 2025).

Race-specific handling is explicit at both the observation and action levels. For observation, “each race has its own tech tree, unit types, resource caps and key mechanics,” and SC2Arena “includes short textual reminders of race-specific structures and abilities” (Shen et al., 14 Aug 2025). For action, “the JSON action schema is the same for all races, but only valid actions for the current race” are presented, such as “BARRACKSTRAIN_MARAUDER” only for Terran (Shen et al., 14 Aug 2025).

The benchmark’s overall motivation is to evaluate “full strategic depth, long-term planning, and real-time adaptation” rather than narrow tactical fragments (Shen et al., 14 Aug 2025). Its design therefore targets full-game control rather than the more restricted task formulations characteristic of prior StarCraft II benchmarks for LLM agents.

3. Observation and Action Formalization

The observation space is defined as a hierarchical text summary derived from raw game state while “preserving spatial relationships and reducing redundancy” (Shen et al., 14 Aug 2025). At time $t$, the observation is

$$O_t = \bigl(\mathrm{Res}_t,\; U_t,\; S_t,\; A_t,\; E_t,\; M_t\bigr)$$

with six categorical sections: “resource levels, unit states, building statuses, ongoing activities, visible enemies, tech progress” (Shen et al., 14 Aug 2025). The components include $\mathrm{Res}_t \in \mathbb{R}^3$, containing “(minerals, vespene, supply_unused),” $U_t \in \mathbb{R}^{N \times d_u}$ for own units with $N \lesssim 200$, and $S_t \in \mathbb{R}^{M \times d_s}$ for own structures with $M \lesssim 50$ (Shen et al., 14 Aug 2025).

Two observation optimizations are central. First, “proximity-based unit ordering” lists units “in greedy nearest-neighbor order (starting from main base) to encode spatial adjacency” (Shen et al., 14 Aug 2025). Second, “worker aggregation” groups “all idle/collecting workers” into “a single summary line to avoid text overload” (Shen et al., 14 Aug 2025). The paper reports an ablation of “$-5$–$8\%$ WR without proximity ordering/aggregation,” and identifies “full low-level action exposure and spatial encoding” as crucial (Shen et al., 14 Aug 2025).

The action space exposes “the full low-level SC2 action set,” denoted

$t$0

where each action is a JSON object with fields including "action", "units", and optional "target_unit" or "target_position" (Shen et al., 14 Aug 2025). The action repertoire includes “macro-level actions: build structure, train unit, research upgrade” and “micro-level actions: MOVE, ATTACK, ability casts, positional targeting” (Shen et al., 14 Aug 2025). This architecture is intended to couple strategic and tactical control in a single interface rather than separating macro policy from a hard-coded micro executor.

4. Tasks, Metrics, and Baseline Results

SC2Arena evaluates agents through “full-length mirror matches (TvT, PvP, ZvZ) on an open-map (‘Melee’ Flat48),” “Agent-vs-Agent (self-play) for Elo-based ranking,” and “Agent-vs-Built-In AI at Harder (LV6) and VeryHard (LV7)” (Shen et al., 14 Aug 2025). The reported performance metrics are Elo rating with $t$1, Win Rate (WR), Time Cost of Winning (TCW), Supply Block Ratio (SBR), Resource Utilization Ratio (RUR), Tokens Per Decision (TPD), and Valid Action Ratio (VAR) (Shen et al., 14 Aug 2025).

The metric definitions are given explicitly. For Elo,

$t$2

and for WR,

$t$3

while TCW, SBR, and RUR are also formally specified in the benchmark description (Shen et al., 14 Aug 2025). These metrics jointly capture competitive strength, temporal efficiency, macro-management regularity, and action-generation quality.

Baseline results against Built-In AI (LV6) are reported for several models without self-improvement: “Qwen2.5-7B-Instruct: WR=55% $t$4 10, TCW=812s, VAR$t$5,” “Qwen3-8B (no_think): WR=45%, TCW=477s, VAR$t$6,” and “DeepSeek-V3-0324: WR$t$7, TCW=382s, VAR$t$8” (Shen et al., 14 Aug 2025). After supervised fine-tuning in StarEvolve SFT, “Qwen2.5-7B: WR\uparrow from 55% to 72% at LV6, VAR\uparrow from 48% to 86%,” and “At LV7: WR\uparrow from 55% to 73%” (Shen et al., 14 Aug 2025).

These results are framed by several explicit insights. The benchmark reports that “hierarchical Planner-Executor with iterative self-correction (via Verifiers) dramatically increases VAR and WR,” with an ablation of “$t$9 WR without Verifiers,” and that “LLMs can learn coherent long-term strategies when given full game context, but struggle with raw micro control without structured guidance and validation” (Shen et al., 14 Aug 2025).

5. Relation to StarEvolve

SC2Arena is paired with StarEvolve, described as “a hierarchical framework that integrates strategic planning with tactical execution, featuring iterative self-correction and continuous improvement via fine-tuning on high-quality gameplay data” (Shen et al., 14 Aug 2025). Its key components include “a Planner-Executor-Verifier structure to break down gameplay, and a scoring system for selecting high-quality training samples” (Shen et al., 14 Aug 2025).

The benchmark’s role is not merely evaluative. The paper states that “comprehensive analysis using SC2Arena provides valuable insights into developing generalist agents that were not possible with previous benchmarks,” and that StarEvolve “achieves superior performance in strategic planning” (Shen et al., 14 Aug 2025). Within this pairing, SC2Arena is the controlled environment and measurement substrate, whereas StarEvolve is the learning and self-improvement framework instantiated on top of it.

A plausible implication is that SC2Arena’s principal novelty lies less in isolated task design than in enabling a closed loop among long-horizon observation compression, low-level action generation, verifier-mediated correction, and post hoc data curation for fine-tuning. The paper’s ablation results on spatial encoding and verifier removal support that interpretation (Shen et al., 14 Aug 2025).

6. SC2Arena as Code Alias for SC2BA in Adversarial MARL

A separate line of work uses the phrase “SC2Arena” differently. In the SC2BA summary, the environment is introduced as “StarCraft II battle arena (SC2BA, often referred to as ‘SC2Arena’ in code)” (Li et al., 18 Dec 2025). SC2BA is organized around three principal modules: a “Configuration Module,” an “Interaction Module (gym-style),” and a “Bottom-Level Control Module” built on “Linux SC2 binary + Blizzard SC2API + Google’s PySC2 wrapper” (Li et al., 18 Dec 2025).

The interaction module exposes a vectorized per-agent observation dictionary with fields for movements, enemies, allies, and personal state, and returns a joint reward tuple $$O_t = \bigl(\mathrm{Res}_t,\; U_t,\; S_t,\; A_t,\; E_t,\; M_t\bigr)$$0 (Li et al., 18 Dec 2025). Its action space is discrete, comprising “{move_N,S,E,W, attack, stop},” with “an out-of-range attack $$O_t = \bigl(\mathrm{Res}_t,\; U_t,\; S_t,\; A_t,\; E_t,\; M_t\bigr)$$1 attack-move macro” (Li et al., 18 Dec 2025). The environment emphasizes “fairness,” “usability,” and “customizability,” including “matched combat forces,” “symmetric spawn and mirrored map,” a “Gym-like API,” and a “unified map schema” (Li et al., 18 Dec 2025).

SC2BA supports two adversarial modes. In “Dual-Algorithm Paired Adversary,” both teams are controlled by online-learning MARL algorithms, with pairwise testing over $$O_t = \bigl(\mathrm{Res}_t,\; U_t,\; S_t,\; A_t,\; E_t,\; M_t\bigr)$$2 unique matchups (Li et al., 18 Dec 2025). In “Multi-Algorithm Mixed Adversary,” the red team trains online while the blue team is sampled each episode from “a fixed pool of $$O_t = \bigl(\mathrm{Res}_t,\; U_t,\; S_t,\; A_t,\; E_t,\; M_t\bigr)$$3 pretrained policies $$O_t = \bigl(\mathrm{Res}_t,\; U_t,\; S_t,\; A_t,\; E_t,\; M_t\bigr)$$4,” using

$$O_t = \bigl(\mathrm{Res}_t,\; U_t,\; S_t,\; A_t,\; E_t,\; M_t\bigr)$$5

and performance aggregation over scenarios is reported as

$$O_t = \bigl(\mathrm{Res}_t,\; U_t,\; S_t,\; A_t,\; E_t,\; M_t\bigr)$$6

(Li et al., 18 Dec 2025).

This environment is accompanied by APyMARL, which “extends the popular PyMARL codebase to support dual-team algorithm-vs-algorithm training/testing” through an “EnvWrapper module,” “Trainer/Controller module,” and “Configurator” (Li et al., 18 Dec 2025). The environment was used to benchmark “eight representative MARL algorithms,” including “VDN, QMIX, QPLEX, QTRAN, COMA, IQL, FOP, DOP,” over symmetric and asymmetric combat scenarios (Li et al., 18 Dec 2025).

7. Extensions, Approximation Models, and Research Significance

The broader SC2Arena ecosystem also includes proposed integration points for fast combat evaluation. The combat-approximation work “Approximation Models of Combat in StarCraft 2” develops APX1–APX4 as “a computationally efficient model that is accurate at predicting the results of complex battles between diverse armies, including which army will win and how many units will remain” (Helmke et al., 2014). In the supplied details, SC2Arena can incorporate this approximation “as a ‘fast evaluator’ in its decision-making pipeline,” for example “in model-based MCTS,” “in RL training,” and “in build-order search” (Helmke et al., 2014).

The APX framework models battle state in discrete rounds through army health sets $$O_t = \bigl(\mathrm{Res}_t,\; U_t,\; S_t,\; A_t,\; E_t,\; M_t\bigr)$$7, living unit counts $$O_t = \bigl(\mathrm{Res}_t,\; U_t,\; S_t,\; A_t,\; E_t,\; M_t\bigr)$$8, and total DPS output $$O_t = \bigl(\mathrm{Res}_t,\; U_t,\; S_t,\; A_t,\; E_t,\; M_t\bigr)$$9, with refinements in APX2–APX4 for ranged damage, bonus DPS, and target prioritization (Helmke et al., 2014). The update is a discrete-time “batch” procedure in which damage budgets are computed and then applied by removing enemy units one at a time until the budget is exhausted, yielding runtime $\mathrm{Res}_t \in \mathbb{R}^3$0 per round and $\mathrm{Res}_t \in \mathbb{R}^3$1 over a battle (Helmke et al., 2014). The evaluation summary reports that APX1 “often mispredicts ranged advantages,” APX2 “drastically improves Terran-favored matches,” APX3 “refines further,” and APX4 yields “mixed gains,” with “on average APX3” identified as “the ‘sweet spot’ for cost vs. accuracy” (Helmke et al., 2014).

Within SC2Arena, the proposed API is

$\mathrm{Res}_t \in \mathbb{R}^3$2

with vectorized batch calls for parallel evaluation (Helmke et al., 2014). The description further states that, because APX “runs in linear time per simulation (no expensive pathfinding or collision), it can be millions of times faster than full-engine battles,” enabling “high-throughput rollouts and faster convergence in RL or planning” (Helmke et al., 2014).

Taken together, these strands situate SC2Arena at the intersection of benchmark design, long-horizon sequential decision-making, adversarial evaluation, and systems for efficient tactical approximation. The LLM benchmark version foregrounds full-game context, text-conditioned control, and hierarchical self-correction (Shen et al., 14 Aug 2025). The SC2BA usage foregrounds online co-adaptation, fairness-controlled combat scenarios, and algorithm-vs-algorithm stress testing in MARL (Li et al., 18 Dec 2025). The APX integration proposal suggests an additional systems layer in which approximate combat prediction can serve as a fast evaluator within planning or training loops (Helmke et al., 2014).