CREW-Wildfire Benchmark for Multi-Agent Response

• CREW-Wildfire is an open, scalable benchmark for multi-agent systems that integrates wildfire physics, heterogeneous roles, and partial observability.
• It extends the CREW platform by incorporating realistic fire dynamics, procedural map generation, and LLM-compatible perception and execution modules.
• The benchmark supports tests with 2000+ agents over 1M+ cells, enabling rigorous evaluation of scalability, spatial reasoning, and adaptive long-horizon planning.

Searching arXiv for the primary paper and closely related benchmark context.

CREW-Wildfire is an open, scalable benchmark for agentic multi-agent systems that grounds LLM agents in physically meaningful, dynamic wildfire response. Built atop the CREW human–AI teaming platform, it extends CREW from human-in-the-loop interaction to autonomous, heterogeneous, multi-agent teams, and is intended to evaluate scalability, robustness, coordination, communication, spatial reasoning, and long-horizon planning under partial observability and stochastic wildfire dynamics (Hyun et al., 7 Jul 2025).

1. Scope, motivation, and benchmark niche

CREW-Wildfire was introduced to address a specific evaluation gap in LLM-based multi-agent research. The benchmark targets settings in which existing environments are small, fully observable, or symbolic, and therefore under-stress embodied control, uncertainty, and long-horizon coordination. In contrast, CREW-Wildfire combines large-scale maps, heterogeneous and role-differentiated agents, strict partial observability, stochastic and nonstationary wildfire dynamics, and coupled objectives that require both low-level execution and high-level coordination via natural language (Hyun et al., 7 Jul 2025).

The benchmark’s scale is central to its design. It supports 2000+ agents and 1,000,000+ cell maps, including grids up to $1000 \times 1000$, and thereby moves beyond the small-scale environments that dominate much prior evaluation. This scale is paired with decentralized information asymmetry: agents receive localized views, occlusions matter, and sensing is limited. The result is an environment in which coordination quality depends not only on planning competence but also on what information is perceived, shared, or omitted (Hyun et al., 7 Jul 2025).

CREW-Wildfire also extends the original CREW simulation backbone with wildfire physics, a modular perception and execution stack tailored to LLMs, heterogeneous assets, and a benchmarking suite organized around levels, metrics, and behavior-oriented evaluation. This makes the benchmark simultaneously an environment for policy learning, a testbed for LLM-mediated coordination, and an instrument for analyzing failure modes in scalable multi-agent systems (Hyun et al., 7 Jul 2025).

2. Environment construction and scenario dynamics

The environment is fully grid-embedded, with map sizes ranging from small scenarios of approximately $30$–$60$ cells per axis to medium and large scenarios of $100$–$250$, while retaining support for very large $1000 \times 1000$ maps. Procedural generation is based on Perlin noise layers that produce continuous elevation, moisture, and wind vector fields, together with discrete land-cover types such as forest, brush, rock, and water, plus randomly placed settlements controlled by seeds. Each run is therefore unique and uncertain, while still reproducible through fixed seeds (Hyun et al., 7 Jul 2025).

Wildfire propagation is modeled as an advanced cellular automaton informed by wind, slope, vegetation or fuel, and moisture. This creates moving hazards, breaches of firelines, and evolving priorities. Partial observability is enforced through local minimap windows, with occluded cells marked until agents approach closely enough to reveal them. Observations may be rendered visually as images, encoded as ASCII maps, or provided as ground-truth vectors, allowing the same scenario to support both embodied and symbolic control interfaces (Hyun et al., 7 Jul 2025).

The tasks are deliberately long-horizon and coupled. Representative objectives include locate-and-suppress, search-and-rescue with transport, multi-resource deployment, containment, and logistical transport. These tasks require hierarchical decomposition, role assignment, and continual replanning as fire spread changes the geometry of the problem. The same level can be approached either through low-level vector control or through higher-level natural-language commands that invoke execution primitives, which allows direct comparison between structured policy learning and language-mediated agency (Hyun et al., 7 Jul 2025).

A common misconception is that wildfire realism in a benchmark is reducible to a spread simulator. CREW-Wildfire is broader: wildfire dynamics are only one component of a setting that also includes heterogeneous embodiment, partial observability, human-readable interfaces, and team-level objectives that couple reconnaissance, suppression, transport, and rescue (Hyun et al., 7 Jul 2025).

3. Agent types, observations, and the language interface

CREW-Wildfire defines four principal agent modalities. Firefighters can cut trees to create firelines, spray water to extinguish or reduce flammability, rescue civilians, pick up and drop civilians, and refill water at sources. Bulldozers can rapidly clear flammable vegetation using a plow and are suited to line construction, but they have lower mobility and cannot extinguish fires or rescue civilians. Drones provide reconnaissance only, with wide-area scouting to detect fires and civilians. Helicopters transport agents and civilians, deliver water payloads, refill water, and become essential for mobility on larger maps (Hyun et al., 7 Jul 2025).

Action spaces support structured low-level tensors, textual actions, and multi-step primitives. The primitives include Move to Location, Cut All Trees in current cell, Spray Water Cone toward target, Drive to Location (Clear Path), Fly to Location, Pick Up or Drop Off Firefighters or Civilians, Refill Water, and Drop Water. These abstractions reduce token usage and structure long-horizon control by compressing repeated embodied sequences into executable macros (Hyun et al., 7 Jul 2025).

The language interface is organized around modular Perception and Execution modules. Perception transforms raw observations into concise textual summaries suitable for LLM reasoning. It prompts an LLM with an ASCII minimap and legend, after which the agent returns grounded text such as relative fire location, nearby water sources, or civilian coordinates. The design explicitly discourages literal character transcription and instead encourages spatial interpretation. Execution maps LLM-generated text actions into executable formats using a standard structure, Action{ int type; int param1; int param2; string description }, with examples such as [1, 500, 500, "move to coordinate location of (500, 500)"]. If an action is unexecutable or violates constraints, the module re-prompts until a valid action vector is produced (Hyun et al., 7 Jul 2025).

Communication occurs through natural-language channels. Message costs, API call frequency, and input and output tokens are tracked, but communication penalties are not baked into the reward by default. Policies may therefore be centralized, decentralized, or hybrid, including leader-based and consensus-making schemes, without changing the environment’s underlying interaction model (Hyun et al., 7 Jul 2025).

4. Formal model, tasks, and evaluation methodology

Formally, CREW-Wildfire is posed as a decentralized partially observable Markov decision process,

$$(N, S, \{A_i\}_{i=1}^N, A, \{O_i\}_{i=1}^N, T, \Omega, R, \gamma, T_h),$$

where $N$ is the number of heterogeneous agents, $S$ is the global state space, $A_i$ is the action space of agent $30$0, $30$1 is the observation space of agent $30$2, $30$3 is the transition kernel, $30$4 is the observation function under partial observability, $30$5 is the team-level reward, $30$6 is the discount, and $30$7 is the planning horizon or time limit. Decentralized policies are written as $30$8, where $30$9 includes action-observation history and messages when present. The optimization objective is

$60$0

In penalty-style levels, maximizing $60$1 corresponds to minimizing cumulative losses over time (Hyun et al., 7 Jul 2025).

The benchmark defines level-specific score functions rather than a single generic success metric.

Task	Score definition
Cut Trees	$60$2 number of correctly cut trees at marked cells or lines
Scout Fire	$60$3
Transport Firefighters	$60$4
Rescue Civilians	$60$5
Suppress Fire	$60$6
Full Environment	$60$7

To compare heterogeneous tasks, CREW-Wildfire introduces the Behavioral Competency Score (BCS). Task-level scores are first normalized: $60$8 where $60$9 is a worst-case baseline and $100$0 is the target outcome. For a behavioral goal $100$1 with task set $100$2,

$100$3

The behavioral goals are Task Designation, Agent Capitalization, Spatial Reasoning, Observation Sharing, Realtime Coordination, Plan Adaptation, and Objective Prioritization. In addition, the benchmark records API calls per timestep and per agent, input and output token counts, success rates, and task durations, with aggregate reporting as mean $100$4 standard deviation across multiple seeds (Hyun et al., 7 Jul 2025).

5. Baselines, experimental protocol, and empirical findings

The reported baselines comprise CAMON, COELA, Embodied, HMAS-2, and Do Nothing. CAMON is a hybrid leader-based framework with dynamic leadership transfer. COELA uses decentralized propose/evaluate messaging. Embodied alternates communication rounds with independent action planning. HMAS-2 uses a hybrid centralized planner refined through agent feedback to reach consensus. All experiments use GPT-4o with temperature $100$5 and one completion per step, and all frameworks share the same Perception and Execution modules for fairness. The reported setup uses a 3.0 GHz CPU, an RTX 3060 GPU, 16 GB RAM, and 3 seeds per level across 12 levels (Hyun et al., 7 Jul 2025).

The empirical pattern is sharply stratified by task complexity. On simple tasks, all LLM frameworks show emergent collaboration in Cut Trees, and CAMON often leads, including on sparse and line variants. As map size increases, decentralized frameworks exhibit more overlap and redundancy because global knowledge sharing is weak. On complex tasks such as Suppress Fire and Search-and-Rescue, performance degrades substantially; in some cases, penalties from agent loss make results worse than Do Nothing. Centralized and hybrid planners struggle with multi-level subtask decomposition and often issue overlapping or non-adaptive plans, whereas decentralized systems show repetitive or vague messaging and fail to converge (Hyun et al., 7 Jul 2025).

Communication scaling is a major empirical result. API calls and output token counts scale roughly linearly with agent count, but centralized or hybrid schemes can incur much larger input overhead. In particular, HMAS-2’s input tokens can grow quadratically when global-state vectors are shared under partial observability, and universal context sharing becomes inefficient on large maps such as $100$6. The benchmark therefore exposes a systems-level bottleneck in information routing, not only a reasoning bottleneck in task execution. The reported findings point to unsolved problems in hierarchical coordination, attention-based message routing, spatial abstraction, and robust replanning under uncertainty (Hyun et al., 7 Jul 2025).

6. Reproducibility, comparative position, and open problems

CREW-Wildfire is implemented on Unity through the CREW platform and is described as providing scalable simulation, visually rich and customizable environments, LLM-compatible Perception and Execution modules, and primitives for hierarchical action abstraction. Code, maps, seeds, and baseline implementations are released, including scenario catalogs with 12 benchmark levels and seeds, and the benchmark reports demonstrated support for 2000+ agents and 1M+ cells on commodity hardware, with parallelization workflows supported by Unity’s simulation loop and batched evaluation scripts (Hyun et al., 7 Jul 2025).

Within the benchmark landscape, CREW-Wildfire is presented as addressing limitations seen in prior environments such as Lyfe Game, VirtualHome, PettingZoo, and CUISINEWORLD, which are described as limited in one or more of agent scale, embodiment realism, heterogeneity, partial observability, or complex long-horizon tasks. CREW-Wildfire’s distinctive combination is embodied control, realistic stochastic physics, heterogeneous roles, low-level control, direct observations, partial observability, and language interfaces in one environment. This combination enables both quantitative benchmarking and qualitative behavioral analysis of agentic LLM systems (Hyun et al., 7 Jul 2025).

The benchmark also has explicit limitations. Roles and low-level physics are simplified, and actuator or robot dynamics are abstracted to emphasize planning and coordination rather than transfer to real hardware. Token inefficiencies constrain large-scale testing, and centralized global-state sharing can induce $100$7 input-token growth. Future directions identified for the benchmark include scalable communication architectures, better task abstraction and modular planning, uncertainty-aware reasoning, adaptive leadership, policy learning that integrates language with structured state-space control, more seeds, more prompt variants, and increased physics fidelity, including richer Rothermel-style modeling (Hyun et al., 7 Jul 2025).

Adjacent wildfire AI systems emphasize different layers of the problem: hierarchical human–autonomy teaming for initial attack (Al-Husseini et al., 2024), GPU-accelerated differentiable wildfire simulation (Xia et al., 26 Feb 2025), and collaborative UAV escape-route planning (Raj et al., 27 Oct 2025). This suggests a complementary relationship rather than direct substitution: CREW-Wildfire functions as a benchmark layer for scalable multi-agent Agentic AI, while those systems emphasize operational forecasting, coordination, or evacuation. In that sense, CREW-Wildfire’s primary significance lies not in replacing a suppression or forecasting stack, but in making its coordination assumptions measurable, comparable, and failure-revealing at scale.