DisasterBench: Typed Workflow Planning

• DisasterBench is a benchmark that evaluates LLMs’ ability to plan and execute multi-step disaster response workflows using typed tool interfaces.
• It enforces strict JSON schema constraints to ensure correct parameter binding, dependency propagation, and execution-grounded consistency in workflows.
• The benchmark comprises 26 tools across diverse disaster-response tasks and employs exact-match and partial-credit metrics with First-Point-of-Failure analysis.

Searching arXiv for DisasterBench and closely related disaster-response benchmarks to ground the article in current papers. DisasterBench is a benchmark for evaluating whether LLMs can plan executable multi-step workflows in disaster response when constrained by typed tool interfaces. In its 2026 formulation, it is presented as a domain-grounded benchmark for structured multi-agent planning over semantically similar but operationally distinct disaster-response tools, with an emphasis on executable workflow generation, correct parameter binding, and dependency propagation rather than merely semantically plausible tool selection (Chen et al., 27 May 2026). The same name also appears in a separate 2026 arXiv record on a UAV-based multimodal disaster-response benchmark, but the available paper contents for that record do not provide dataset descriptions, tasks, metrics, or implementation details specific to the claimed benchmark or model (Zhang et al., 4 Jun 2026). In practice, the term “DisasterBench” therefore most concretely denotes the typed-tool planning benchmark introduced for execution-grounded coordination in disaster-response pipelines (Chen et al., 27 May 2026).

1. Definition and problem setting

DisasterBench evaluates planning under the operational reality that emergency pipelines span satellite image analysis, adverse-weather perception, hydrological modeling, damage assessment, post-disaster mobility prediction, and geospatial reasoning, and that these pipelines must be composed as executable workflows rather than as loosely plausible action sequences (Chen et al., 27 May 2026). The benchmark targets the case in which an LLM serves as an orchestrator and must choose the correct tool, bind structured fields correctly, satisfy type and schema constraints, and propagate dependencies so that each downstream step can consume exactly the upstream output it expects (Chen et al., 27 May 2026).

The central distinction in DisasterBench is between semantic grounding and execution-grounded consistency. Many tools are semantically similar but operationally distinct, such as foggy-object versus low-light-object detection, forest versus urban anomaly detection, or temporal image-sequence versus RGB image classification (Chen et al., 27 May 2026). DisasterBench is designed to separate these dimensions and diagnose their failures. This suggests that the benchmark is not merely about plan plausibility, but about whether a plan is well-typed, structurally admissible, and executable under explicit interface constraints.

Formally, a tool $i$ is modeled as a typed function $f_i : X_i \rightarrow Y_i$, with $\theta_i$ specifying input/output fields, types, and constraints (Chen et al., 27 May 2026). A workflow plan $\pi$ is an ordered sequence $P=(s_1,\dots,s_T)$, or equivalently a path in a global DAG $G=(V,E)$, with each step represented as $$(step_t, agent_t, inputs_t, outputs_t, dep_t, dep\_content_t)$$ (Chen et al., 27 May 2026). Plan validity requires type constraints, dependency satisfaction, and execution-grounded consistency over the entire path (Chen et al., 27 May 2026).

2. Benchmark composition and task structure

DisasterBench includes 26 openly available disaster-response tools spanning 13 categories, including perception, prediction, generation, and reasoning (Chen et al., 27 May 2026). The benchmark operationalizes a controlled but diverse tool ecosystem. The tool categories and representative tools are enumerated explicitly in the benchmark description.

Category	Count	Examples
Image Generation/Restoration	4	Temporal_High_Resolution_Image_Generation; High-Resolution_Image_Reconstructor; Metadata_and_Text_Prompt_Image_Generation; Weather_Degraded_Image_Restoration
Classification	3	Temporal_Image_Sequence_Classifier; RGB_GeoImage_Classifier; Multi_Spectral_Classifier
Anomaly Detection	3	Anomaly_Detection_Forest; Landslide_Segmentation; Urban_Anomaly_Detection
Object Detection/Segmentation	3	Foggy_Scenario_Object_Detection; Low-Light_Object_Detection; Geospatial_Object_Segmentation
Building Damage	1	Building_damage_assessment
Flood/Precipitation	3	Precipitation_Nowcasting; Flood_depth_prediction; depth_speed_model
Text/Event Processing	2	Toponym_Detection; Event_detection
Video Analysis	1	Video_anomaly_detection
Mobility Prediction	2	Post_Disaster_Mobility_Recovery; Multimodal_mobility_prediction_under_events
Multimodal Reasoning	1	GeoChat
Data Conversion	1	precipitation_data_convert_tool

The dataset contains 233 tasks, each a natural-language description mapped to a single executable ground-truth plan (Chen et al., 27 May 2026). These tasks are distributed across three structural types: Node (single-step, $n=35$, 15.0%), Chain (sequential multi-step, $n=166$, 71.2%), and Branching (fan-out dependencies, $n=32$, 13.7%) (Chen et al., 27 May 2026). Plan depths range from 1 to 9, with mean depth 2.53 (Chen et al., 27 May 2026). Models are given tool descriptions and schema, but not the DAG, and must infer valid plans from natural-language intent (Chen et al., 27 May 2026).

The benchmark defines an expert-curated global DAG over the 26 tools with 81 directed typed dependencies organized into four workflow families: Temporal High-Resolution Analysis (33 edges), Image Reconstruction (23), Adverse-Weather Perception (18), and Hydrological Modeling (7) (Chen et al., 27 May 2026). The DAG constrains admissible tool compositions during evaluation. A plausible implication is that DisasterBench combines language understanding with latent graph induction over interface-compatible tool chains.

3. Typed interfaces and executable workflow constraints

A distinctive feature of DisasterBench is its structured JSON plan schema, enforced by prompts during evaluation (Chen et al., 27 May 2026). Each step specifies an agent, step, dependence, dependence_content, inputs, and outputs, with dependence equal to [-1] if there is no upstream dependency, and otherwise listing the single step index it consumes (Chen et al., 27 May 2026). dependence_content maps the depended step index to output keys, while generated inputs can reference upstream outputs through placeholders such as "<GENERATED>-<STEP>-<OutputKey>" (Chen et al., 27 May 2026).

Plan validity is defined by three criteria. First, all steps must respect the tool schema $f_i : X_i \rightarrow Y_i$0, including admissible fields and types (Chen et al., 27 May 2026). Second, dependencies must be well-formed: referenced upstream steps must exist, referenced output keys must be present, and the tool pair must be admitted by the DAG (Chen et al., 27 May 2026). Third, the entire plan must induce a well-typed path in the global DAG, with consistent input-output binding across steps and no missing or extra steps relative to the ground truth when evaluated (Chen et al., 27 May 2026).

These interface constraints are not incidental. The benchmark description explicitly states that execution consistency, defined by schemas and dependencies, is orthogonal to semantic grounding (Chen et al., 27 May 2026). This is the core technical claim of DisasterBench. It frames disaster-response planning as typed program synthesis over a constrained tool graph rather than as unconstrained chain-of-thought generation.

4. Metrics and first-point failure analysis

DisasterBench uses a strict exact-match metric. Given a predicted plan $f_i : X_i \rightarrow Y_i$1 and ground truth $f_i : X_i \rightarrow Y_i$2, exact match is defined as

$f_i : X_i \rightarrow Y_i$3

where step equality jointly checks agent identity, parameter bindings, dependency structure, and dependency content (Chen et al., 27 May 2026). The strictness is motivated by the fact that a single incorrect binding may reroute downstream analysis (Chen et al., 27 May 2026).

To decompose errors, DisasterBench defines partial-credit metrics. Tool Accuracy is

$f_i : X_i \rightarrow Y_i$4

Parameter Accuracy is

$f_i : X_i \rightarrow Y_i$5

and Dependency Accuracy is

$f_i : X_i \rightarrow Y_i$6

The benchmark defines the semantic-execution gap per task as $f_i : X_i \rightarrow Y_i$7, capturing failures after correct tool selection due to parameter or dependency errors (Chen et al., 27 May 2026).

For root-cause localization, DisasterBench introduces First-Point-of-Failure (FPoF), defined for a failed prediction $f_i : X_i \rightarrow Y_i$8 relative to $f_i : X_i \rightarrow Y_i$9 as

$\theta_i$0

If aligned steps match but lengths differ, the failure is length-based, either early_stop or hallucinated_extra_steps (Chen et al., 27 May 2026). FPoF classifies the earliest root cause into tool mismatch (agent_mismatch), parameter binding errors (parameter_error, dependency_error, or dependency_content_error), or structural errors (hallucinated_extra_steps, early_stop, empty_output, format_error) (Chen et al., 27 May 2026). The analyzer scans steps in priority order agent $\theta_i$1 parameters $\theta_i$2 dependencies, then checks length mismatch (Chen et al., 27 May 2026).

This failure formalization is one of the benchmark’s most important contributions. It operationalizes the difference between a primary planning error and downstream cascades, allowing analysis at the step level rather than only at the whole-plan level (Chen et al., 27 May 2026).

5. Evaluation setup and empirical findings

The benchmark evaluates fourteen LLMs across three capability tiers: Frontier, Strong open-source, and Efficiency-oriented (Chen et al., 27 May 2026). The Frontier models are Gemini 3.1 Pro Preview and GPT-5.4; the Strong open-source models include DeepSeek-V3.2, DeepSeek-R1, Qwen3-Max, Qwen3-Max-Thinking, Qwen3.5-27B, Gemma-4-31B, and Llama-3.3-70B; the Efficiency-oriented models are Qwen3.5-9B, Llama-3.1-8B, Ministral-14B, Ministral-8B, and Ministral-3B (Chen et al., 27 May 2026).

Five planning paradigms are compared: Direct Prompting (DP), Chain-of-Thought (CoT), Tree-of-Thought (ToT), Reasoning via Planning (RAP), and ReAct (Chen et al., 27 May 2026). ToT uses breadth-first search with branching factor 7, critic samples 5, beam 1, and search depth 2 (Chen et al., 27 May 2026). RAP uses Monte Carlo Tree Search over tools with 5 rollouts, max depth 5, subquestion samples 16, confidence samples 32, and reward $\theta_i$3 with $\theta_i$4 (Chen et al., 27 May 2026). Deterministic decoding is used for DP, CoT, and ReAct; mild stochasticity (0.5) is used for ToT and RAP (Chen et al., 27 May 2026).

Overall LLM averages in exact-match accuracy are DP 44.91%, CoT 51.44%, ToT 46.87%, RAP 41.20%, and ReAct 54.29% (Chen et al., 27 May 2026). No method dominates universally (Chen et al., 27 May 2026). The best model-method combination is Gemini 3.1 Pro Preview with ToT at 73.39% (Chen et al., 27 May 2026). GPT-5.4 peaks at RAP 65.67% (Chen et al., 27 May 2026). Efficiency-tier models often prefer ReAct; for example, Ministral-14B reaches 50.64% versus CoT 33.48%, a +17 percentage-point difference (Chen et al., 27 May 2026).

Three empirical findings are emphasized. First, planning method effectiveness depends strongly on model capacity (Chen et al., 27 May 2026). Tier differences exceed method differences, indicating that execution consistency is capacity-constrained (Chen et al., 27 May 2026). Second, depth is a bottleneck: accuracy drops sharply beyond depth 2, and even Gemini falls to roughly 53–61% at depth 4 across methods (Chen et al., 27 May 2026). Third, tool mismatch and parameter-binding errors dominate first failures, indicating distinct bottlenecks in semantic grounding and execution consistency (Chen et al., 27 May 2026).

6. Error taxonomy, instruction clash, and practical implications

Across 8,523 failures, FPoF reveals distinct error distributions by method (Chen et al., 27 May 2026). For DP, tool mismatch accounts for 50.4%, Parameter Binding 34.6%, and Structural 15.0% ($\theta_i$5) (Chen et al., 27 May 2026). For CoT, the respective figures are 54.6%, 31.7%, and 13.7% ($\theta_i$6) (Chen et al., 27 May 2026). For ToT they are 52.3%, 31.0%, and 16.7% ($\theta_i$7) (Chen et al., 27 May 2026). RAP is distinctive, with Parameter Binding overtaking tool mismatch: 32.2%, 50.1%, and 17.7% ($\theta_i$8) (Chen et al., 27 May 2026). ReAct shows 58.2%, 28.7%, and 13.1% ($\theta_i$9) (Chen et al., 27 May 2026). Tool mismatch dominates for four methods, while RAP shows elevated parameter-binding failure and an early_stop rate of 11.1% of RAP failures, suggesting search truncation when models mis-evaluate partial plans (Chen et al., 27 May 2026).

The benchmark also quantifies the semantic-execution gap. Averaged across models, $\pi$0 equals 8.2 percentage points for DP, 6.7 for CoT, 9.0 for ToT, 7.9 for RAP, and 6.8 for ReAct (Chen et al., 27 May 2026). A notable case is Llama-3.3-70B under DP, where Tool Accuracy is 60.94% but EM is 38.63%, a 22.31-point gap (Chen et al., 27 May 2026). This directly demonstrates that correct tool selection does not imply executable planning.

Another major result concerns instruction clash. DisasterBench reports that verbose intermediate reasoning can conflict with schema-constrained generation (Chen et al., 27 May 2026). DeepSeek-V3.2 under CoT drops from 57.94% in DP to 36.05%, a decrease of 21.89 percentage points, with FPoF shifting toward Parameter Binding errors (Chen et al., 27 May 2026). Enforcing JSON mode rescues EM to 62.23% and restores a tool mismatch profile, confirming the clash hypothesis (Chen et al., 27 May 2026). This suggests that free-form reasoning can degrade structured-output compliance when output control is not externally enforced.

The paper provides a canonical hydrological modeling example: Precipitation_Nowcasting → precipitation_data_convert_tool → Flood_depth_prediction → depth_speed_model → Traffic Speed (Chen et al., 27 May 2026). A predicted plan that skips the conversion step and applies Flood_depth_prediction directly to Predicted Precipitation fails at the first incompatible step, as either agent_mismatch or parameter_error (Chen et al., 27 May 2026). This example illustrates how FPoF localizes the earliest divergence in a typed workflow.

7. Position within the disaster-benchmark landscape

DisasterBench is explicitly compared with AgentBench, ToolBench, APIBench, TaskBench, SWE-bench, and GAIA (Chen et al., 27 May 2026). The distinction drawn is that prior tool-use benchmarks emphasize tool selection and multi-step orchestration, but largely treat agents as freely composable (Chen et al., 27 May 2026). DisasterBench instead foregrounds typed interface constraints and a global execution DAG, evaluates executable grounding across semantically overlapping tools, and adds step-level FPoF localization to distinguish root cause from cascaded symptoms (Chen et al., 27 May 2026).

Within the broader disaster-benchmark ecosystem, other works cover adjacent but distinct axes. DisastQA provides a large-scale benchmark of 3,000 questions spanning eight disaster types, with Base, Mix, and Golden evidence conditions and a human-verified keypoint-based evaluation protocol for open-ended factual completeness (Chen et al., 7 Jan 2026). DisasterInsight restructures xBD into approximately 112K building-centered instances and supports building-function classification, damage-level and disaster-type classification, counting, and structured report generation for disaster imagery (Tehrani et al., 26 Jan 2026). DisasterVQA introduces 1,395 real-world images and 4,405 expert-curated question-answer pairs for disaster-scene VQA grounded in FEMA ESF and OCHA MIRA (Al-Mohannadi et al., 20 Jan 2026). GeoDisaster focuses on operational geospatial disaster reasoning with 2,921 verified instances across 43 question types and five task families, using executable geospatial workflows and orchestrated agents (Hasan et al., 15 Jun 2026). These benchmarks address QA, VLM evaluation, geospatial reasoning, and multimodal assessment rather than typed workflow planning itself.

A separate record titled "DisasterBench: A Multimodal Benchmark for UAV-Based Disaster Response in Complex Environments" claims a multi-stage multimodal benchmark and a lightweight model named DisasterVL, but the available paper content explicitly states that dataset descriptions, tasks, metrics, model details, and results are not provided in the supplied document (Zhang et al., 4 Jun 2026). Accordingly, no concrete technical account of that benchmark can be established from the available contents.

DisasterBench’s principal significance lies in exposing what it describes as “a fundamental gap between semantic reasoning and execution-grounded coordination” in disaster-response workflows (Chen et al., 27 May 2026). Its formal metrics, typed DAG, and FPoF analysis make it a benchmark for schema-aware planning rather than for perception, retrieval, or answer generation alone. A plausible implication is that it provides a canonical testbed for studying whether LLM-based disaster-response systems can move from plausible language outputs to operationally valid, executable coordination.