---
title: 'DisasterBench: Typed Workflow Planning'
url: https://www.emergentmind.com/topics/disasterbench
type: topic
---

# DisasterBench: Typed Workflow Planning

Searching arXiv for DisasterBench and closely related disaster-response benchmarks to ground the article in current papers.
DisasterBench is a benchmark for evaluating whether large language models can plan executable multi-step workflows in disaster response when constrained by typed tool interfaces. In its 2026 formulation, it is presented as a domain-grounded benchmark for structured multi-agent planning over semantically similar but operationally distinct disaster-response tools, with an emphasis on executable workflow generation, correct parameter binding, and dependency propagation rather than merely semantically plausible tool selection [2605.27957]. The same name also appears in a separate 2026 arXiv record on a UAV-based multimodal disaster-response benchmark, but the available paper contents for that record do not provide dataset descriptions, tasks, metrics, or implementation details specific to the claimed benchmark or model [2606.06217]. In practice, the term “DisasterBench” therefore most concretely denotes the typed-tool planning benchmark introduced for execution-grounded coordination in disaster-response pipelines [2605.27957].

## 1. Definition and problem setting

DisasterBench evaluates planning under the operational reality that emergency pipelines span satellite image analysis, adverse-weather perception, hydrological modeling, damage assessment, post-disaster mobility prediction, and geospatial reasoning, and that these pipelines must be composed as executable workflows rather than as loosely plausible action sequences [2605.27957]. The benchmark targets the case in which an LLM serves as an orchestrator and must choose the correct tool, bind structured fields correctly, satisfy type and schema constraints, and propagate dependencies so that each downstream step can consume exactly the upstream output it expects [2605.27957].

The central distinction in DisasterBench is between semantic grounding and execution-grounded consistency. Many tools are semantically similar but operationally distinct, such as foggy-object versus low-light-object detection, forest versus urban anomaly detection, or temporal image-sequence versus RGB image classification [2605.27957]. DisasterBench is designed to separate these dimensions and diagnose their failures. This suggests that the benchmark is not merely about plan plausibility, but about whether a plan is well-typed, structurally admissible, and executable under explicit interface constraints.

Formally, a tool $i$ is modeled as a typed function $f_i : X_i \rightarrow Y_i$, with $\theta_i$ specifying input/output fields, types, and constraints [2605.27957]. A workflow plan $\pi$ is an ordered sequence $P=(s_1,\dots,s_T)$, or equivalently a path in a global DAG $G=(V,E)$, with each step represented as $(step_t, agent_t, inputs_t, outputs_t, dep_t, dep\_content_t)$ [2605.27957]. Plan validity requires type constraints, dependency satisfaction, and execution-grounded consistency over the entire path [2605.27957].

## 2. Benchmark composition and task structure

DisasterBench includes 26 openly available disaster-response tools spanning 13 categories, including perception, prediction, generation, and reasoning [2605.27957]. The benchmark operationalizes a controlled but diverse tool ecosystem. The tool categories and representative tools are enumerated explicitly in the benchmark description.

| Category | Count | Examples |
|---|---:|---|
| Image Generation/Restoration | 4 | Temporal_High_Resolution_Image_Generation; High-Resolution_Image_Reconstructor; Metadata_and_Text_Prompt_Image_Generation; Weather_Degraded_Image_Restoration |
| Classification | 3 | Temporal_Image_Sequence_Classifier; RGB_GeoImage_Classifier; Multi_Spectral_Classifier |
| Anomaly Detection | 3 | Anomaly_Detection_Forest; Landslide_Segmentation; Urban_Anomaly_Detection |
| Object Detection/Segmentation | 3 | Foggy_Scenario_Object_Detection; Low-Light_Object_Detection; Geospatial_Object_Segmentation |
| Building Damage | 1 | Building_damage_assessment |
| Flood/Precipitation | 3 | Precipitation_Nowcasting; Flood_depth_prediction; depth_speed_model |
| Text/Event Processing | 2 | Toponym_Detection; Event_detection |
| Video Analysis | 1 | Video_anomaly_detection |
| Mobility Prediction | 2 | Post_Disaster_Mobility_Recovery; Multimodal_mobility_prediction_under_events |
| Multimodal Reasoning | 1 | GeoChat |
| Data Conversion | 1 | precipitation_data_convert_tool |

The dataset contains 233 tasks, each a natural-language description mapped to a single executable ground-truth plan [2605.27957]. These tasks are distributed across three structural types: Node (single-step, $n=35$, 15.0%), Chain (sequential multi-step, $n=166$, 71.2%), and Branching (fan-out dependencies, $n=32$, 13.7%) [2605.27957]. Plan depths range from 1 to 9, with mean depth 2.53 [2605.27957]. Models are given tool descriptions and schema, but not the DAG, and must infer valid plans from natural-language intent [2605.27957].

The benchmark defines an expert-curated global DAG over the 26 tools with 81 directed typed dependencies organized into four workflow families: Temporal High-Resolution Analysis (33 edges), Image Reconstruction (23), Adverse-Weather Perception (18), and Hydrological Modeling (7) [2605.27957]. The DAG constrains admissible tool compositions during evaluation. A plausible implication is that DisasterBench combines language understanding with latent graph induction over interface-compatible tool chains.

## 3. Typed interfaces and executable workflow constraints

A distinctive feature of DisasterBench is its structured JSON plan schema, enforced by prompts during evaluation [2605.27957]. Each step specifies an `agent`, `step`, `dependence`, `dependence_content`, `inputs`, and `outputs`, with `dependence` equal to `[-1]` if there is no upstream dependency, and otherwise listing the single step index it consumes [2605.27957]. `dependence_content` maps the depended step index to output keys, while generated inputs can reference upstream outputs through placeholders such as `"<GENERATED>-<STEP>-<OutputKey>"` [2605.27957].

Plan validity is defined by three criteria. First, all steps must respect the tool schema $\theta_{agent_t}$, including admissible fields and types [2605.27957]. Second, dependencies must be well-formed: referenced upstream steps must exist, referenced output keys must be present, and the tool pair must be admitted by the DAG [2605.27957]. Third, the entire plan must induce a well-typed path in the global DAG, with consistent input-output binding across steps and no missing or extra steps relative to the ground truth when evaluated [2605.27957].

These interface constraints are not incidental. The benchmark description explicitly states that execution consistency, defined by schemas and dependencies, is orthogonal to semantic grounding [2605.27957]. This is the core technical claim of DisasterBench. It frames disaster-response planning as typed program synthesis over a constrained tool graph rather than as unconstrained chain-of-thought generation.

## 4. Metrics and first-point failure analysis

DisasterBench uses a strict exact-match metric. Given a predicted plan $\hat{S}=(\hat{s}_1,\dots,\hat{s}_{\hat{T}})$ and ground truth $S=(s_1,\dots,s_T)$, exact match is defined as

$$
EM(\hat{S}, S) = \mathbb{1}(\hat{T}=T \land \forall t, \hat{s}_t = s_t),
$$

where step equality jointly checks agent identity, parameter bindings, dependency structure, and dependency content [2605.27957]. The strictness is motivated by the fact that a single incorrect binding may reroute downstream analysis [2605.27957].

To decompose errors, DisasterBench defines partial-credit metrics. Tool Accuracy is

$$
A(\hat{S},S)=\mathbb{1}(\forall t:\hat{s}.agent_t=agent_t \land \hat{s}.step_t=step_t),
$$

Parameter Accuracy is

$$
Param(\hat{S},S)=\mathbb{1}(A=1 \land \forall t:\hat{s}.inputs_t=inputs_t \land \hat{s}.outputs_t=outputs_t),
$$

and Dependency Accuracy is

$$
Dep(\hat{S},S)=\mathbb{1}(A=1 \land \forall t: sort(\hat{s}.dep_t)=sort(dep_t) \land \hat{s}.dep\_content_t \cong dep\_content_t).
$$

The benchmark defines the semantic-execution gap per task as $A-EM$, capturing failures after correct tool selection due to parameter or dependency errors [2605.27957].

For root-cause localization, DisasterBench introduces First-Point-of-Failure (FPoF), defined for a failed prediction $\hat{S}$ relative to $S$ as

$$
t^* = \min \{ t \in [0, \min(\hat{T},T){-}1] \mid \hat{s}_t \neq s_t \}.
$$

If aligned steps match but lengths differ, the failure is length-based, either `early_stop` or `hallucinated_extra_steps` [2605.27957]. FPoF classifies the earliest root cause into tool mismatch (`agent_mismatch`), parameter binding errors (`parameter_error`, `dependency_error`, or `dependency_content_error`), or structural errors (`hallucinated_extra_steps`, `early_stop`, `empty_output`, `format_error`) [2605.27957]. The analyzer scans steps in priority order agent $\rightarrow$ parameters $\rightarrow$ dependencies, then checks length mismatch [2605.27957].

This failure formalization is one of the benchmark’s most important contributions. It operationalizes the difference between a primary planning error and downstream cascades, allowing analysis at the step level rather than only at the whole-plan level [2605.27957].

## 5. Evaluation setup and empirical findings

The benchmark evaluates fourteen LLMs across three capability tiers: Frontier, Strong open-source, and Efficiency-oriented [2605.27957]. The Frontier models are Gemini 3.1 Pro Preview and GPT-5.4; the Strong open-source models include DeepSeek-V3.2, DeepSeek-R1, Qwen3-Max, Qwen3-Max-Thinking, Qwen3.5-27B, Gemma-4-31B, and Llama-3.3-70B; the Efficiency-oriented models are Qwen3.5-9B, Llama-3.1-8B, Ministral-14B, Ministral-8B, and Ministral-3B [2605.27957].

Five planning paradigms are compared: Direct Prompting (DP), Chain-of-Thought (CoT), Tree-of-Thought (ToT), Reasoning via Planning (RAP), and ReAct [2605.27957]. ToT uses breadth-first search with branching factor 7, critic samples 5, beam 1, and search depth 2 [2605.27957]. RAP uses Monte Carlo Tree Search over tools with 5 rollouts, max depth 5, subquestion samples 16, confidence samples 32, and reward $R=r_0^\alpha \cdot r_1^{1-\alpha}$ with $\alpha=0.5$ [2605.27957]. Deterministic decoding is used for DP, CoT, and ReAct; mild stochasticity (0.5) is used for ToT and RAP [2605.27957].

Overall LLM averages in exact-match accuracy are DP 44.91%, CoT 51.44%, ToT 46.87%, RAP 41.20%, and ReAct 54.29% [2605.27957]. No method dominates universally [2605.27957]. The best model-method combination is Gemini 3.1 Pro Preview with ToT at 73.39% [2605.27957]. GPT-5.4 peaks at RAP 65.67% [2605.27957]. Efficiency-tier models often prefer ReAct; for example, Ministral-14B reaches 50.64% versus CoT 33.48%, a +17 percentage-point difference [2605.27957].

Three empirical findings are emphasized. First, planning method effectiveness depends strongly on model capacity [2605.27957]. Tier differences exceed method differences, indicating that execution consistency is capacity-constrained [2605.27957]. Second, depth is a bottleneck: accuracy drops sharply beyond depth 2, and even Gemini falls to roughly 53–61% at depth 4 across methods [2605.27957]. Third, tool mismatch and parameter-binding errors dominate first failures, indicating distinct bottlenecks in semantic grounding and execution consistency [2605.27957].

## 6. Error taxonomy, instruction clash, and practical implications

Across 8,523 failures, FPoF reveals distinct error distributions by method [2605.27957]. For DP, tool mismatch accounts for 50.4%, Parameter Binding 34.6%, and Structural 15.0% ($n=1{,}797$) [2605.27957]. For CoT, the respective figures are 54.6%, 31.7%, and 13.7% ($n=1{,}584$) [2605.27957]. For ToT they are 52.3%, 31.0%, and 16.7% ($n=1{,}733$) [2605.27957]. RAP is distinctive, with Parameter Binding overtaking tool mismatch: 32.2%, 50.1%, and 17.7% ($n=1{,}918$) [2605.27957]. ReAct shows 58.2%, 28.7%, and 13.1% ($n=1{,}491$) [2605.27957]. Tool mismatch dominates for four methods, while RAP shows elevated parameter-binding failure and an `early_stop` rate of 11.1% of RAP failures, suggesting search truncation when models mis-evaluate partial plans [2605.27957].

The benchmark also quantifies the semantic-execution gap. Averaged across models, $A-EM$ equals 8.2 percentage points for DP, 6.7 for CoT, 9.0 for ToT, 7.9 for RAP, and 6.8 for ReAct [2605.27957]. A notable case is Llama-3.3-70B under DP, where Tool Accuracy is 60.94% but EM is 38.63%, a 22.31-point gap [2605.27957]. This directly demonstrates that correct tool selection does not imply executable planning.

Another major result concerns instruction clash. DisasterBench reports that verbose intermediate reasoning can conflict with schema-constrained generation [2605.27957]. DeepSeek-V3.2 under CoT drops from 57.94% in DP to 36.05%, a decrease of 21.89 percentage points, with FPoF shifting toward Parameter Binding errors [2605.27957]. Enforcing JSON mode rescues EM to 62.23% and restores a tool mismatch profile, confirming the clash hypothesis [2605.27957]. This suggests that free-form reasoning can degrade structured-output compliance when output control is not externally enforced.

The paper provides a canonical hydrological modeling example: `Precipitation_Nowcasting → precipitation_data_convert_tool → Flood_depth_prediction → depth_speed_model → Traffic Speed` [2605.27957]. A predicted plan that skips the conversion step and applies `Flood_depth_prediction` directly to `Predicted Precipitation` fails at the first incompatible step, as either `agent_mismatch` or `parameter_error` [2605.27957]. This example illustrates how FPoF localizes the earliest divergence in a typed workflow.

## 7. Position within the disaster-benchmark landscape

DisasterBench is explicitly compared with AgentBench, ToolBench, APIBench, TaskBench, SWE-bench, and GAIA [2605.27957]. The distinction drawn is that prior tool-use benchmarks emphasize tool selection and multi-step orchestration, but largely treat agents as freely composable [2605.27957]. DisasterBench instead foregrounds typed interface constraints and a global execution DAG, evaluates executable grounding across semantically overlapping tools, and adds step-level FPoF localization to distinguish root cause from cascaded symptoms [2605.27957].

Within the broader disaster-benchmark ecosystem, other works cover adjacent but distinct axes. DisastQA provides a large-scale benchmark of 3,000 questions spanning eight disaster types, with Base, Mix, and Golden evidence conditions and a human-verified keypoint-based evaluation protocol for open-ended factual completeness [2601.03670]. DisasterInsight restructures xBD into approximately 112K building-centered instances and supports building-function classification, damage-level and disaster-type classification, counting, and structured report generation for disaster imagery [2601.18493]. DisasterVQA introduces 1,395 real-world images and 4,405 expert-curated question-answer pairs for disaster-scene VQA grounded in FEMA ESF and OCHA MIRA [2601.13839]. GeoDisaster focuses on operational geospatial disaster reasoning with 2,921 verified instances across 43 question types and five task families, using executable geospatial workflows and orchestrated agents [2606.17246]. These benchmarks address QA, VLM evaluation, geospatial reasoning, and multimodal assessment rather than typed workflow planning itself.

A separate record titled "DisasterBench: A Multimodal Benchmark for UAV-Based Disaster Response in Complex Environments" claims a multi-stage multimodal benchmark and a lightweight model named DisasterVL, but the available paper content explicitly states that dataset descriptions, tasks, metrics, model details, and results are not provided in the supplied document [2606.06217]. Accordingly, no concrete technical account of that benchmark can be established from the available contents.

DisasterBench’s principal significance lies in exposing what it describes as “a fundamental gap between semantic reasoning and execution-grounded coordination” in disaster-response workflows [2605.27957]. Its formal metrics, typed DAG, and FPoF analysis make it a benchmark for schema-aware planning rather than for perception, retrieval, or answer generation alone. A plausible implication is that it provides a canonical testbed for studying whether LLM-based disaster-response systems can move from plausible language outputs to operationally valid, executable coordination.

Source: https://www.emergentmind.com/topics/disasterbench