---
title: 'AssetOpsBench: Industrial AI Benchmark'
url: https://www.emergentmind.com/topics/assetopsbench
type: topic
---

# AssetOpsBench: Industrial AI Benchmark

AssetOpsBench is a benchmark framework and execution environment for developing, orchestrating, and evaluating AI agents on industrial asset operations and maintenance tasks. It is presented as a benchmark for real-world industrial asset management tasks, aimed at workflow automation across the asset lifecycle rather than isolated point problems, and it combines a multi-source industrial dataset, 141 human-authored scenarios, and an evaluation framework for single-agent and multi-agent systems in a simulated industrial environment [2506.03828].

## 1. Scope, motivation, and conceptual position

AssetOpsBench is situated in **Industry 4.0 asset lifecycle management**. Its motivating claim is that traditional AI/ML in industrial operations has largely focused on narrow, isolated tasks such as anomaly detection, forecasting, or maintenance recommendation as standalone models, whereas real industrial operations require coordination across time-series telemetry, failure mode knowledge, maintenance histories and work orders, alarms and event logs, asset metadata, and operational context. The benchmark is therefore organized around the prospect of **end-to-end automation across the asset lifecycle**, including perception, reasoning, planning, delegation, and action recommendation [2506.03828].

The benchmark’s task hierarchy is aligned to four broad stages: **Asset Configuration**, **Model Selection and Analysis**, **Monitoring and Execution**, and **Maintenance and Response**. The covered tasks include condition monitoring, anomaly detection, root cause analysis, fault explanation, predictive maintenance planning, sensor-to-failure mapping, work order retrieval and summarization, work order bundling, intervention scheduling, service request initiation, asset configuration, KPI identification, forecasting, model tuning, alert review, system health summarization, and maintenance prioritization. This positioning is important because AssetOpsBench is not framed as a narrow question-answering corpus; it is framed as a workflow benchmark over industrial maintenance and operational decision processes [2506.03828].

The benchmark is centered on industrial assets such as **data center chillers** and **air handling units (AHUs)**, while also referencing other industrial assets such as **wind farms** and **standby generators** as possible generalization targets. It is intended for researchers building industrial AI agents and multi-agent systems, benchmark designers interested in domain-specific agent evaluation, and practitioners developing industrial AI workflows [2506.03828].

A recurrent misconception is that AssetOpsBench is merely a maintenance QA benchmark. The benchmark definition does not support that reading. Its core object is **agentic workflow orchestration** over heterogeneous industrial data and tools, with both single-agent and multi-agent settings, rather than one-shot answer generation [2506.03828].

## 2. Architecture, specialized agents, and industrial environment

AssetOpsBench centers on a global coordinator called the **AssetOps Agent** or **AssetOpsAgent**, which interprets high-level user requests and coordinates specialized sub-agents. The benchmark highlights four specialist agents: **IoT Agent**, **FMSR Agent** (Failure Mode Sensor Relations), **TSFM Agent** (Time Series Foundation Model), and **WO Agent** (Work Order Agent). The high-level workflow is: user issues a natural language query; the global coordinator interprets it; the query is decomposed into subtasks; subtasks are delegated to specialized agents; and outputs are combined into a final response [2506.03828].

The appendix formalizes the benchmark’s **Agent-Oriented Problem (AOP)** as
$$
\text{AOP} = \langle \mathcal{A}, \mathcal{T}, \Pi, M, O \rangle
$$
where $\mathcal{A}$ is the set of available agents, $\mathcal{T}$ the task set, $\Pi$ the hierarchical plan space, $M$ the memory system, and $O$ the output space. A task is written as
$$
\tau = \langle g, \mathcal{M}, C \rangle
$$
with task goal $g$, required modalities $\mathcal{M}$, and domain-specific or operational constraints $C$. A plan is represented as
$$
\pi = \left[ \langle \tau_1, A_i \rangle, \langle \tau_2, A_j \rangle, \ldots \right]
$$
and memory as
$$
M = \{(k_i, v_i)\}_{i=1}^m .
$$
This formalization makes explicit that AssetOpsBench evaluates not only final outputs but also planning, coordination, and memory-bearing execution [2506.03828].

The environment is a **simulated industrial environment** based on a **CouchDB-backed IoT telemetry system** and multi-source industrial datasets. The **IoT Agent** provides structured access to site lists, asset metadata, historical telemetry, and selected sensors over selected time windows. The **FMSR Agent** operates over structured failure mode knowledge and sensor-failure mappings derived from FMEA-style records. The **TSFM Agent** performs forecasting, anomaly detection, model tuning, and hybrid prediction/anomaly tasks. The **WO Agent** operates over work orders, alerts, anomalies, ISO failure codes, alert-rule mappings, anomaly-to-failure mappings, and equipment metadata, and supports retrieval, summarization, scheduling, causal review, predictive maintenance recommendation, and bundling [2506.03828].

AssetOpsBench compares two orchestration paradigms. In **Agents-As-Tools / Tool-As-Agent**, a top-level ReAct agent invokes specialist agents as tools, decomposition occurs during execution, and execution stops either when a Review component judges successful completion or when the system reaches the maximum ten iterations. In **Plan-and-Execute**, a **Planner**, **Reviewer**, **Orchestrator**, specialist agents, and a **Summarization module** are separated: the planner decomposes the query, the reviewer vets tasks, the orchestrator assigns them, agents execute independently, and the summarizer combines outputs. The comparison is explicitly characterized as interleaved decomposition versus decomposition-first [2506.03828].

## 3. Scenario inventory, data resources, and evaluation protocol

AssetOpsBench contains **141 scenarios** in its core release: **99 single-agent scenarios** and **42 multi-agent scenarios**. Each scenario is represented as
$$
P = \langle \mathit{id}, \mathit{type}, \mathit{text}, \mathit{category}, \mathit{form} \rangle
$$
where $id$ is a unique identifier, $type$ the task type, $text$ the natural language query, $category$ the operational domain, and $form$ the expected output form or characteristic form. The benchmark also describes four capability dimensions for scenarios: **Tool-Centric**, **Skill-Centric**, **Domain-Centric**, and **LLM-Centric** [2506.03828].

The underlying industrial data are heterogeneous. The sensor telemetry component contains **6 assets**—**4 chillers** and **2 AHUs**—with **2.3 million sensor data points** collected at **15-minute intervals** from HVAC systems. The FMEA/failure component covers **3 industrial assets** and **53 failure records**. The work-order component contains **4.2K records** over **10+ assets**; the appendix further enumerates related business objects including **4392 WorkOrder**, **6929 Event**, **1995 Alert Events**, **542 Anomaly Events**, **137 ISO Failure Code**, **68 ISO Primary Failure Code**, **77 AlertRule**, **22 Equipment**, **46 Alert-Rule Mapping**, and **12 Anomaly Mapping** [2506.03828].

Evaluation is trajectory-based. The evaluation agent takes the original query, the agent trajectory, and the characteristic form:
$$
f : (\mathcal{Q}, \mathcal{T}, \mathcal{C}) \rightarrow \mathbf{y} = (y_1, y_2, \dots, y_6, y_{\text{text}}, y_{\text{skip}})
$$
and returns six scored dimensions, free-form textual feedback, and a skip flag. The six scored dimensions are **Task Completeness**, **Data Retrieval Accuracy**, **Result Verification**, **Agent Sequence**, **Clarity and Justification**, and **Hallucinations**. The paper does not provide a single aggregate benchmark score formula over these six dimensions [2506.03828].

The benchmark uses an **LLM-as-judge** protocol. After human validation on a sample of 40 tasks evaluated by 4 domain experts, the authors selected **llama-4-maverick** as the evaluation model because it aligned best with expert ratings: **75% accuracy, Cohen’s $\kappa = 0.55$**, compared with **69% accuracy, Cohen’s $\kappa = 0.44$** for **gpt-4.1**. The reported default setup runs the main AssetOpsAgent **once per task**, the evaluation agent **5 times**, with generation temperature **0** for the AssetOpsAgent and **0.3** for the evaluation agent. The paper also distinguishes **Pass$^k$** from Pass@k, arguing that Pass$^k$—success on all $k$ attempts—is more appropriate for industrial settings where retries may be infeasible [2506.03828].

## 4. Baseline systems, empirical findings, and failure modes

The benchmark evaluates multiple LLMs under the two orchestration paradigms. In **Tool-As-Agent / Agents-As-Tools**, **gpt-4.1** is reported as the strongest model across nearly all metrics, with **Task completion 65%**, **Data retrieval accuracy 77%**, **Result verification 65%**, **Agent sequence 67%**, **Clarity and justification 87%**, and **Hallucination rate 6%**. **llama-4-maverick** is also competitive, with **Task completion 59%**, **Data retrieval accuracy 57%**, **Result verification 60%**, **Agent sequence 74%**, **Clarity and justification 78%**, and **Hallucination rate 11%**. A smaller model such as **granite-3-3-8b** trails materially, with **Task completion 35%**, **Data retrieval accuracy 43%**, **Result verification 37%**, **Agent sequence 58%**, **Clarity and justification 66%**, and **Hallucination rate 30%** [2506.03828].

In **Plan-and-Execute**, the strongest reported model is **mistral-large**, with **Task completion 46.5%** and **Agent sequencing 58.1%**; **llama-4-maverick** is reported at **Result verification 44%**, **Clarity and justification 57%**, and **Hallucination rate 47%**. The paper’s main architectural conclusion is therefore a quality-efficiency tradeoff: **Agents-as-Tools** yields better task quality, retrieval correctness, verification, and lower hallucination, while **Plan-and-Execute** uses fewer steps and shows more runtime efficiency and lower variability [2506.03828].

Runtime measurements reinforce that distinction. For **gpt-4.1** in Agents-As-Tools, the paper reports **$6.0 \pm 2.4$ steps** and **$104 \pm 178$ sec** for single-agent tasks, versus **$6.4 \pm 2.5$ steps** and **$218 \pm 371$ sec** for multi-agent tasks. In Plan-and-Execute, the same model uses **$2.6 \pm 1.0$ steps** and **$93.3 \pm 105.6$ sec** for single-agent tasks, and **$2.9 \pm 1.5$ steps** and **$180.2 \pm 122.6$ sec** for multi-agent tasks. This suggests that AssetOpsBench is not only an accuracy benchmark but also a workload for measuring orchestration cost [2506.03828].

Two ablations are especially consequential. First, when the authors injected **10 out-of-domain distractor agents** such as **SREAgent**, **EchoAgent**, **CustomerSupportAgent**, **HRPolicyAgent**, **FrontendDevAgent**, **EdgeMLAgent**, and **RULPredictorAgent**, performance on **99 single-agent scenarios** improved rather than degraded. Second, on **65 single-agent tasks** from IoT, FMSR, and TSFM, removing in-context examples caused very large collapses: **gpt-4.1** fell from about **80%** average performance to **34%**, and **granite-3-8b** fell from about **60%** to **3%**. The paper treats this as direct evidence that in-context examples are essential for ReAct-style industrial agents in the benchmark [2506.03828].

The failure-mode analysis uses **881 execution trajectories** from Agents-as-Tools. It reports **185 trajectories** with one novel failure mode beyond the adopted taxonomy and **164 trajectories** with two emergent issues. Frequently observed emergent failures include **Overstatement of Task Completion** with **122 cases (23.8%)**, **Extraneous or Ambiguous Output Formatting** with **110 cases (21.4%)**, and **Ineffective Error Recovery** with **160 cases (31.2%)**. Additional clustered failures include **Lack of Error Handling for Tool Failure** with **53 cases (10.3%)**, **Failure to Incorporate Feedback** with **41 cases (8.0%)**, and **Invalid Action Formatting** with **27 cases (5.3%)**. This supports the benchmark’s claim that fixed failure taxonomies are inadequate for modern industrial multi-agent systems [2506.03828].

## 5. AssetOpsBench-Live, the CODS 2025 challenge, and leaderboard interpretation

A privacy-aware online deployment of the benchmark was released as **AssetOpsBench-Live**, and the **CODS 2025 AssetOpsBench challenge** built on that deployment. The retrospective distinguishes three layers: **AssetOpsBench** as the underlying benchmark, **AssetOpsBench-Live** as the privacy-preserving Codabench deployment with hidden evaluation, and the **CODS 2025 challenge** as the concrete competition instance. The online deployment preserved industrial confidentiality through Dockerized evaluation, hidden data, restricted editable regions, and Codabench-mediated execution [2605.08518].

The competition had two tracks. **Track 1: Planning** fixed the executor and allowed modifications to prompt construction and agent formatting in `track1_planning.py`; the intended output was a **DAG over domain agents**. **Track 2: Execution** fixed the planner and allowed edits to workflow logic in `track2_execution.py`, including parallel or conditional execution paths, multi-agent collaboration per task, context aggregation, fallback, cleanup, and validation behavior. All submissions used a **fixed LLaMA-3-70B model**. Evaluation had three stages: an optional local warm-up on 2–3 scenarios, **Phase 1** on **11 scenarios** drawn from the public pool, and **Phase 2** on **11 hidden scenarios** from unseen asset classes [2605.08518].

The official scoring formulas were
$$
C_t = 0.6\,S^{\text{pub}}_t + 0.3\,S^{\text{priv}}_t + 0.1\,\tau_t,\quad
F = 0.4\,C_{\text{plan}} + 0.6\,C_{\text{exec}},\quad
\Delta = S_{\text{priv}} - S_{\text{pub}},
$$
where $S^{\text{pub}}_t$ and $S^{\text{priv}}_t$ are public and hidden scores for track $t$, and $\tau_t$ is semantic **t-match**. The retrospective’s central methodological critique is that $\tau_t$ was combined on a **0–1** scale with public and private percentages on a **0–100** scale, making the t-match term **numerically almost inert**. The paper reports that removing t-match barely changed rankings, while rescaling t-match by 100 would swap the top two teams and also swap third and fourth place [2605.08518].

The leaderboard findings are striking. The public planning leaderboard saturated at **72.73%**, with four teams reaching that value, and the public board had only **8 distinct positive public scores across 20 teams**. Hidden evaluation changed the story unevenly across tracks: in planning, public and hidden scores had moderate positive correlation at about **0.69**; in execution, public and hidden scores had **$r=-0.13$**. The retrospective highlights execution-track cases where systems with **45.45%** public score achieved **63.64%** on hidden evaluation, while the top public execution system dropped from **72.73** to **45.45** on the hidden set. The authors therefore argue that the execution leaderboard did not straightforwardly measure “best industrial multi-agent system”; it mostly rewarded **guardrails**, **response selection**, **contamination cleanup**, **fallback**, and **context control**, rather than novel agent architectures [2605.08518].

The challenge also exposed severe attrition. The retrospective analyzes **149 registered teams**, **349 declared member slots**, and **300 submission attempts**, but only **24 teams with non-zero public scores** and **11 fully ranked teams**. It further reports that **78/149 teams (52.3%)** listed multiple Codabench usernames, implying that the competition was operationally account-based but substantively team-based. This became part of the paper’s broader recommendation that future benchmark releases treat team metadata, exact evaluated artifacts, and container digests as first-class benchmark outputs [2605.08518].

## 6. Subsequent extensions, reinterpretations, and benchmark-centered ecosystem

AssetOpsBench has rapidly become a shared testbed for work on industrial agent design, orchestration, and evaluation. A central reinterpretation appears in **“Knowledge Graphs as the Missing Data Layer for LLM-Based Industrial Asset Operations”**, which treats AssetOpsBench not as a replacement target but as a fixed scenario set for testing whether the data representation behind the tools is the main bottleneck. On a **139-scenario v1 snapshot** of the benchmark, that paper reports **91/139 (65%)** for the original document-based baseline, **114–116/139 (82–83%)** for LLM-generated Cypher over a typed knowledge graph, and **137/139 (99%)** for deterministic graph handlers. It also adds **40 graph-native scenarios** and evaluates an expanded HuggingFace release of **467 scenarios** across **6 domains**, where deterministic handlers reach **467/467 (100%)** with average score **0.848**. The paper’s stated conclusion is that, for structured industrial operations, the **data layer** may be the primary bottleneck rather than orchestration alone [2605.26874].

A second line of work uses AssetOpsBench as a tool-planning benchmark. **“Internalizing Tool Knowledge in Small Language Models via QLoRA Fine-Tuning”** focuses on the **152 human-generated queries defined in the MCP setting** and asks whether a small model can internalize a fixed tool catalog well enough to plan without having tool schemas in the prompt. Using approximately **1,700** supervised examples derived from AssetOpsBench and a catalog of **5 agents and 23 tools**, the paper reports that a fine-tuned **Gemma 4 E4B-it** reaches **AT-F1 = 0.65** and **overall judge score = 3.88**, compared with **0.47** and **2.88** for the informed unfine-tuned baseline, while reducing input length by **82.6%**. A fine-tuned **Qwen3-4B** reaches **AT-F1 = 0.605** and **overall judge score = 3.78**, while using **62% less memory** and running **2.49× faster** than Gemma in the reported setup [2605.17774].

Several systems papers use AssetOpsBench primarily as an orchestration and latency testbed. **DynAMO** treats benchmark queries as schema-constrained workflow graphs over **IoT**, **TSFM**, **FMSR**, and **WO** agents, reporting that across **141 industrial queries** the plans contain **3–5 tasks on average** and about **40%** have at least one parallelizable branch. Its **ParallelWorkflow** reduces median latency from **308.53 s** to **190.49 s**, approximately **1.62×**, and context pruning lowers inference latency by about **30%** while maintaining correct functional behaviour under controlled fault injection [2606.19382]. **“Evaluating Temporal Semantic Caching and Workflow Optimization in Agentic Plan-Execute Pipelines”** studies a related optimization surface over AssetOpsBench’s MCP-backed pipelines, reporting **1.67×** end-to-end speedup and about **40.0%** median latency reduction from workflow optimizations, while a temporal semantic cache yields a median **30.6×** speedup on cache hits but also exposes a failure mode of pure semantic caching for parameter-rich industrial queries [2605.20630].

The broader AssetOpsBench open-source effort has also hosted adjacent systems rather than only the core benchmark. **IndustryAssetEQA** is explicitly presented as a neurosymbolic operational intelligence system for embodied question answering in industrial maintenance, released under the AssetOpsBench GitHub tree. It evaluates four industrial datasets—rotating machinery, turbofan engines, hydraulic systems, and cyber-physical production systems—and reports that, relative to LLM-only baselines, the full system improves **structural validity by up to 0.51**, **counterfactual accuracy by up to 0.47**, and **explanation entailment by 0.64**, while reducing severe expert-rated overclaims from **28%** to **2%**. This does not redefine AssetOpsBench’s core scenario benchmark, but it shows that the repository has become a broader benchmark-and-method scaffold for industrial operational intelligence [2604.23446].

Taken together, these follow-on studies suggest that AssetOpsBench functions simultaneously as a benchmark, an execution environment, and a controlled systems testbed. They also sharpen two methodological points. First, measured performance depends strongly on what is being varied: orchestration policy, tool descriptions, data substrate, or caching logic. Second, leaderboard-style results should not be read as a complete measure of industrial autonomy. The benchmark has repeatedly been used to show that apparently agentic limitations can instead arise from prompt format, hidden-evaluation design, data modeling choices, or backend orchestration assumptions [2605.26874].

Source: https://www.emergentmind.com/topics/assetopsbench