AssetOpsBench: Industrial AI Benchmark

• AssetOpsBench is a benchmark framework and simulation environment designed to support end-to-end industrial asset lifecycle management with integrated AI agents.
• It orchestrates both single-agent and multi-agent systems using specialized agents (IoT, FMSR, TSFM, WO) to execute and evaluate complex maintenance workflows.
• Empirical findings reveal a trade-off between task accuracy and orchestration efficiency, highlighting the crucial role of in-context examples and workflow optimization.

AssetOpsBench is a benchmark framework and execution environment for developing, orchestrating, and evaluating AI agents on industrial asset operations and maintenance tasks. It is presented as a benchmark for real-world industrial asset management tasks, aimed at workflow automation across the asset lifecycle rather than isolated point problems, and it combines a multi-source industrial dataset, 141 human-authored scenarios, and an evaluation framework for single-agent and multi-agent systems in a simulated industrial environment (Patel et al., 4 Jun 2025).

1. Scope, motivation, and conceptual position

AssetOpsBench is situated in Industry 4.0 asset lifecycle management. Its motivating claim is that traditional AI/ML in industrial operations has largely focused on narrow, isolated tasks such as anomaly detection, forecasting, or maintenance recommendation as standalone models, whereas real industrial operations require coordination across time-series telemetry, failure mode knowledge, maintenance histories and work orders, alarms and event logs, asset metadata, and operational context. The benchmark is therefore organized around the prospect of end-to-end automation across the asset lifecycle, including perception, reasoning, planning, delegation, and action recommendation (Patel et al., 4 Jun 2025).

The benchmark’s task hierarchy is aligned to four broad stages: Asset Configuration, Model Selection and Analysis, Monitoring and Execution, and Maintenance and Response. The covered tasks include condition monitoring, anomaly detection, root cause analysis, fault explanation, predictive maintenance planning, sensor-to-failure mapping, work order retrieval and summarization, work order bundling, intervention scheduling, service request initiation, asset configuration, KPI identification, forecasting, model tuning, alert review, system health summarization, and maintenance prioritization. This positioning is important because AssetOpsBench is not framed as a narrow question-answering corpus; it is framed as a workflow benchmark over industrial maintenance and operational decision processes (Patel et al., 4 Jun 2025).

The benchmark is centered on industrial assets such as data center chillers and air handling units (AHUs), while also referencing other industrial assets such as wind farms and standby generators as possible generalization targets. It is intended for researchers building industrial AI agents and multi-agent systems, benchmark designers interested in domain-specific agent evaluation, and practitioners developing industrial AI workflows (Patel et al., 4 Jun 2025).

A recurrent misconception is that AssetOpsBench is merely a maintenance QA benchmark. The benchmark definition does not support that reading. Its core object is agentic workflow orchestration over heterogeneous industrial data and tools, with both single-agent and multi-agent settings, rather than one-shot answer generation (Patel et al., 4 Jun 2025).

2. Architecture, specialized agents, and industrial environment

AssetOpsBench centers on a global coordinator called the AssetOps Agent or AssetOpsAgent, which interprets high-level user requests and coordinates specialized sub-agents. The benchmark highlights four specialist agents: IoT Agent, FMSR Agent (Failure Mode Sensor Relations), TSFM Agent (Time Series Foundation Model), and WO Agent (Work Order Agent). The high-level workflow is: user issues a natural language query; the global coordinator interprets it; the query is decomposed into subtasks; subtasks are delegated to specialized agents; and outputs are combined into a final response (Patel et al., 4 Jun 2025).

The appendix formalizes the benchmark’s Agent-Oriented Problem (AOP) as

$$\text{AOP} = \langle \mathcal{A}, \mathcal{T}, \Pi, M, O \rangle$$

where $\mathcal{A}$ is the set of available agents, $\mathcal{T}$ the task set, $\Pi$ the hierarchical plan space, $M$ the memory system, and $O$ the output space. A task is written as

$\tau = \langle g, \mathcal{M}, C \rangle$

with task goal $g$, required modalities $\mathcal{M}$, and domain-specific or operational constraints $C$. A plan is represented as

$\mathcal{A}$0

and memory as

$\mathcal{A}$1

This formalization makes explicit that AssetOpsBench evaluates not only final outputs but also planning, coordination, and memory-bearing execution (Patel et al., 4 Jun 2025).

The environment is a simulated industrial environment based on a CouchDB-backed IoT telemetry system and multi-source industrial datasets. The IoT Agent provides structured access to site lists, asset metadata, historical telemetry, and selected sensors over selected time windows. The FMSR Agent operates over structured failure mode knowledge and sensor-failure mappings derived from FMEA-style records. The TSFM Agent performs forecasting, anomaly detection, model tuning, and hybrid prediction/anomaly tasks. The WO Agent operates over work orders, alerts, anomalies, ISO failure codes, alert-rule mappings, anomaly-to-failure mappings, and equipment metadata, and supports retrieval, summarization, scheduling, causal review, predictive maintenance recommendation, and bundling (Patel et al., 4 Jun 2025).

AssetOpsBench compares two orchestration paradigms. In Agents-As-Tools / Tool-As-Agent, a top-level ReAct agent invokes specialist agents as tools, decomposition occurs during execution, and execution stops either when a Review component judges successful completion or when the system reaches the maximum ten iterations. In Plan-and-Execute, a Planner, Reviewer, Orchestrator, specialist agents, and a Summarization module are separated: the planner decomposes the query, the reviewer vets tasks, the orchestrator assigns them, agents execute independently, and the summarizer combines outputs. The comparison is explicitly characterized as interleaved decomposition versus decomposition-first (Patel et al., 4 Jun 2025).

3. Scenario inventory, data resources, and evaluation protocol

AssetOpsBench contains 141 scenarios in its core release: 99 single-agent scenarios and 42 multi-agent scenarios. Each scenario is represented as

$\mathcal{A}$2

where $\mathcal{A}$3 is a unique identifier, $\mathcal{A}$4 the task type, $\mathcal{A}$5 the natural language query, $\mathcal{A}$6 the operational domain, and $\mathcal{A}$7 the expected output form or characteristic form. The benchmark also describes four capability dimensions for scenarios: Tool-Centric, Skill-Centric, Domain-Centric, and LLM-Centric (Patel et al., 4 Jun 2025).

The underlying industrial data are heterogeneous. The sensor telemetry component contains 6 assets—4 chillers and 2 AHUs—with 2.3 million sensor data points collected at 15-minute intervals from HVAC systems. The FMEA/failure component covers 3 industrial assets and 53 failure records. The work-order component contains 4.2K records over 10+ assets; the appendix further enumerates related business objects including 4392 WorkOrder, 6929 Event, 1995 Alert Events, 542 Anomaly Events, 137 ISO Failure Code, 68 ISO Primary Failure Code, 77 AlertRule, 22 Equipment, 46 Alert-Rule Mapping, and 12 Anomaly Mapping (Patel et al., 4 Jun 2025).

Evaluation is trajectory-based. The evaluation agent takes the original query, the agent trajectory, and the characteristic form:

$\mathcal{A}$8

and returns six scored dimensions, free-form textual feedback, and a skip flag. The six scored dimensions are Task Completeness, Data Retrieval Accuracy, Result Verification, Agent Sequence, Clarity and Justification, and Hallucinations. The paper does not provide a single aggregate benchmark score formula over these six dimensions (Patel et al., 4 Jun 2025).

The benchmark uses an LLM-as-judge protocol. After human validation on a sample of 40 tasks evaluated by 4 domain experts, the authors selected llama-4-maverick as the evaluation model because it aligned best with expert ratings: 75% accuracy, Cohen’s $\mathcal{A}$9, compared with 69% accuracy, Cohen’s $\mathcal{T}$0 for gpt-4.1. The reported default setup runs the main AssetOpsAgent once per task, the evaluation agent 5 times, with generation temperature 0 for the AssetOpsAgent and 0.3 for the evaluation agent. The paper also distinguishes Pass$\mathcal{T}$1 from Pass@k, arguing that Pass$\mathcal{T}$2—success on all $\mathcal{T}$3 attempts—is more appropriate for industrial settings where retries may be infeasible (Patel et al., 4 Jun 2025).

4. Baseline systems, empirical findings, and failure modes

The benchmark evaluates multiple LLMs under the two orchestration paradigms. In Tool-As-Agent / Agents-As-Tools, gpt-4.1 is reported as the strongest model across nearly all metrics, with Task completion 65%, Data retrieval accuracy 77%, Result verification 65%, Agent sequence 67%, Clarity and justification 87%, and Hallucination rate 6%. llama-4-maverick is also competitive, with Task completion 59%, Data retrieval accuracy 57%, Result verification 60%, Agent sequence 74%, Clarity and justification 78%, and Hallucination rate 11%. A smaller model such as granite-3-3-8b trails materially, with Task completion 35%, Data retrieval accuracy 43%, Result verification 37%, Agent sequence 58%, Clarity and justification 66%, and Hallucination rate 30% (Patel et al., 4 Jun 2025).

In Plan-and-Execute, the strongest reported model is mistral-large, with Task completion 46.5% and Agent sequencing 58.1%; llama-4-maverick is reported at Result verification 44%, Clarity and justification 57%, and Hallucination rate 47%. The paper’s main architectural conclusion is therefore a quality-efficiency tradeoff: Agents-as-Tools yields better task quality, retrieval correctness, verification, and lower hallucination, while Plan-and-Execute uses fewer steps and shows more runtime efficiency and lower variability (Patel et al., 4 Jun 2025).

Runtime measurements reinforce that distinction. For gpt-4.1 in Agents-As-Tools, the paper reports $\mathcal{T}$4 steps and $\mathcal{T}$5 sec for single-agent tasks, versus $\mathcal{T}$6 steps and $\mathcal{T}$7 sec for multi-agent tasks. In Plan-and-Execute, the same model uses $\mathcal{T}$8 steps and $\mathcal{T}$9 sec for single-agent tasks, and $\Pi$0 steps and $\Pi$1 sec for multi-agent tasks. This suggests that AssetOpsBench is not only an accuracy benchmark but also a workload for measuring orchestration cost (Patel et al., 4 Jun 2025).

Two ablations are especially consequential. First, when the authors injected 10 out-of-domain distractor agents such as SREAgent, EchoAgent, CustomerSupportAgent, HRPolicyAgent, FrontendDevAgent, EdgeMLAgent, and RULPredictorAgent, performance on 99 single-agent scenarios improved rather than degraded. Second, on 65 single-agent tasks from IoT, FMSR, and TSFM, removing in-context examples caused very large collapses: gpt-4.1 fell from about 80% average performance to 34%, and granite-3-8b fell from about 60% to 3%. The paper treats this as direct evidence that in-context examples are essential for ReAct-style industrial agents in the benchmark (Patel et al., 4 Jun 2025).

The failure-mode analysis uses 881 execution trajectories from Agents-as-Tools. It reports 185 trajectories with one novel failure mode beyond the adopted taxonomy and 164 trajectories with two emergent issues. Frequently observed emergent failures include Overstatement of Task Completion with 122 cases (23.8%), Extraneous or Ambiguous Output Formatting with 110 cases (21.4%), and Ineffective Error Recovery with 160 cases (31.2%). Additional clustered failures include Lack of Error Handling for Tool Failure with 53 cases (10.3%), Failure to Incorporate Feedback with 41 cases (8.0%), and Invalid Action Formatting with 27 cases (5.3%). This supports the benchmark’s claim that fixed failure taxonomies are inadequate for modern industrial multi-agent systems (Patel et al., 4 Jun 2025).

5. AssetOpsBench-Live, the CODS 2025 challenge, and leaderboard interpretation

A privacy-aware online deployment of the benchmark was released as AssetOpsBench-Live, and the CODS 2025 AssetOpsBench challenge built on that deployment. The retrospective distinguishes three layers: AssetOpsBench as the underlying benchmark, AssetOpsBench-Live as the privacy-preserving Codabench deployment with hidden evaluation, and the CODS 2025 challenge as the concrete competition instance. The online deployment preserved industrial confidentiality through Dockerized evaluation, hidden data, restricted editable regions, and Codabench-mediated execution (Patel et al., 8 May 2026).

The competition had two tracks. Track 1: Planning fixed the executor and allowed modifications to prompt construction and agent formatting in track1_planning.py; the intended output was a DAG over domain agents. Track 2: Execution fixed the planner and allowed edits to workflow logic in track2_execution.py, including parallel or conditional execution paths, multi-agent collaboration per task, context aggregation, fallback, cleanup, and validation behavior. All submissions used a fixed LLaMA-3-70B model. Evaluation had three stages: an optional local warm-up on 2–3 scenarios, Phase 1 on 11 scenarios drawn from the public pool, and Phase 2 on 11 hidden scenarios from unseen asset classes (Patel et al., 8 May 2026).

The official scoring formulas were

$\Pi$2

where $\Pi$3 and $\Pi$4 are public and hidden scores for track $\Pi$5, and $\Pi$6 is semantic t-match. The retrospective’s central methodological critique is that $\Pi$7 was combined on a 0–1 scale with public and private percentages on a 0–100 scale, making the t-match term numerically almost inert. The paper reports that removing t-match barely changed rankings, while rescaling t-match by 100 would swap the top two teams and also swap third and fourth place (Patel et al., 8 May 2026).

The leaderboard findings are striking. The public planning leaderboard saturated at 72.73%, with four teams reaching that value, and the public board had only 8 distinct positive public scores across 20 teams. Hidden evaluation changed the story unevenly across tracks: in planning, public and hidden scores had moderate positive correlation at about 0.69; in execution, public and hidden scores had $\Pi$8. The retrospective highlights execution-track cases where systems with 45.45% public score achieved 63.64% on hidden evaluation, while the top public execution system dropped from 72.73 to 45.45 on the hidden set. The authors therefore argue that the execution leaderboard did not straightforwardly measure “best industrial multi-agent system”; it mostly rewarded guardrails, response selection, contamination cleanup, fallback, and context control, rather than novel agent architectures (Patel et al., 8 May 2026).

The challenge also exposed severe attrition. The retrospective analyzes 149 registered teams, 349 declared member slots, and 300 submission attempts, but only 24 teams with non-zero public scores and 11 fully ranked teams. It further reports that 78/149 teams (52.3%) listed multiple Codabench usernames, implying that the competition was operationally account-based but substantively team-based. This became part of the paper’s broader recommendation that future benchmark releases treat team metadata, exact evaluated artifacts, and container digests as first-class benchmark outputs (Patel et al., 8 May 2026).

6. Subsequent extensions, reinterpretations, and benchmark-centered ecosystem

AssetOpsBench has rapidly become a shared testbed for work on industrial agent design, orchestration, and evaluation. A central reinterpretation appears in “Knowledge Graphs as the Missing Data Layer for LLM-Based Industrial Asset Operations”, which treats AssetOpsBench not as a replacement target but as a fixed scenario set for testing whether the data representation behind the tools is the main bottleneck. On a 139-scenario v1 snapshot of the benchmark, that paper reports 91/139 (65%) for the original document-based baseline, 114–116/139 (82–83%) for LLM-generated Cypher over a typed knowledge graph, and 137/139 (99%) for deterministic graph handlers. It also adds 40 graph-native scenarios and evaluates an expanded HuggingFace release of 467 scenarios across 6 domains, where deterministic handlers reach 467/467 (100%) with average score 0.848. The paper’s stated conclusion is that, for structured industrial operations, the data layer may be the primary bottleneck rather than orchestration alone (Mandarapu et al., 26 May 2026).

A second line of work uses AssetOpsBench as a tool-planning benchmark. “Internalizing Tool Knowledge in Small LLMs via QLoRA Fine-Tuning” focuses on the 152 human-generated queries defined in the MCP setting and asks whether a small model can internalize a fixed tool catalog well enough to plan without having tool schemas in the prompt. Using approximately 1,700 supervised examples derived from AssetOpsBench and a catalog of 5 agents and 23 tools, the paper reports that a fine-tuned Gemma 4 E4B-it reaches AT-F1 = 0.65 and overall judge score = 3.88, compared with 0.47 and 2.88 for the informed unfine-tuned baseline, while reducing input length by 82.6%. A fine-tuned Qwen3-4B reaches AT-F1 = 0.605 and overall judge score = 3.78, while using 62% less memory and running 2.49× faster than Gemma in the reported setup (Shemla et al., 18 May 2026).

Several systems papers use AssetOpsBench primarily as an orchestration and latency testbed. DynAMO treats benchmark queries as schema-constrained workflow graphs over IoT, TSFM, FMSR, and WO agents, reporting that across 141 industrial queries the plans contain 3–5 tasks on average and about 40% have at least one parallelizable branch. Its ParallelWorkflow reduces median latency from 308.53 s to 190.49 s, approximately 1.62×, and context pruning lowers inference latency by about 30% while maintaining correct functional behaviour under controlled fault injection (Kushwaha et al., 14 Jun 2026). “Evaluating Temporal Semantic Caching and Workflow Optimization in Agentic Plan-Execute Pipelines” studies a related optimization surface over AssetOpsBench’s MCP-backed pipelines, reporting 1.67× end-to-end speedup and about 40.0% median latency reduction from workflow optimizations, while a temporal semantic cache yields a median 30.6× speedup on cache hits but also exposes a failure mode of pure semantic caching for parameter-rich industrial queries (Merchant et al., 20 May 2026).

The broader AssetOpsBench open-source effort has also hosted adjacent systems rather than only the core benchmark. IndustryAssetEQA is explicitly presented as a neurosymbolic operational intelligence system for embodied question answering in industrial maintenance, released under the AssetOpsBench GitHub tree. It evaluates four industrial datasets—rotating machinery, turbofan engines, hydraulic systems, and cyber-physical production systems—and reports that, relative to LLM-only baselines, the full system improves structural validity by up to 0.51, counterfactual accuracy by up to 0.47, and explanation entailment by 0.64, while reducing severe expert-rated overclaims from 28% to 2%. This does not redefine AssetOpsBench’s core scenario benchmark, but it shows that the repository has become a broader benchmark-and-method scaffold for industrial operational intelligence (Shyalika et al., 25 Apr 2026).

Taken together, these follow-on studies suggest that AssetOpsBench functions simultaneously as a benchmark, an execution environment, and a controlled systems testbed. They also sharpen two methodological points. First, measured performance depends strongly on what is being varied: orchestration policy, tool descriptions, data substrate, or caching logic. Second, leaderboard-style results should not be read as a complete measure of industrial autonomy. The benchmark has repeatedly been used to show that apparently agentic limitations can instead arise from prompt format, hidden-evaluation design, data modeling choices, or backend orchestration assumptions (Mandarapu et al., 26 May 2026).