OccuBench: Benchmarking AI Agents in Professional Tasks

• OccuBench is a benchmark that simulates 100 real-world professional task scenarios across 10 industries using Language World Models.
• It evaluates agents on cross-industry task completion and environmental robustness by incorporating explicit and implicit fault regimes.
• The framework employs a multi-agent synthesis pipeline and rubric-based verification to ensure realistic simulation and task solvability.

to=arxiv_search.search 天天爱彩票app='{}' code? OccuBench is a benchmark for evaluating AI agents on real-world professional occupational tasks through Language World Models (LWMs) that simulate domain-specific environments via LLM-driven tool response generation. It targets the evaluation gap created by the fact that most existing agent benchmarks operate only in domains with public, engineerable environments, whereas many high-stakes occupations depend on enterprise systems, privacy-constrained workflows, and tool-mediated procedures that are not publicly accessible. OccuBench covers 100 real-world professional task scenarios across 10 industry categories and 65 specialized domains, yielding 382 final evaluation instances after filtering, and evaluates agents along two complementary dimensions: cross-industry task completion and environmental robustness under controlled fault injection (Hu et al., 13 Apr 2026).

1. Motivation and benchmark definition

OccuBench is designed for the “untestable majority” of professional work: domains such as emergency department triage, nuclear reactor safety monitoring, customs import processing, factory scheduling, network intrusion response, and wildfire evacuation coordination, where realistic public evaluation environments are generally unavailable. The benchmark defines its target tasks as professional occupational tasks, namely multi-step, stateful, tool-mediated workflows that mirror real job roles and their operational protocols (Hu et al., 13 Apr 2026).

The benchmark has two explicit goals. The first is cross-industry task completion, intended to reveal each agent’s occupational capability profile across diverse professional domains. The second is environmental robustness, intended to quantify resilience to realistic environment faults, including explicit errors, implicit data degradation, and mixed conditions. This framing distinguishes OccuBench from “happy-path” evaluations that measure only nominal task success.

A central claim of the benchmark is that professional agent evaluation should not be reduced to a single averaged capability score. The cross-industry design suggests that agent competence is heterogeneous and deployment-specific: an agent that performs strongly in one industrial context may underperform in another. The benchmark therefore treats occupational breadth, not just aggregate accuracy, as a core property of agent evaluation.

2. Coverage, task structure, and dataset composition

OccuBench spans 100 scenarios across 10 industry categories and 65 specialized domains, producing 382 solvable task instances with calibrated difficulty. Each scenario maps to a real job role and requires multi-step tool use and state tracking. Each scenario is further decomposed into 16 non-overlapping sub-topics, and each sub-topic is grounded by a professional reference document encoding terminology, workflows, state variables, edge cases, and constraints. The paper characterizes this as structural, document-grounded diversity rather than superficial paraphrase (Hu et al., 13 Apr 2026).

Industry category	Scenarios
Business / Enterprise	19
Technology / IT	16
Industrial / Engineering	12
Transportation / Logistics	11
Commerce / Consumer	9
Education / Culture	8
Healthcare / Life Sciences	7
Public Service / Governance	7
Agriculture / Environment	7
Science / Research	4

The scenario distribution is explicitly uneven: 19/16/12/11/9/8/7/7/7/4 across the ten categories above. Representative scenarios include resume screening, expense auditing, anti-money-laundering review, Linux operations, CI/CD recovery, intrusion response, production scheduling, mine ventilation, last-mile delivery, train dispatch, dynamic pricing, hotel revenue management, adaptive curriculum, fact-checking, emergency department triage, drug interaction screening, permit processing, irrigation control, telescope scheduling, and excavation planning.

The benchmark also reports several structural statistics. The average number of tools per environment is 5.5, with a range of 2–10 and median 5. The average number of tool calls per task is 16.2. Each task uses the hardest difficulty level, defined as the lowest autonomous success among generated options that remains solvable. This makes the released evaluation set deliberately nontrivial while excluding trivial or impossible instances.

3. Language World Models as simulated environments

OccuBench operationalizes environment construction through Language World Models. An LWM is defined as an LLM configured to simulate an interactive environment by generating stateful, structured tool responses. The environment configuration is

$$c = (\mathrm{system\_prompt}, \mathrm{tool\_schema}, \mathrm{initial\_state}, \mathrm{state\_description}),$$

and the transition formalization is

$(s_{t+1}, o_{t+1}) = f_\theta(s_t, a_t; c),$

where $s_t$ is an implicit in-context environment state, $a_t$ is the agent action, and $o_{t+1}$ is the structured JSON tool response. The interaction loop is written as

$o_t = \mathrm{LWM}(a_t, c, H_{t-1}),$

with $H_{t-1}$ denoting the trajectory history used to maintain state (Hu et al., 13 Apr 2026).

The four environment components have distinct roles. The system prompt specifies simulation logic, state evolution, error handling, and response formats. The tool schema defines callable functions with typed arguments and example outputs. The initial state encodes structured JSON starting conditions such as a patient queue, room inventory, or network topology. The state description provides causal consistency rules, for example that a booking reduces remaining inventory.

The benchmark argues that LLMs can serve as world models because pretraining provides strong format priors and broad domain knowledge, while prompts and in-context state tracking support coherent multi-turn simulation. It also notes that LLMs can handle atypical inputs more gracefully than hand-written simulators. At evaluation time, the agent under test issues tool calls, the LWM returns structured observations, and a rubric-based verifier scores the full trajectory for pass or fail. This decouples the simulator from the tested agent and allows different LLMs to serve as either side of the evaluation interface.

4. Task synthesis, solvability, and verification

OccuBench uses a multi-agent synthesis pipeline to generate tasks with guaranteed solvability, calibrated difficulty, and document-grounded diversity. For each scenario, 16 sub-topics are defined and a professional reference document is generated. A multi-agent pipeline then produces environment configurations, tool definitions, tasks, solution plans, and verification rubrics, using Gemini-3-Flash-Preview as the world model during synthesis (Hu et al., 13 Apr 2026).

Solvability is not assumed; it is empirically checked. Tasks are executed multiple times both with and without a reference plan to confirm solvability and calibrate difficulty. A majority-vote verifier grades trajectories against rubrics. When synthesis errors occur, a repair module diagnoses them and re-executes the generation process. Trivially easy tasks with 100% autonomous success, unsolvable tasks with 0% success, and invalid-schema tasks are filtered out. For each surviving task, the benchmark selects the hardest difficulty level that remains solvable.

This synthesis protocol is central to OccuBench’s methodological claim. Rather than treating LLM-generated environments as inherently reliable, the benchmark imposes a validation loop with repair, filtering, and rubric-based verification. A plausible implication is that the benchmark’s difficulty distribution is intentionally shaped rather than passively inherited from raw generation. That feature matters because occupational tasks are not only domain-specific but also highly sensitive to edge cases, latent constraints, and tool contract violations.

5. Evaluation protocol, metrics, and fault regimes

OccuBench evaluates agents under one clean regime and three fault-injected regimes. The clean environment is $E0$. The faulted environments are $E1$ for explicit faults, $E2$ for implicit faults, and $(s_{t+1}, o_{t+1}) = f_\theta(s_t, a_t; c),$0 for mixed faults. Explicit faults include visible error responses such as HTTP 500, TimeoutError, ConnectionRefused, and ServiceUnavailable, where correct behavior is generally retry with backoff. Implicit faults consist of degraded but superficially valid responses such as truncated lists, missing fields, empty or null values, and stale cached values; these provide no overt error signal and require the agent to detect data-quality anomalies and re-query. Mixed faults are approximately half explicit and half implicit (Hu et al., 13 Apr 2026).

The core task metric is completion rate,

$(s_{t+1}, o_{t+1}) = f_\theta(s_t, a_t; c),$1

where $(s_{t+1}, o_{t+1}) = f_\theta(s_t, a_t; c),$2 and $(s_{t+1}, o_{t+1}) = f_\theta(s_t, a_t; c),$3 indicates pass or fail for task $(s_{t+1}, o_{t+1}) = f_\theta(s_t, a_t; c),$4. All completion rates are reported over the full denominator of 382 for comparability. Robustness is defined as worst-case robustness relative to clean performance,

$(s_{t+1}, o_{t+1}) = f_\theta(s_t, a_t; c),$5

An $(s_{t+1}, o_{t+1}) = f_\theta(s_t, a_t; c),$6 value of 1.0 indicates no degradation under faults.

Fault injection is parameterized by fault_count, default 2, and fault_duration, default 2. Faults are transient, retries eventually recover, and events are spaced across the trajectory. Sensitivity analyses show monotonic degradation as fault count or fault duration increases. Under mixed faults, Claude Opus 4.6 drops from 71.5% at fc=1 to 60.2% at fc=4, and from 67.8% at fd=1 to 57.9% at fd=4; Qwen 3.5 Plus declines from 61.3% to 49.7% as fault_count increases and from 59.7% to 49.2% as fault_duration increases.

The paper provides concrete examples of the fault taxonomy. In a last-mile delivery routing task under $(s_{t+1}, o_{t+1}) = f_\theta(s_t, a_t; c),$7, a successful agent profiles telemetry, notices 28% battery, recharges first, and completes delivery while maintaining the battery-above-15% constraint; a failing agent navigates immediately and arrives at 12.5%, violating the constraint. In public transit schedule recovery under $(s_{t+1}, o_{t+1}) = f_\theta(s_t, a_t; c),$8, a resilient agent retries on each explicit fault and completes all required actions, whereas a fragile agent halts after a 500 error. In property valuation under $(s_{t+1}, o_{t+1}) = f_\theta(s_t, a_t; c),$9, a robust agent detects that only 2 of 15 units have been returned, re-queries, and correctly computes $s_t$0; a failing agent assumes the truncated result is complete and wrongly reports $s_t$1.

6. Empirical findings and occupational capability profiles

OccuBench evaluates 15 frontier models across 8 families: OpenAI, Anthropic, Google, DeepSeek, Moonshot, MiniMax, Zhipu, and Alibaba. The model list includes GPT-5.2; Claude Opus 4, 4.5, and 4.6; Claude Sonnet 4, 4.5, and 4.6; Gemini 3.1 Pro and Gemini 3.1 Flash-Lite; DeepSeek V3.2; Kimi K2.5; MiniMax M2.7; GLM-5; and Qwen 3.5 Plus and Qwen 3.5 Flash. All models use thinking or reasoning mode where available, and the main $s_t$2 rankings use high effort if configurable (Hu et al., 13 Apr 2026).

The principal empirical result is that no single model dominates all industries. GPT-5.2 leads overall with an $s_t$3 average of 79.6% and tops Agriculture at 84%, Business at 86%, Industrial at 85%, and Science at 94%, yet trails in Commerce at 67%, where Qwen 3.5 Plus leads with 81%. Gemini 3.1 Pro, at 72.3% overall, leads Education at 84% and Science at 81%. Claude Opus 4.6, at 71.5% overall, excels in Transportation at 77% and Business at 78% but lags in Commerce at 53%. The paper argues that this cross-industry lens reveals distinct occupational capability profiles that single-domain benchmarks do not expose.

A second major finding is that implicit faults are the hardest regime. The average completion rates are 67.5% for $s_t$4, 62.6% for $s_t$5, 53.4% for $s_t$6, and 54.4% for $s_t$7. Many models perform worse on pure implicit faults than on mixed faults because $s_t$8 lacks overt error signals and forces independent detection of degraded data. This directly challenges the common assumption that visible operational errors are the primary robustness bottleneck.

A third finding concerns scaling and inference-time reasoning. Larger models outperform smaller variants within families: Qwen Plus exceeds Flash by 10.2 points, Gemini Pro exceeds Flash-Lite by 11.0 points, and Claude Opus 4.6 exceeds Sonnet 4.6 by 7.1 points. Newer generations also improve, as shown by Claude Opus progression from 61.3% to 65.2% to 71.5%. Higher reasoning effort improves performance: GPT-5.2 rises from 54.7% at none to 82.2% at xhigh, a gain of 27.5 points, while Claude Opus 4.6 improves by 3.6 points from low to max. On robustness, the best $s_t$9 among nine flagship models is 0.87 for Gemini 3.1 Pro and MiniMax M2.7, with GPT-5.2 at 0.84.

7. Reliability, simulator quality, limitations, and use

OccuBench’s reliability argument depends heavily on simulator quality. The default simulator for synthesis and evaluation is Gemini-3-Flash-Preview, and the paper compares rankings under Gemini Flash, Qwen 3.5 Plus, and GPT-5.2 used as simulators. With a sufficiently capable simulator, rankings are stable: Gemini Flash and Qwen 3.5 Plus yield 85.7% pairwise ranking agreement and identical top-3 agents. By contrast, GPT-5.2, despite being the top agent under a capable simulator at 79.6% $a_t$0, produces the lowest-quality simulations when used as the LWM, collapsing average agent scores to about 29.3% because of simulator contract violations such as state fabrication, entity omission, and rule invention (Hu et al., 13 Apr 2026).

This result is methodologically important. Strong agents are not necessarily strong environment simulators, and LWM-based evaluation is reliable only when the simulator is vetted. The paper therefore advises validating simulator quality or re-validating task solvability when switching simulators. It also identifies broader threats to validity: LWM fidelity emphasizes domain logic rather than exact real data; simulator dependence can shift rankings if the environment contract is violated; and, despite 65 domains, many occupations remain uncovered. The authors also note that simulation fidelity should be complemented by real-environment testing where precise numeric fidelity is required, such as cents-accurate finance.

For practical use, the benchmark recommends reporting both clean completion and faulted robustness, inspecting per-industry profiles rather than only overall averages, configuring reasoning effort high for complex tasks when supported, adding defensive checks for implicit degradation, using retries and backoff for explicit faults, and integrating response-quality validators and recovery policies into agents. Code, data, environment configurations, tool schemas, prompts, rubrics, and scripts for clean and fault-injected evaluation are provided through the project page, GitHub repository, and Hugging Face dataset. In that form, OccuBench serves both as a benchmark and as a concrete experimental framework for evaluating professional agents under realistic operational noise.