OneMillion-Bench: Evaluating Professional Agents
- OneMillion-Bench is a bilingual benchmark consisting of 400 expert-curated tasks spanning law, finance, healthcare, industry, and natural sciences, representing over $1M in labor value.
- It simulates real professional workflows with long-context, multi-step tasks that require retrieval of authoritative sources, resolution of conflicting evidence, and strict compliance to domain-specific rules.
- The benchmark employs rubric-driven scoring and multi-stage expert curation to assess language agents’ process-based reasoning and their ability to meet professional standards.
Searching arXiv for OneMillion-Bench and closely related benchmark-generation/evaluation work to ground the article with current papers.
OneMillion-Bench, often written as $1M-Bench`, is a benchmark of 400 expert-curated tasks spanning Law, Finance, Industry, Healthcare, and Natural Science, built to evaluate language agents in economically consequential scenarios rather than in exam-style settings. Its central premise is that professional work requires retrieving authoritative sources, resolving conflicting evidence, applying domain-specific rules, and making constrained decisions, so correctness depends on the reasoning process and on professional compliance as much as on the final answer. The benchmark is bilingual, rubric-driven, and explicitly tied to the labor value of senior experts, with the aggregate task set valued at more than `$1M of human work (Yang et al., 9 Mar 2026).
1. Definition, purpose, and naming
OneMillion-Bench was introduced to evaluate LLMs as professional agents rather than as chat assistants answering short, structured questions. The benchmark asks how much expert work agents can actually do, and how close they are to replacing or substantially augmenting human experts in economically valuable tasks. In this framing, the relevant unit of evaluation is not a multiple-choice item but a context-heavy deliverable produced under strict professional constraints (Yang et al., 9 Mar 2026).
A common misconception is that the name refers to the number of questions. In fact, the benchmark contains 400 tasks total. The term “OneMillion” comes from the economic valuation scheme: each task is assigned a monetary value based on the estimated time required for a senior professional and the market wage of that role, and the total benchmark value exceeds $1M in real labor value. The benchmark therefore shifts emphasis from raw item count to the value and realism of the work being simulated (Yang et al., 9 Mar 2026).
The paper positions the benchmark between static QA suites and fully live environments. It keeps enough structure to support diagnosis and reproducible scoring, but it is intended to reflect real professional workflows more closely than benchmarks centered on short, well-posed questions with a single correct answer. This suggests that OneMillion-Bench is designed less as a general-purpose knowledge test than as a testbed for agentic reliability under professional norms.
2. Scope, composition, and task profile
The benchmark is balanced across five macro-domains and split evenly between English-language “Global” tasks and Chinese-language “CN” tasks. The Chinese tasks are not translations; they are separate tasks grounded in Chinese regulations, standards, and institutional settings (Yang et al., 9 Mar 2026).
| Dimension | Composition |
|---|---|
| Total tasks | 400 |
| Domains | 80 Finance, 80 Law, 80 Healthcare, 80 Natural Science, 80 Industry |
| Language split | 200 “Global” tasks, 200 “CN” tasks |
Within those macro-domains, the benchmark is further refined into 37 subdomains and 92 third-level categories. The examples given in the paper include criminal defense, M&A and restructuring, data compliance, equities, bonds, derivatives, life insurance, oncology, cardiology, software, semiconductors, telecom, condensed matter, quantum, chemistry, mathematics, molecular biology, and ecology. The stated goal is broad coverage of modern knowledge work rather than narrow concentration on a single profession (Yang et al., 9 Mar 2026).
The tasks are described as semi–open-ended, multi-step, and long-context. They are not multiple-choice items. Typical prompts resemble real assignments such as writing a legal opinion, analyzing the Japanese yen’s depreciation cycle in 2025, proposing a treatment plan consistent with 2024–2025 guidelines, or diagnosing a quantum spin liquid phase from inelastic neutron scattering data. The paper reports average prompt lengths in the hundreds to thousands of tokens, including approximately 3278 tokens for Global Law, 1811 for Global Finance, 709 for CN Law, and 965 for CN Industry. This long-context structure is central to the benchmark’s claim that it evaluates professional reasoning rather than test-taking heuristics (Yang et al., 9 Mar 2026).
The benchmark also distinguishes itself by requiring more than answer production. Tasks are designed to force retrieval of authoritative sources, resolution of conflicting evidence, application of domain-specific rules, and constrained decision-making under legal, safety, or institutional requirements. In that sense, its task design assumes a tool-using agent rather than a bare LLM.
3. Expert curation, realism, and economic valuation
OneMillion-Bench is expert-built rather than mined from existing exams or public question banks. The curation pipeline has three stages. First, a domain expert creates a specialized, practically valuable task, writes a detailed reference answer with reasoning, and defines a rubric with criteria, weights, penalties, tags, and reference sources. Second, another expert in the same domain reviews the task, answer, and rubric for clarity, fairness, and alignment. Third, when necessary, a third expert arbitrates disagreements or refines high-risk tasks. The resulting dataset is then further filtered through lower bound elimination, which removes tasks solved by all tested agents, and upper bound review, which re-examines tasks on which all agents fail badly to separate realistic difficulty from “mission impossible” design (Yang et al., 9 Mar 2026).
Difficulty calibration is therefore adversarial. Tasks are retained only if current frontier agents fail them under rubric-based evaluation. This is intended to ensure meaningful differentiation across strong systems rather than easy wins on routine prompts. The benchmark’s realism claim rests not only on expert authorship but also on the use of real regulations, guidelines, project structures, and workflow constraints.
Economic valuation is a formal part of the benchmark rather than an informal narrative. Each task is assigned an estimated time required for senior experts and an hourly wage , and the task value is defined as
The paper reports representative values by domain. For Global Finance, the average time is 26.1h, average hourly wage is $175.7`**, average value is **`$4,593 per question, and total value is $183,726` for 40 questions**. For Global Law, the corresponding figures are **31.4h**, **`$243/h, $7,666` per question**, and **`$306,646 total. For Global Healthcare, they are 22.9h, $354.9/h`**, **`$8,189 per question, and $327,557 total. The benchmark’s name and framing are therefore directly tied to explicit labor-value estimates rather than metaphorical difficulty claims (Yang et al., 9 Mar 2026).
4. Rubric architecture and scoring protocol
Each task is evaluated with a rubric set $R_qqT_{\text{Expert}}$0 has a score $T_{\text{Expert}}$1 assigned by the judge and a predefined weight $T_{\text{Expert}}$2, which may be positive or negative. The benchmark defines Expert Score as
$T_{\text{Expert}}$3
where $T_{\text{Expert}}$4 is the subset of rubrics with positive weights. Scores are normalized to $T_{\text{Expert}}$5, and negative rubrics can reduce the numerator below zero before clipping. The excerpt notes that a typical question has approximately 16–19 rubrics, with about 4 negative rubrics on average, and that negative weights can reach -20. This makes penalties a first-class part of the measurement design rather than a post hoc qualitative comment (Yang et al., 9 Mar 2026).
To approximate professional adequacy, the benchmark also defines Pass Rate with a threshold of $T_{\text{Expert}}$6:
$T_{\text{Expert}}$7
This distinction matters. Expert Score measures partial rubric fulfillment, whereas Pass Rate measures how often an output reaches an expert-like standard. The paper emphasizes that many models obtain moderate average scores while still failing to cross the pass threshold on most tasks (Yang et al., 9 Mar 2026).
Rubrics are tagged by capability type. The four rubric categories are Factual Information (FI), Analytical Reasoning (AR), Instruction Following (IF), and Structure & Formatting (SF). Scores can be aggregated by domain, by rubric type, or overall. Negative rubrics explicitly penalize unsafe or unprofessional recommendations, hallucinations on critical points, and failure to follow explicit structural or role constraints. This is a notable design choice: the benchmark rewards not only coverage of salient content but also avoidance of professional failure modes.
Human experts create tasks, reference answers, sources, and rubrics, but the paper uses LLM-as-judge to scale scoring. It reports that multiple judges, including GPT-5.2-High and GLM-5, were studied. Rankings are described as reasonably stable across judges, but absolute scores vary: GPT-5.2-High is stricter and yields lower scores with better top-end discrimination, whereas GLM-5 is more lenient. The paper recommends reporting judge identity and using multi-judge evaluation to mitigate bias (Yang et al., 9 Mar 2026).
5. Agent configurations and empirical results
The evaluation covers 35 systems organized into three categories: 17 vanilla models, 17 search agents, and 3 deep research agents. The deep research systems are o3-DeepResearch, o4-Mini-DeepResearch, and Sonar-DeepResearch. Search-enabled agents use provider-native tool APIs or OpenRouter-based scaffolds, and the benchmark explicitly studies scaffold effects as part of agent performance (Yang et al., 9 Mar 2026).
On the Global subset, the best-performing vanilla model is Claude-Opus-4.6. Its reported Vanilla Economic Value is 8483.8k with search, with Expert Score improving from 55.0 to 63.0 and Pass Rate from 36.5% to 43.5%. On the CN subset, the same model again leads, with Economic Value rising from ¥350.3k to ¥470.2k, Expert Score from 55.8 to 64.5, and Pass Rate from 35.0% to 48.5%. Other strong systems named in the paper include GPT-5.4-High, GPT-5.2-High, GPT-5.3-Codex, Gemini-3.x, and Qwen3.5-Plus (Yang et al., 9 Mar 2026).
The headline limitation of current agents is clear: even the best models do not pass most tasks. Pass Rates remain below 50% on both Global and CN subsets. This result supports the paper’s broader claim that current language agents are not yet reliable standalone professionals in high-stakes domains.
Search is important but not uniformly beneficial. Strong models often improve with search, and some mid-tier systems improve substantially. The paper gives Doubao Seed as an example, with Global Expert Score increasing from approximately 39 to 52 and Pass Rate from 13.5% to 28.5%. But search can also degrade performance. Hunyuan-2.0-Thinking declines on Global from 34.7 to 30.2 in Expert Score and from 8.5% to 3.0% in Pass Rate, and Step-3.5-Flash shows a slight decline. The reported interpretation is that retrieval acts as a capability amplifier: strong models can exploit it, while weaker integration can inject noisy or conflicting evidence and reduce compliance (Yang et al., 9 Mar 2026).
The deep research agents achieve mid-tier Expert Scores but are generally below the best search-enabled generalist models in Expert Score, Pass Rate, and Economic Value. The paper therefore argues that success depends less on a long research pipeline per se than on rubric coverage and professional compliance. It also reports that official provider scaffolds generally outperform third-party scaffolds for the same base model, and that in some cases removing search altogether can outperform a mediocre scaffold.
The capability profile is uneven across rubric types. Structure & Formatting is the strongest area, often above 80%. Instruction Following reaches roughly 60–75% for top models. Factual Information is typically 40–55%, and Analytical Reasoning is roughly 50–60% for top systems. Search mainly helps FI and AR for stronger models, but weaker models can lose ground on FI, AR, IF, and SF when longer, citation-heavy outputs become less coherent (Yang et al., 9 Mar 2026).
The benchmark also reports two further findings with broader methodological significance. First, all models perform better on temporally agnostic tasks than on time-sensitive ones, and top models can lose 15–20 percentage points on time-sensitive tasks. Second, in test-time scaling on Global Finance, pass@k increases roughly logarithmically as 9 rises, while “pass-of-k” decays toward zero. The paper uses this to argue that sampling multiple drafts may improve best-of-0 success, but naïve aggregation can reduce reliability.
6. Failure modes, benchmark position, and limitations
The qualitative error analysis identifies several recurring failure patterns. In reasoning-heavy tasks, web search can degrade performance by introducing outdated or tangential information. In finance and economics, agents frequently mis-extract numbers from tables, miscalculate ratios, or omit critical metrics. In law and compliance tasks, they often display partial normative knowledge, apply the wrong jurisdiction, or map fact patterns inconsistently to legal qualifications. In healthcare and industry tasks, they tend to produce high-level but non-operational guidance, omit contraindications or follow-up details, and skip systematic diagnosis in favor of generic suggestions. In natural science, they can summarize known facts yet fail to interpret experimental signatures mechanistically (Yang et al., 9 Mar 2026).
Within the benchmark landscape, OneMillion-Bench is distinguished from three families of prior evaluation. Relative to hard QA benchmarks such as GPQA, LiveBench, MMLU-Pro, and Humanity’s Last Exam, it replaces isolated exam-style questions with long, open-ended tasks and process-based scoring. Relative to agentic workflow benchmarks such as SWE-bench, XBench, TravelPlanner, τ-bench, Terminal-Bench, and AndroidWorld, it adds a strong rubric-based emphasis on process quality, compliance, and economic value. Relative to highly reality-grounded settings such as LiveTradeBench and Lab-Bench, it retains more diagnostic structure, allowing failure analysis through rubric components rather than only task completion outcomes (Yang et al., 9 Mar 2026).
The benchmark also sits in a broader ecosystem of work on dynamic and automatically generated evaluation. YourBench is a document-driven framework for generating bespoke, temporally controlled, contamination-resistant evaluation sets from arbitrary user documents, and is explicitly described as complementary to “mega-benchmarks” such as OneMillion-Bench because it prioritizes freshness, customization, and domain specificity over a fixed shared suite (Shashidhar et al., 2 Apr 2025). BenchBench addresses automated benchmark generation itself through domain cards, quota-controlled generation, panel validation, and designer–answerer matrices, providing a blueprint for auditing large auto-generated benchmark suites at scale (Zheng et al., 21 Mar 2026). OR-Bench contributes a distinct large-scale safety module centered on over-refusal, measuring how often models reject benign prompts that merely appear harmful; this illustrates a safety/helpfulness dimension that is not the core target of OneMillion-Bench’s professional-task design (Cui et al., 2024).
Several limitations are explicit. Coverage is restricted to five domains, despite their breadth. Jurisdictional scope remains partial even with the Global/CN split. Task selection and rubrics inevitably reflect the industries, regions, and professional norms of contributors. Expert creation and rubric design required more than 2,000 expert hours, and LLM-as-judge reduces but does not eliminate evaluation cost or judge bias. The paper therefore identifies future directions in domain expansion, dynamic updating with current laws and market conditions, more automated process evaluation, and agent architectures better aligned to risk management and rubric-driven performance (Yang et al., 9 Mar 2026).
Taken together, OneMillion-Bench operationalizes a specific claim about modern language agents: the relevant question is not whether they can answer difficult prompts, but whether they can produce expert-grade work under real professional constraints. Its results indicate substantial progress in drafting, search-assisted analysis, and structured response generation, but they also indicate that the gap to human experts remains large in finance, law, healthcare, industry, and natural science when the evaluation standard is professional adequacy rather than surface plausibility (Yang et al., 9 Mar 2026).