---
title: 'OneMillion-Bench: Evaluating Professional Agents'
url: https://www.emergentmind.com/topics/onemillion-bench
type: topic
---

# OneMillion-Bench: Evaluating Professional Agents

Searching arXiv for OneMillion-Bench and closely related benchmark-generation/evaluation work to ground the article with current papers.
OneMillion-Bench, often written as `$1M-Bench`, is a benchmark of 400 expert-curated tasks spanning Law, Finance, Industry, Healthcare, and Natural Science, built to evaluate language agents in economically consequential scenarios rather than in exam-style settings. Its central premise is that professional work requires retrieving authoritative sources, resolving conflicting evidence, applying domain-specific rules, and making constrained decisions, so correctness depends on the reasoning process and on professional compliance as much as on the final answer. The benchmark is bilingual, rubric-driven, and explicitly tied to the labor value of senior experts, with the aggregate task set valued at more than `$1M` of human work [2603.07980].

## 1. Definition, purpose, and naming

OneMillion-Bench was introduced to evaluate language models as professional agents rather than as chat assistants answering short, structured questions. The benchmark asks how much expert work agents can actually do, and how close they are to replacing or substantially augmenting human experts in economically valuable tasks. In this framing, the relevant unit of evaluation is not a multiple-choice item but a context-heavy deliverable produced under strict professional constraints [2603.07980].

A common misconception is that the name refers to the number of questions. In fact, the benchmark contains **400 tasks total**. The term “OneMillion” comes from the economic valuation scheme: each task is assigned a monetary value based on the estimated time required for a senior professional and the market wage of that role, and the total benchmark value exceeds `$1M` in real labor value. The benchmark therefore shifts emphasis from raw item count to the value and realism of the work being simulated [2603.07980].

The paper positions the benchmark between static QA suites and fully live environments. It keeps enough structure to support diagnosis and reproducible scoring, but it is intended to reflect real professional workflows more closely than benchmarks centered on short, well-posed questions with a single correct answer. This suggests that OneMillion-Bench is designed less as a general-purpose knowledge test than as a testbed for agentic reliability under professional norms.

## 2. Scope, composition, and task profile

The benchmark is balanced across five macro-domains and split evenly between English-language “Global” tasks and Chinese-language “CN” tasks. The Chinese tasks are not translations; they are separate tasks grounded in Chinese regulations, standards, and institutional settings [2603.07980].

| Dimension | Composition |
|---|---|
| Total tasks | 400 |
| Domains | 80 Finance, 80 Law, 80 Healthcare, 80 Natural Science, 80 Industry |
| Language split | 200 “Global” tasks, 200 “CN” tasks |

Within those macro-domains, the benchmark is further refined into **37 subdomains** and **92 third-level categories**. The examples given in the paper include criminal defense, M\&A and restructuring, data compliance, equities, bonds, derivatives, life insurance, oncology, cardiology, software, semiconductors, telecom, condensed matter, quantum, chemistry, mathematics, molecular biology, and ecology. The stated goal is broad coverage of modern knowledge work rather than narrow concentration on a single profession [2603.07980].

The tasks are described as **semi–open-ended**, **multi-step**, and **long-context**. They are not multiple-choice items. Typical prompts resemble real assignments such as writing a legal opinion, analyzing the Japanese yen’s depreciation cycle in 2025, proposing a treatment plan consistent with 2024–2025 guidelines, or diagnosing a quantum spin liquid phase from inelastic neutron scattering data. The paper reports average prompt lengths in the hundreds to thousands of tokens, including approximately **3278 tokens** for Global Law, **1811** for Global Finance, **709** for CN Law, and **965** for CN Industry. This long-context structure is central to the benchmark’s claim that it evaluates professional reasoning rather than test-taking heuristics [2603.07980].

The benchmark also distinguishes itself by requiring more than answer production. Tasks are designed to force retrieval of authoritative sources, resolution of conflicting evidence, application of domain-specific rules, and constrained decision-making under legal, safety, or institutional requirements. In that sense, its task design assumes a tool-using agent rather than a bare language model.

## 3. Expert curation, realism, and economic valuation

OneMillion-Bench is expert-built rather than mined from existing exams or public question banks. The curation pipeline has three stages. First, a domain expert creates a specialized, practically valuable task, writes a detailed reference answer with reasoning, and defines a rubric with criteria, weights, penalties, tags, and reference sources. Second, another expert in the same domain reviews the task, answer, and rubric for clarity, fairness, and alignment. Third, when necessary, a third expert arbitrates disagreements or refines high-risk tasks. The resulting dataset is then further filtered through **lower bound elimination**, which removes tasks solved by all tested agents, and **upper bound review**, which re-examines tasks on which all agents fail badly to separate realistic difficulty from “mission impossible” design [2603.07980].

Difficulty calibration is therefore adversarial. Tasks are retained only if current frontier agents fail them under rubric-based evaluation. This is intended to ensure meaningful differentiation across strong systems rather than easy wins on routine prompts. The benchmark’s realism claim rests not only on expert authorship but also on the use of real regulations, guidelines, project structures, and workflow constraints.

Economic valuation is a formal part of the benchmark rather than an informal narrative. Each task is assigned an estimated time $T_{\text{Expert}}$ required for senior experts and an hourly wage $W_{\text{Hourly}}$, and the task value is defined as

$$
V = T_{\text{Expert}} \times W_{\text{Hourly}}.
$$

The paper reports representative values by domain. For Global Finance, the average time is **26.1h**, average hourly wage is **`$175.7`**, average value is **`$4,593` per question**, and total value is **`$183,726` for 40 questions**. For Global Law, the corresponding figures are **31.4h**, **`$243/h`**, **`$7,666` per question**, and **`$306,646` total**. For Global Healthcare, they are **22.9h**, **`$354.9/h`**, **`$8,189` per question**, and **`$327,557` total**. The benchmark’s name and framing are therefore directly tied to explicit labor-value estimates rather than metaphorical difficulty claims [2603.07980].

## 4. Rubric architecture and scoring protocol

Each task is evaluated with a rubric set $R_q$ specific to question $q$. A rubric $r \in R_q$ has a score $s_r$ assigned by the judge and a predefined weight $w_r$, which may be positive or negative. The benchmark defines **Expert Score** as

$$
\text{Expert Score}(q) = \max\left(0, \frac{\displaystyle\sum_{r \in R_q} s_r}{\displaystyle\sum_{r \in R_q^{+}} w_r} \right),
$$

where $R_q^{+}$ is the subset of rubrics with positive weights. Scores are normalized to $[0,1]$, and negative rubrics can reduce the numerator below zero before clipping. The excerpt notes that a typical question has approximately **16–19 rubrics**, with about **4 negative rubrics on average**, and that negative weights can reach **`-20`**. This makes penalties a first-class part of the measurement design rather than a post hoc qualitative comment [2603.07980].

To approximate professional adequacy, the benchmark also defines **Pass Rate** with a threshold of $0.7$:

$$
\text{Pass Rate}(Q) = \frac{1}{|Q|}\sum_{q \in Q} \mathds{1}\left[\text{Expert Score}(q) \ge 0.7\right].
$$

This distinction matters. Expert Score measures partial rubric fulfillment, whereas Pass Rate measures how often an output reaches an expert-like standard. The paper emphasizes that many models obtain moderate average scores while still failing to cross the pass threshold on most tasks [2603.07980].

Rubrics are tagged by capability type. The four rubric categories are **Factual Information (FI)**, **Analytical Reasoning (AR)**, **Instruction Following (IF)**, and **Structure \& Formatting (SF)**. Scores can be aggregated by domain, by rubric type, or overall. Negative rubrics explicitly penalize unsafe or unprofessional recommendations, hallucinations on critical points, and failure to follow explicit structural or role constraints. This is a notable design choice: the benchmark rewards not only coverage of salient content but also avoidance of professional failure modes.

Human experts create tasks, reference answers, sources, and rubrics, but the paper uses **LLM-as-judge** to scale scoring. It reports that multiple judges, including **GPT-5.2-High** and **GLM-5**, were studied. Rankings are described as reasonably stable across judges, but absolute scores vary: GPT-5.2-High is stricter and yields lower scores with better top-end discrimination, whereas GLM-5 is more lenient. The paper recommends reporting judge identity and using multi-judge evaluation to mitigate bias [2603.07980].

## 5. Agent configurations and empirical results

The evaluation covers **35 systems** organized into three categories: **17 vanilla models**, **17 search agents**, and **3 deep research agents**. The deep research systems are **o3-DeepResearch**, **o4-Mini-DeepResearch**, and **Sonar-DeepResearch**. Search-enabled agents use provider-native tool APIs or OpenRouter-based scaffolds, and the benchmark explicitly studies scaffold effects as part of agent performance [2603.07980].

On the Global subset, the best-performing vanilla model is **Claude-Opus-4.6**. Its reported **Vanilla Economic Value** is **`$439.2k`**, increasing to **`$483.8k`** with search, with **Expert Score** improving from **55.0** to **63.0** and **Pass Rate** from **36.5%** to **43.5%**. On the CN subset, the same model again leads, with **Economic Value** rising from **`¥350.3k`** to **`¥470.2k`**, **Expert Score** from **55.8** to **64.5**, and **Pass Rate** from **35.0%** to **48.5%**. Other strong systems named in the paper include **GPT-5.4-High**, **GPT-5.2-High**, **GPT-5.3-Codex**, **Gemini-3.x**, and **Qwen3.5-Plus** [2603.07980].

The headline limitation of current agents is clear: **even the best models do not pass most tasks**. Pass Rates remain below **50%** on both Global and CN subsets. This result supports the paper’s broader claim that current language agents are not yet reliable standalone professionals in high-stakes domains.

Search is important but not uniformly beneficial. Strong models often improve with search, and some mid-tier systems improve substantially. The paper gives **Doubao Seed** as an example, with Global Expert Score increasing from approximately **39** to **52** and Pass Rate from **13.5%** to **28.5%**. But search can also degrade performance. **Hunyuan-2.0-Thinking** declines on Global from **34.7** to **30.2** in Expert Score and from **8.5%** to **3.0%** in Pass Rate, and **Step-3.5-Flash** shows a slight decline. The reported interpretation is that retrieval acts as a capability amplifier: strong models can exploit it, while weaker integration can inject noisy or conflicting evidence and reduce compliance [2603.07980].

The deep research agents achieve mid-tier Expert Scores but are generally below the best search-enabled generalist models in Expert Score, Pass Rate, and Economic Value. The paper therefore argues that success depends less on a long research pipeline per se than on rubric coverage and professional compliance. It also reports that official provider scaffolds generally outperform third-party scaffolds for the same base model, and that in some cases removing search altogether can outperform a mediocre scaffold.

The capability profile is uneven across rubric types. **Structure \& Formatting** is the strongest area, often above **80%**. **Instruction Following** reaches roughly **60–75%** for top models. **Factual Information** is typically **40–55%**, and **Analytical Reasoning** is roughly **50–60%** for top systems. Search mainly helps FI and AR for stronger models, but weaker models can lose ground on FI, AR, IF, and SF when longer, citation-heavy outputs become less coherent [2603.07980].

The benchmark also reports two further findings with broader methodological significance. First, all models perform better on temporally agnostic tasks than on time-sensitive ones, and top models can lose **15–20 percentage points** on time-sensitive tasks. Second, in test-time scaling on Global Finance, **pass@k** increases roughly logarithmically as $k$ rises, while **“pass-of-k”** decays toward zero. The paper uses this to argue that sampling multiple drafts may improve best-of-$k$ success, but naïve aggregation can reduce reliability.

## 6. Failure modes, benchmark position, and limitations

The qualitative error analysis identifies several recurring failure patterns. In reasoning-heavy tasks, web search can degrade performance by introducing outdated or tangential information. In finance and economics, agents frequently mis-extract numbers from tables, miscalculate ratios, or omit critical metrics. In law and compliance tasks, they often display partial normative knowledge, apply the wrong jurisdiction, or map fact patterns inconsistently to legal qualifications. In healthcare and industry tasks, they tend to produce high-level but non-operational guidance, omit contraindications or follow-up details, and skip systematic diagnosis in favor of generic suggestions. In natural science, they can summarize known facts yet fail to interpret experimental signatures mechanistically [2603.07980].

Within the benchmark landscape, OneMillion-Bench is distinguished from three families of prior evaluation. Relative to **hard QA** benchmarks such as **GPQA**, **LiveBench**, **MMLU-Pro**, and **Humanity’s Last Exam**, it replaces isolated exam-style questions with long, open-ended tasks and process-based scoring. Relative to **agentic workflow** benchmarks such as **SWE-bench**, **XBench**, **TravelPlanner**, **`τ`-bench**, **Terminal-Bench**, and **AndroidWorld**, it adds a strong rubric-based emphasis on process quality, compliance, and economic value. Relative to highly **reality-grounded** settings such as **LiveTradeBench** and **Lab-Bench**, it retains more diagnostic structure, allowing failure analysis through rubric components rather than only task completion outcomes [2603.07980].

The benchmark also sits in a broader ecosystem of work on dynamic and automatically generated evaluation. **YourBench** is a document-driven framework for generating bespoke, temporally controlled, contamination-resistant evaluation sets from arbitrary user documents, and is explicitly described as complementary to “mega-benchmarks” such as OneMillion-Bench because it prioritizes freshness, customization, and domain specificity over a fixed shared suite [2504.01833]. **BenchBench** addresses automated benchmark generation itself through domain cards, quota-controlled generation, panel validation, and designer–answerer matrices, providing a blueprint for auditing large auto-generated benchmark suites at scale [2603.20807]. **OR-Bench** contributes a distinct large-scale safety module centered on over-refusal, measuring how often models reject benign prompts that merely appear harmful; this illustrates a safety/helpfulness dimension that is not the core target of OneMillion-Bench’s professional-task design [2405.20947].

Several limitations are explicit. Coverage is restricted to **five domains**, despite their breadth. Jurisdictional scope remains partial even with the Global/CN split. Task selection and rubrics inevitably reflect the industries, regions, and professional norms of contributors. Expert creation and rubric design required **more than 2,000 expert hours**, and LLM-as-judge reduces but does not eliminate evaluation cost or judge bias. The paper therefore identifies future directions in domain expansion, dynamic updating with current laws and market conditions, more automated process evaluation, and agent architectures better aligned to risk management and rubric-driven performance [2603.07980].

Taken together, OneMillion-Bench operationalizes a specific claim about modern language agents: the relevant question is not whether they can answer difficult prompts, but whether they can produce expert-grade work under real professional constraints. Its results indicate substantial progress in drafting, search-assisted analysis, and structured response generation, but they also indicate that the gap to human experts remains large in finance, law, healthcare, industry, and natural science when the evaluation standard is professional adequacy rather than surface plausibility [2603.07980].

Source: https://www.emergentmind.com/topics/onemillion-bench