---
title: 'OneMillion-Bench: Language Agent Benchmark'
url: https://www.emergentmind.com/topics/onemillion-bench-1m-bench
type: topic
---

# OneMillion-Bench: Language Agent Benchmark

OneMillion-Bench (1M-Bench) is a comprehensive benchmark designed to evaluate language agents in economically consequential, professional domains. It consists of 400 expert-curated tasks spanning Finance, Law, Healthcare, Natural Science, and Industry, each paired with explicit economic value based on expert time and prevailing market wages. Tasks are constructed to mirror real-world professional workflows, requiring not only accurate answers but robust multi-step reasoning, evidence integration, and adherence to domain-specific constraints. The evaluation protocol employs fine-grained, rubric-based scoring, departing from traditional benchmarks focused solely on answer correctness and targeting professional readiness and agentic reliability in domain-intensive contexts [2603.07980].

## 1. Motivation and Limitations of Prior Benchmarks

Classic QA and multiple-choice evaluations such as MMLU, GPQA, and HLE are limited to isolated knowledge units or formal reasoning tasks in highly structured environments. These approaches fail to reflect the complexity of real-world deliverables, neglecting factors such as multi-step workflows, the need for authoritative references, evidence reconciliation, and nuanced domain rules. Tool-agent evaluations like SWE-bench, TravelPlanner, and Terminal-Bench introduce planning and tool usage but lack rigorous domain-grounded reasoning assessment. Crucially, there has been no standard benchmark to quantify how much real economic value (senior professional time × market wages) language models might deliver on tasks of direct economic import. OneMillion-Bench was instituted to systematically bridge these gaps through high-value domains, explicit task valuation, and comprehensive process-oriented rubrics [2603.07980].

## 2. Benchmark Composition and Task Distribution

OneMillion-Bench comprises 400 semi-open-ended, expert-curated tasks:

- **Domains:** Finance, Law, Healthcare, Natural Science, Industry (80 questions each).
- **Language/Cultural Subsets:** Each domain is divided evenly between Chinese (CN; 200 tasks) reflecting Mainland-China standards, and English (Global; 200 tasks) based on U.S./international norms.
- **Hierarchical Coverage:** 37 subdomains and 92 third-level categories ensure diversity (e.g., Finance → “Life Insurance,” “VC/PE”; Healthcare → “Orthopedics,” “Dentistry”; Law → “M&A,” “Data Compliance”; Industry → “Semiconductors,” “Civil Engineering”; Natural Science → “Quantum Physics,” “Genetics”).

| Domain        | Subset   | Task Count | Avg Value per Q | Total Value      |
|---------------|----------|------------|-----------------|------------------|
| Finance       | CN (¥)   | 80         | ¥7,410.8        | ¥296,432         |
| Finance       | Global ($) | 80       | \$4,593.2       | \$183,726        |
| **Total**     | CN       | 200        | —               | ¥921,832         |
| **Total**     | Global   | 200        | —               | \$1,008,370      |

Tasks are exemplified by prompts such as, “Review the Yen’s 2025 depreciation cycle phases [...] and project 2026 trend-reversion triggers,” requiring structured, multi-source synthesis, quantitative analysis, and forward-looking judgments.

## 3. Rubric-Based Evaluation Protocol

Each question $q$ is paired with a set of rubrics $R_q$, where every rubric $r$ includes:

- Explicit measurement criteria (e.g., legal citation, multi-step calculation, guideline application)
- Weight $w_r$ (ranging up to $+10$ or $-20$) reflecting impact
- Functional tag: Factual Information (FI), Analytical Reasoning (AR), Instruction Following (IF), Structure & Formatting (SF)
- (Optionally) references to authoritative sources

Scoring is performed either by an LM judge or human expert, assigning $s_r \in [-w_r, +w_r]$ for each rubric. The **ExpertScore** for a question $q$:

\[
\text{ExpertScore}(q) = \max\!\Bigl(0,\; \frac{\sum_{r\in R_q} s_r}{\sum_{r\in R_q^+} w_r}\Bigr)\,,
\]

where $R_q^+$ is the set of rubrics with positive weights. Scores are clipped to $[0,1]$ for normalization. The **PassRate** across a question set $Q$ is defined as

\[
\text{PassRate}(Q) = \frac{1}{|Q|} \sum_{q \in Q} \mathbf{1}\bigl[\text{ExpertScore}(q) \ge 0.7 \bigr].
\]

Aggregations can be performed per domain, rubric type, or as an overall mean benchmark score [2603.07980].

## 4. Task Requirements and Operational Distinctions

Distinct from prior benchmarks, 1M-Bench tasks explicitly require:

- Retrieval and citation of **authoritative sources** (e.g., SEC filings, medical guidelines)
- **Resolution of conflicting evidence** (e.g., divergent regulatory thresholds)
- Application of **domain-specific rules** (e.g., IFRS 17 methodology, Supreme People’s Court precedents)
- **Constraint decisions** (e.g., compliance with “non-extraction only” directives in clinical or industrial contexts)

These dimensions introduce nontrivial professional trade-offs such as risk management and feasibility, and cannot be addressed by pattern-matching or simplistic chain-of-thought approaches. This design aims to reveal agent limitations in handling conflicting information, integrating multi-source evidence, and following professional standards under real-world constraints [2603.07980].

## 5. Experimental Results and Performance Insights

Thirty-five models and agentic systems were evaluated under “Vanilla” (no tools) and “Search” (web search) conditions for both Global and CN subsets. Top-performing models (Global subset) included Claude-Opus-4.6 and GPT-5.4-High. Representative metrics:

| Model            | Economic Value ($k) | ExpertScore (%) | PassRate (%) |
|------------------|--------------------|-----------------|-------------|
|                  | Vanilla | Search   | Vanilla | Search | Vanilla | Search |
| Claude-Opus-4.6  | 439.2   | 483.8↑44.6 | 55.0   | 63.0↑8.1 | 36.5   | 43.5↑7.0 |
| GPT-5.4-High     | 330.9   | 365.5↑34.6 | 55.0   | 59.2↑4.2 | 31.5   | 38.0↑6.5 |

Key findings:

- **Claude-Opus-4.6** achieved \$483.8k economic value, 63% ExpertScore, and 43.5% PassRate with search.
- **Search augmentation** improved leading models by +4–12 points (ExpertScore) and +6–13 points (PassRate); however, weaker models sometimes deteriorated with noisy retrieval.
- **DeepResearch agents** (e.g., o3-DeepResearch, Sonar-DeepResearch) delivered mid-tier results (40–50% ExpertScore) but did not surpass top generalists.
- Domain difficulty varied: **Finance** was most challenging; Healthcare and Law yielded the highest scores. This profile was consistent across language subsets.
- Rubric-type breakdown: Structure & Formatting (85–90%), Instruction Following (65–82%), Analytical Reasoning (40–60%), Factual Information (40–65%). Search most improved Factual Information and Analytical Reasoning for strong models.

Pareto analysis indicated that search-enabled agents dominate non-search baselines at equivalent inference costs, while smaller agents offer mid-range performance at reduced computational expense [2603.07980].

## 6. Professional Readiness, Conclusions, and Future Directions

OneMillion-Bench reveals a persistent **reliability gap** between leading language model agents and human experts in high-stakes domains. Even the best agents (with search capabilities) attain only ~60% ExpertScore and ~40% PassRate, falling short on complex, multi-constraint deliverables. High scores in Structure & Formatting can mask deeper failures in factual and analytical dimensions, indicating limitations in substantive reasoning. Tool-use (web search) has conditional benefits—empowering strong models but introducing risk for retrieval-sensitive systems—and temporal sensitivity remains a challenge, as performance declines on time-sensitive tasks.

Planned future developments include domain expansion (to energy, climate, public policy), dynamic benchmark adaptation reflecting evolving professional workflows, and the development of rule-based or learned judges for scalable evaluation of intermediate reasoning, citations, and compliance. OneMillion-Bench thus advances the field toward economically grounded, compliance-oriented, and reliability-demanding benchmarks for language agents capable of meaningful value creation in expert settings [2603.07980].

Source: https://www.emergentmind.com/topics/onemillion-bench-1m-bench