GPQA-Diamond Benchmark

• GPQA-Diamond is a graduate-level benchmark featuring 448 expert-designed, multi-domain multiple-choice questions resistant to simple web retrieval.
• It employs rigorous evaluation protocols including accuracy metrics, McNemar’s test, and inter-model disagreement to assess deep reasoning capabilities.
• The benchmark exposes vulnerabilities such as data laundering and format leakage, highlighting the impact of orchestration and evaluation protocols on LLM performance.

GPQA-Diamond is a graduate-level, multi-domain, multiple-choice benchmark developed to rigorously evaluate the deep reasoning capabilities of LLMs in a setting intentionally designed to resist web search (“Google-proofing”) and to expose systematic weaknesses in model orchestration and evaluation protocols. Its design and downstream applications have catalyzed critical reexamination of accuracy-based leaderboards, measurement of inter-model disagreement, and the need for robust anti-contamination protocols in model evaluation (Tian et al., 28 Sep 2025, Yang et al., 12 Feb 2026, Mansurov et al., 2024).

1. Provenance, Scope, and Structure

GPQA-Diamond is derived from the GPQA suite of “expert-level,” graduate-style questions in biology, chemistry, and physics. The dataset consists of $N=448$ multiple-choice items, each with a question stem and four labeled answer options, drawn by subject-matter experts to be “Google-proof”—not answerable via direct web search or trivial retrieval (Yang et al., 12 Feb 2026). Questions focus on multi-step, domain-specific reasoning and typically target knowledge requiring higher-order synthesis rather than rote fact recall.

No preprocessing or item filtering beyond standard multiple-choice formatting is applied in principal studies. The same stem-plus-options structure is preserved, and evaluation is strictly via selection from the four presented choices. The benchmark’s input format and static test set have made it a canonical testbed for both single-model and multi-agent ensemble methodologies (Tian et al., 28 Sep 2025).

2. Evaluation Protocols and Measurement

The principal evaluation metric on GPQA-Diamond is accuracy: $$\text{Accuracy} = \frac{\#\text{correct answers}}{N}$$ or, more formally, for item set $T$ and model predictions $\hat{y}_t \in \{A, B, C, D\}$,

$$\text{Accuracy} = \frac{1}{|T|}\sum_{t \in T} \mathbf{1}[\hat{y}_t = y_t]$$

(Tian et al., 28 Sep 2025, Yang et al., 12 Feb 2026).

For comparative analysis between orchestration protocols and single-LLM baselines, the exact McNemar’s test is employed: for counts $b$ (orchestration correct, LLM wrong) and $c$ (reverse), the $p$-value is derived from the binomial distribution over $(b, c)$ pairs. Additional outcome rates specific to the multi-agent setting include:

• Self-Voting Rate: $(\text{# self-votes}) / (\text{# total votes})$
• First-Voted-Selected Rate: fraction of items where the first answer proposed is ultimately selected by majority
• Consensus Tie Rate: fraction of items where no answer attains strict majority (i.e., stalemates) (Tian et al., 28 Sep 2025).

To assess reproducibility and eliminate decoding randomness, all single-LLM experiments utilize deterministic decoding (greedy, $$\text{Accuracy} = \frac{\#\text{correct answers}}{N}$$0), with environments controlled for strict repeatability; confidence thresholds are not applied, as models output a single best guess for each item (Yang et al., 12 Feb 2026).

3. Baseline Model Performance and Orchestration Outcomes

GPQA-Diamond serves as an evaluation ground for both individual state-of-the-art LLMs and orchestrated, multi-agent ensembles:

Model	GPQA-Diamond Accuracy
Grok 4	85.4%
GPT-5	84.8%
Gemini 2.5 Pro	85.9%
Claude Sonnet 4	68.2%
Orchestration	87.4%

(Tian et al., 28 Sep 2025)

The orchestration protocol involves four heterogeneous agents (each a different LLM), proceeding asynchronously through “propose or vote” phases. If new proposals arise during voting, a dynamic restart invalidates previous votes. Consensus (majority) is determined when all agents have proposed and voted; ties without strict majority are recorded. The final answer is synthesized by the agent whose answer attains majority, aggregating reasoning and votes.

With orchestration, 87.4% accuracy is achieved—statistically matching or exceeding the strongest single LLM, confirming that ensemble consensus can outperform all underlying components if coordinated effectively. McNemar’s test versus Gemini 2.5 Pro yields $$\text{Accuracy} = \frac{\#\text{correct answers}}{N}$$1 (not significant), while the difference with Claude Sonnet 4 is highly significant ($$\text{Accuracy} = \frac{\#\text{correct answers}}{N}$$2), indicating robust benefits over weaker baselines (Tian et al., 28 Sep 2025).

4. Model Disagreement, Robustness, and Benchmark Illusions

Subtle but consequential disagreement exists between high-performing models. Pairwise inter-model disagreement is quantified as: $$\text{Accuracy} = \frac{\#\text{correct answers}}{N}$$3 where $$\text{Accuracy} = \frac{\#\text{correct answers}}{N}$$4 denotes the rate at which models $$\text{Accuracy} = \frac{\#\text{correct answers}}{N}$$5 and $$\text{Accuracy} = \frac{\#\text{correct answers}}{N}$$6 select different answers for the same item (Yang et al., 12 Feb 2026).

Empirical findings:

• Across all model pairs in benchmark-scale runs ($$\text{Accuracy} = \frac{\#\text{correct answers}}{N}$$720 models), $$\text{Accuracy} = \frac{\#\text{correct answers}}{N}$$8 ranges from 17% to 70%
• Among models exceeding 60% accuracy (“frontier” tier), $$\text{Accuracy} = \frac{\#\text{correct answers}}{N}$$9 ranges from 17% to 32%
• Even models with matched overall accuracy can disagree on up to 32% of individual question responses

This reveals a “benchmark illusion”: high accuracy does not guarantee epistemic agreement. In downstream scientific applications, such as automated annotation, the choice of LLM introduces hidden variability—model selection alone can flip 1 in 3 predictions and alter statistical inferences, such as experimental treatment effects, by $T$080% in real data scenarios (Yang et al., 12 Feb 2026). This signifies that model identity is a hidden but consequence-laden degree of freedom in empirical science.

5. Vulnerabilities: Data Laundering and Evaluation Integrity

GPQA-Diamond has been instrumental in exposing the ease with which knowledge can be covertly transferred into models via knowledge distillation—termed “data laundering” (Mansurov et al., 2024). The process is:

1. Placement Phase: Fine-tune a large teacher model directly on the GPQA test set (“dirty” knowledge).
2. Layering Phase: Distill this teacher via standard knowledge distillation onto a student using a distinct intermediate dataset (e.g., MedMCQA, RACE) with a convex mixture of hard and soft-label loss.
3. Integration Phase: Evaluate the student—which never directly saw GPQA—on the GPQA-Diamond test set.

Surprisingly, a 2-layer BERT student can attain 74.75% accuracy, approaching state-of-the-art LLMs (OpenAI o1 at 77.3%), despite not developing genuine reasoning skill. Format-based leakage, not semantic understanding, drives the gain. Similar inflation occurs even when distillation data is corrupted (e.g., random answer choices), strongly suggesting the static format and prompt structure themselves form a covert channel for illicit information transfer.

Key findings:

• Test-set leakage via distillation can occur even when intermediate data is meaningless or highly corrupted (e.g., random strings as answers yield 56.57%)
• Iterative distillation preserves high accuracy, evidencing latent information retention
• This exposes a critical fragility: multiple-choice benchmarks are vulnerable even under superficially “clean” evaluation practices (Mansurov et al., 2024)

6. Orchestration Dynamics, Coordination, and Ablation Insights

Multi-agent orchestration on GPQA-Diamond is sensitive to both agent-identity and voting configuration:

Coordination-lever (Ablation) Findings

1. Authorship Disclosure:
   • Anonymous voting: GPT-5’s self-voting rate is 81.0%; consensus tie rate is 14.1%
   • Identified voting: GPT-5’s self-voting rate rises to 88.4%; consensus tie rate climbs to 23.2%
   • Tied tasks with at least two agents self-voting nearly double under identity exposure
   • Skew toward strong brand selection emerges, reducing weaker model impact
2. Vote Tally Visibility:
   • Hidden tally: First-voted-selected rate is 54.1%
   • Visible tally: Rate jumps to 67.8%, with individual agent win rates surging up to +40 percentage points for GPT-5
   • Explicit herding language is observed; visible tallies foster social-proof dynamics, accelerating but sometimes misguiding convergence (Tian et al., 28 Sep 2025)

These results highlight that simple changes to voting transparency and identity can drastically reshape ensemble dynamics, increasing susceptibility to social or reputational biases and premature consensus.

7. Headroom, Methodological Implications, and Recommendations

A critical observation is the discrepancy between achieved and best-achievable orchestration accuracy. On GPQA-Diamond, at least one agent is correct for 95.5% of tasks, but the actual orchestration achieves only 87.4%. In 64% of ensemble failures, at least one agent chose the correct answer; in 31% of these, two or more agents had converged on the correct answer, but the majority still selected wrongly. This ~8.1 percentage point gap illustrates headroom for improved consensus mechanisms—suggesting further work on adaptive aggregation, such as weighting, cross-validation, or confidence-based voting (Tian et al., 28 Sep 2025).

To address benchmark illusion and model identity confounding, recommendations have emerged:

• Publish both accuracy and pairwise disagreement $T$1 scores alongside leaderboards
• Introduce “stability” tracks (prediction change under perturbation)
• Partition results by topic and difficulty
• Publish human-graded calibration sets for uncertainty estimation
• Require ensemble or consensus agreement thresholds for critical applications
• Field inferential robustness audits to understand label-driven impact on scientific conclusions (Yang et al., 12 Feb 2026)

For contamination resilience, the following are recommended:

• Server-side, private test evaluation (no item distribution)
• Dynamic formatting and randomized answer order
• Adversarial inclusion of trap (“honeypot”) items for contamination detection
• Regular contamination audits via model logit analysis (Mansurov et al., 2024)

These collective measures are necessary for GPQA-Diamond to function as a scientifically robust benchmark—informing not only absolute accuracy, but also the stability, agreement, and inferential reliability of LLMs in high-stakes research environments.