ScienceArena Evaluation Platform

• ScienceArena is an open, collaborative benchmarking system that rigorously evaluates foundation models on scientific literature tasks using pairwise comparisons.
• The platform integrates retrieval-augmented generation with a modular three-stage workflow, ensuring transparent question submission, model invocation, and expert voting.
• Expert annotators employ robust pairwise voting and advanced aggregation techniques, achieving high inter-annotator agreement and reliable performance metrics.

The ScienceArena (SciArena) Evaluation Platform is an open and collaborative benchmarking system designed for rigorous assessment of foundation models on scientific literature tasks. Distinct from static question-answer leaderboards, SciArena employs a community-driven, pairwise comparison methodology to evaluate open-ended, literature-grounded tasks that reflect real-world scholarly information needs. Integrating retrieval-augmented generation, model-agnostic evaluation, and a dynamic ranking system, SciArena positions itself as the canonical evaluation infrastructure for foundation models targeting scientific and academic workflows (Zhao et al., 1 Jul 2025).

1. System Architecture and Workflow

SciArena is organized as a modular, web-based platform composed of three principal stages: (a) question submission and retrieval, (b) model invocation and response collection, and (c) community voting and aggregation. The architecture ensures transparent logging, consistent moderation, and robust aggregation of expert annotations.

System pipeline:

1. Question Submission: Researchers submit questions via a moderated web form, filtering inappropriate or off-topic content by an OpenAI omni-moderation module.
2. Literature Retrieval Pipeline: Questions are decomposed using GPT-4.x for query generation, followed by contextual passage retrieval from scientific databases (via the Semantic Scholar API) and re-ranking (e.g., cross-encoder models) to select the top 30 abstract/snippet contexts.
3. Model Invocation: The system randomly samples two models from a curated pool. Each receives the original question and the selected literature contexts, then outputs a citation-attributed, plain-text response.
4. Voting Interface: Trusted annotators—domain experts across 20+ scientific fields—compare the anonymized responses side-by-side (Model A vs. Model B) and cast their preference: A, B, Tie, or Both Bad, with optional justification.
5. Data Storage and Processing: Platform logs include the question, retrieved contexts, both model outputs, votes, timestamp, and annotator ID. Aggregation uses the Bradley-Terry model for Elo-style ranking. The dynamic leaderboard and meta-evaluation splits are updated via a public API.

The platform’s technical scaffolding comprises a React-based interface, a persistently stored data warehouse, and a ranking/aggregation layer using bootstrapped confidence intervals.

2. Supported Models and Task Taxonomy

As of June 30, 2025, SciArena integrates 23 foundation models spanning both proprietary and open-source paradigms. This inclusivity facilitates wide comparative coverage:

Category	Model Examples
Proprietary	OpenAI o3, o4-mini, GPT-4.1-series; Google Gemini-2.5-Series; Anthropic Claude-4-series; xAI Grok3
Open-Source	Mistral-Small-3.1, Mistral-Medium-3; Meta Llama-4-series; Qwen3-series; DeepSeek-R1-series; MiniMax-M1

Tasks are intentionally open-ended to capture scientific information work:

• Literature Question Answering: Synthesis from multiple cited works to address current research problems.
• Conceptual Explanation: Domain-specific technical elucidation anchored in the literature.
• State-of-the-Art Assessment: Identification of field trends and emerging directions.
• Methodology Inquiry: Discussion of research design, protocols, and approaches.
• Paper Finding/Evidence Synthesis: Locating and aggregating primary literature for a given query.
• Cross-disciplinary, unclassified questions are also represented.

Distribution across task types over 13,204 votes (as of data cutoff): Conceptual Explanation (35.2%), State-of-the-Art Assessment (23.9%), Challenges & Limitations (23.4%), Methodology Inquiry (9.3%), Paper Finding (4.5%), Others (3.8%).

3. Community-Driven Evaluation Protocol

Evaluation is fundamentally benchmarked by human preference through side-by-side comparisons. The process is as follows:

1. Two models are selected at random for each question.
2. Each provides a citation-attributed, long-form response.
3. Expert annotators (n=102) in natural sciences, engineering, humanities/social sciences, and healthcare undertake the pairwise evaluation after a calibration session.
4. Four annotation outcomes: A (prefer), B (prefer), Tie (equally good), Both Bad (neither sufficient).
5. Quality control leverages sequential p-value testing and Fisher’s method for anomaly detection, admitting only qualified votes.

Evaluation metrics:

• Inter-Annotator Agreement (IAA): Weighted Cohen’s $\kappa = \frac{p_o - p_e}{1 - p_e}$ where $p_o$ is observed, $p_e$ is chance agreement.
• Self-Consistency Score: $$S = \frac{1}{N} \sum_{i=1}^N I\bigl(v_i^{(1)} = v_i^{(2)}\bigr)$$, where votes on the same instance are spaced by two or more weeks.

Observed statistics indicate IAA avg. accuracy 0.82 ($\kappa=0.76$) and self-consistency 0.94 ($\kappa=0.91$), signaling robust reliability in researcher preferences.

4. Leaderboard Results, Case Studies, and Performance Insights

The dynamic leaderboard is a central feature, ranking each model via Elo ratings estimated by the Bradley-Terry model. Model o3 leads overall (1172.5 Elo), followed by Claude-4-Opus (1080.5), Gemini-2.5-Pro (1063.0), DeepSeek-R1-0528 (1061.9), and o4-mini (1053.9), with open-source and lighter-weight models generally trailing.

Domain-specific distinctions:

• o3 excels notably in Natural Sciences and Engineering (Elo > 1150).
• Claude-4-Opus demonstrates relative strength in Humanities/Social Sciences and Healthcare.

Common failure modes (when both responses are classified as Both Bad):

• Incomplete or omitted answers.
• Citation conflicts or irrelevance.
• Insufficient detail.
• Misunderstanding of scientific terminology.
• Disorganized or incoherent logical flow.

User preference bias is reduced compared to general-purpose evaluation arenas: relevance and correct attribution are more heavily weighted than sheer length or citation count.

5. SciArena-Eval: Meta-Evaluation Benchmark

SciArena-Eval is a meta-evaluation benchmark released to facilitate research into automated model-based evaluation:

• Dataset construction: 2,000 pairwise comparisons, drawn from SciArena votes (500 per discipline), excluding Ties.
• Task: An automated evaluator is given the question and two model responses, and must select the response aligned with the human vote.
• Metrics: Accuracy ($$\mathrm{Acc} = \# \mathrm{correct~model~judgments}/\# \mathrm{total~instances}$$), as well as precision/recall for claims of superiority (A better, B better).

Performance:

• Best evaluator (o3) achieves 65.1% alignment with human judgments (compared to random 50.0%).
• Other evaluators: o4-mini (64.8%), Llama-4-Maverick (57.5%).

This sustained gap between human and automated evaluators highlights the complexity of long-form, citation-grounded scientific literature tasks.

6. Core Challenges, Empirical Insights, and Prospects

Principal challenges:

• Scientific queries demand nuanced domain knowledge and robust retrieval; static metrics and LLM-as-judge approaches often fail to match expert preferences.
• Human evaluation is resource-intensive; relying solely on crowdsourced arenas risks bias and inconsistent quality.

Key empirical insights:

• Arena-style voting by domain experts achieves high inter-annotator agreement and self-consistency.
• Evaluation on SciArena displays lower susceptibility to stylistic or superficial biases than in general-purpose chatbot arenas.
• Automated LLM-based evaluators consistently underperform human experts on the platform’s core tasks, with performance plateauing at ~65% agreement.

Planned future directions:

• Incorporation of enhanced, domain-centered automatic metrics (e.g., citation factuality, argumentation structure).
• Continued expansion of model coverage, including agent-based research assistants as access becomes viable.
• Feature upgrades, such as multi-way comparisons, style-invariant controls, and question clustering for diagnostic analysis.
• Exploiting annotated data to train supervised evaluators specialized for scientific literature tasks.

By integrating retrieval, rigorous blinded annotation, robust aggregation, and open benchmarking infrastructure, the ScienceArena Evaluation Platform establishes a new standard for transparent and reliable evaluation of foundation models in scientific literature analysis (Zhao et al., 1 Jul 2025).