BrainBench: Predictive Neuroscience Benchmark

• BrainBench is a cutting-edge evaluation benchmark that measures predictive reasoning in neuroscience by contrasting original and altered abstracts.
• The benchmark employs perplexity-based forced choice, comparing LLM performance with human experts and highlighting complementary strengths.
• BrainBench’s robust design, including domain-specific fine-tuning and memorization checks, offers actionable insights for advancing AI in scientific discovery.

BrainBench is a forward-looking evaluation benchmark designed to assess whether LLMs trained on the scientific literature can predict the outcomes of neuroscience experiments—effectively testing the models’ capacity for integrative “forecasting” of novel results rather than merely recalling known facts. The benchmark is centered on comparing the predictive performance of LLMs, including both general-purpose and domain-specialized models, against human neuroscience experts. This methodology, which evaluates prospective inference from methods and background to results, provides a new paradigm for measuring scientific reasoning and predictive ability in artificial intelligence.

1. Purpose and Conceptual Motivation

The primary aim of BrainBench is to operationalize the forward-looking abilities required for scientific discovery within the context of neuroscience. Unlike conventional benchmarks, which are mostly retrospective—probing for factual recall or established reasoning—BrainBench tasks require integrating large spans of noisy, heterogeneous scientific evidence to anticipate the outcomes of previously unpublished, real experimental studies. This reflects the type of expert synthesis underpinning actual scientific progress and addresses a previously unmeasured capability for LLMs.

There were no prior benchmarks evaluating an LLM’s ability to “forecast” the outcomes of experiments. BrainBench aims to fill this gap, supporting investigations into whether LLMs can generalize experimental knowledge across methodologies, domains, and contexts.

2. Benchmark Construction and Dataset

Each BrainBench test item consists of an abstract pair from a recent (2023) Journal of Neuroscience paper:

• The original abstract is unmodified and contains the true results.
• The altered abstract has its results section meaningfully rewritten (by a domain expert or by GPT-4), modifying the scientific findings while maintaining consistency and textual coherence. The background and methods remain unchanged.

Participants (model or human) are tasked with identifying which abstract reflects the actual result.

Dataset specifics:

• 200 test cases were created by neuroscience experts; an additional 100 by GPT-4, all using source material from recent literature.
• Abstracts span five primary neuroscience subfields: Behavioral/Cognitive, Systems/Circuits, Neurobiology of Disease, Cellular/Molecular, and Development/Plasticity/Repair.
• Edits made to the results sections involve logical reversals, swapped mappings between variables, or other modifications ensuring plausible but non-trivial alternative interpretations.
• Both human and GPT-4 curation ensured that alterations could not be trivially detected based on linguistic or shallow semantic cues.

Table 1: Benchmark Construction Overview

Aspect	Details
Abstract Sources	300 abstracts from 2023 Journal of Neuroscience; 200 human, 100 GPT-4
Subfields	5 distinct neuroscience domains
Alteration Method	Expert or GPT-4 edits results; maintains background/methods integrity
Task Type	Binary forced-choice: original vs. altered results

3. Evaluation Procedures

Human Expert Evaluation

• 202 neuroscience experts were recruited; data from 171 were retained post-quality filtering (mean experience 10.1 years; includes doctoral students, postdocs, faculty).
• Each participant considered at least 9 cases and, per case:
  • Chose the abstract with the original result,
  • Indicated confidence (slider),
  • Reported self-assessed expertise,
  • Noted prior familiarity with the paper to control for recall effects.

LLM Evaluation

• For each abstract pair, both texts were input to the LLM, each prefixed by the prompt: “You are a neuroscientist with deep knowledge in neuroscience. Here is an abstract...”
• Rather than requesting explicit categorical output, the evaluation used perplexity ($PPL$), measuring the negative log-likelihood of the abstract given the model.
• The model “selects” the abstract with the lower perplexity (higher probability under the model).

Mathematical Formulation

Given an abstract $X = (x_0, x_1,\dots,x_t)$: $$PPL(X) = \exp \left\{ -\frac{1}{t} \sum_{i=1}^t \log p_\theta(x_i | x_{<i}) \right\}$$ where $p_\theta(x_i | x_{<i})$ is the model-parameterized probability of token $x_i$.

The selected abstract: $$X_{chosen} = \begin{cases} X_{orig} & \text{if } PPL(X_{orig}) < PPL(X_{alt}) \ X_{alt} & \text{otherwise} \end{cases}$$

4. Metrics, Controls, and Statistical Analysis

Accuracy: Percentage of test cases where the participant correctly selects the original abstract.

Confidence calibration: For LLMs, absolute difference in perplexity between pair members serves as a “confidence” score. Binning by confidence allows assessment of the correspondence between confidence and accuracy. For humans, calibration utilizes the self-reported slider value.

Item difficulty correlation: Spearman correlation measures agreement between model and human accuracy/difficulty on individual items. For LLMs, item-level difficulty is indexed by PPL difference; for humans, by mean accuracy.

Generalization (integration analysis): LLMs’ performance is compared using either the full abstract or just the manipulated result section. Performance degradation when provided only the local context demonstrates reliance on integrating methodological and background information, rather than simple local pattern-matching.

Memorization check: To determine if models simply memorize abstracts seen during training, the ratio of zlib-compressed entropy to PPL is calculated: $ratio = \frac{ZLIB(X)}{PPL(X)}$ No relationship with abstract age or known inclusion in pretraining data was found, suggesting generalization rather than memorization.

Statistical significance: Paired $t$-tests are used to analyze the improvement from domain-specific fine-tuning (see Section 5).

5. LLMs, Domain Adaptation, and Predictive Performance

Key results:

• LLMs collectively achieve 81.4% accuracy; human experts score 63.4%. Even the top 20% of human experts achieve only 66.2%.
• LLMs outperform humans in all five neuroscience subdomains.
• Chat/instruction-optimized LLMs do worse than their base equivalents.
• Model parameter count is not a primary driver: 7B-parameter LLMs perform comparably to larger models.

BrainGPT (domain-specific model):

• Constructed by LoRA-based fine-tuning of Llama-2-chat (7B) on 1.3B neuroscience-specific tokens (covering abstracts and full texts from over 100 journals, years 2002–2022).
• Only small LoRA adapter weights (0.26% of parameters) are trained, leaving the remainder frozen.
• Fine-tuning yields a statistically significant ~3% accuracy increase and increased PPL separation ($t(199) = -6.3, p < .001$ for perplexity difference).

Complementary performance: LLMs’ item-level accuracy patterns are only weakly correlated with human difficulty ratings (Spearman $r = 0.15$), suggesting partially non-overlapping strengths.

6. Scientific and Methodological Implications

Calibration and human-AI teaming: Both models and human experts display robust calibration—greater confidence predicts higher accuracy. The weak correlation between LLM and human error patterns suggests complementary strengths, supporting the potential for combined decision-making frameworks.

Transferability: The methodology underlying BrainBench—the paired abstract construction, perplexity-based forced choice, calibration assessment, domain adaptation—applies broadly to any domain where outcomes can be forecast from methodological metadata and prior context. This includes any knowledge-intensive area in science.

Implications for scientific practice: BrainBench supports the use of LLMs in roles that require not only recall and reasoning, but evidence synthesis and predictive inference, and provides a structured forward-looking metric for evaluating AI systems in scientific workflows.

Table 2: Summary of Core Features and Outcomes

Benchmark Feature	Specification	Performance Outcome
Task Type	Predict real vs. altered paper results from abstract pairs	LLM (81.4%) > Human (63.4%)
Evaluation Metric	Forced-choice via perplexity minimization	BrainGPT ≈ +3% over baseline
Domains Covered	5 major neuroscience subfields	Outperformance in all domains
Statistical Tests	Paired $t$-test, Spearman correlations (LLM r=0.75; LLM-human r=0.15)	Results robust and significant
Generalization Controls	Memorization check (zlib/PPL), datedness, context integration analysis	Indicates generalization

7. Conclusion

BrainBench constitutes a rigorously controlled, prediction-based benchmark that enables direct comparison of human and LLM performance in scientific outcome forecasting. Its structure departs from recall-based paradigms by requiring evidence integration from background and methods to results, demonstrating that LLMs—including compact, domain-tuned models—can surpass expert humans in forecasting experimental neuroscience outcomes. Diagnostic analyses confirm LLMs' generalization rather than memorization, and the methodology is both modular and transferable to other scientific fields. BrainBench thus serves both as a measurement tool for model capabilities and as a template for constructing forward-looking scientific benchmarks across domains.