SciR: Scientific Reasoning Benchmark

• SciR is a multi-paradigm scientific reasoning benchmark that tests deduction, induction, and causal abduction with controlled inference and extraction difficulty.
• It employs formal objects and domain-tuned scientific genres to generate tasks with verifiable, mechanistic ground truth for realistic evaluation.
• Its design decomposes performance into extraction and inference components, offering diagnostic insights to enhance neuro-symbolic and formal reasoning approaches.

Searching arXiv for the SciR benchmark and closely related scientific/formal reasoning benchmarks to ground citations. arXiv search: SciR benchmark (Beckmann et al., 11 Jun 2026); related benchmarks: ProofWriter, EntailmentBank, SciFact, QASPER, DiscoveryBench. SciR is a benchmark for scientific reasoning in LLMs that evaluates three paradigmatic forms of inference—deduction, induction, and causal abduction—under controllable scientific rendering. Its central design principle is to generate tasks from formal objects with verifiable answers and then render those tasks into multi-document scientific discourse through per-track domain-tuned genres. This makes it possible to vary, independently, how difficult it is to extract the key premises and how difficult it is to carry out the principled inference itself. To its authors’ knowledge, SciR is the first multi-paradigm scientific-reasoning benchmark with parametric control on both extraction and inference difficulty (Beckmann et al., 11 Jun 2026).

1. Conceptual position within reasoning benchmarks

SciR is motivated by a gap between two benchmark families. Formal reasoning benchmarks such as ProofWriter (Tafjord et al., 2020) and EntailmentBank (Yasunaga et al., 2021) provide verifiable ground truth and tunable logical complexity, but they live on templated or generic prose surfaces. Science-facing benchmarks such as SciFact (Wadden et al., 2020) and QASPER (Dasigi et al., 2021) embed real scientific text, but each covers only one paradigm, lacks parametric ground-truth difficulty, and yields human-annotated answers rather than mechanistic ones. SciR is defined explicitly as a benchmark that combines formal control, scientific grounding, and surface heterogeneity in order to bridge this gap (Beckmann et al., 11 Jun 2026).

The benchmark therefore does not reduce scientific reasoning to a single task family. Instead, it treats deduction, induction, and causal abduction as distinct inferential regimes that recur across scientific reasoning. In SciR, each regime is tied to a canonical biological problem: developmental biology syllogisms for deduction, drug–drug interaction rule learning for induction, and Sachs signalling network discovery for the causal track. This suggests a diagnostic aim rather than a purely aggregate one: performance can be decomposed by inferential family, by extraction burden, and by formal complexity.

A further distinguishing feature is the insistence on mechanistic ground truth. Answers are determined from a latent formal object before scientific rendering occurs. As a result, the benchmark is intended to preserve auditability while still testing model behavior on realistic scientific documents rather than on generic logical paraphrases.

2. Formal task construction and reasoning paradigms

SciR formalizes solving as a two-stage process. Let rendered documents be denoted by $\Delta=\{\delta_1,\dots,\delta_m\}$. Then task solution is written as

$$\Delta \xrightarrow{\mathcal{E}} \Gamma,\quad \Gamma \vdash_f y,$$

where $\mathcal{E}$ extracts typed premises $\Gamma$, and $\vdash_f$ is the family-appropriate inference relation. Generation proceeds in reverse:

$$z_f \xrightarrow{\mathrm{ground}} s_f \xrightarrow{\mathcal{O}} \Delta,$$

with formal object $z_f$, scientifically grounded state $s_f$, and rendering operator $\mathcal{O}$ (Beckmann et al., 11 Jun 2026).

In the deduction track, the formal object is a rooted tree of first-order-logic syllogisms, $T=(N,E)$. Nodes are formulas, and edges represent one-step entailments. The solver must decide whether the hypothesis is “valid,” “invalid,” or “unknown.” Inference is FOL entailment, written as $$\Delta \xrightarrow{\mathcal{E}} \Gamma,\quad \Gamma \vdash_f y,$$0. Validity can be altered during generation by negating the conclusion, and underdetermination can be induced by deleting one premise.

In the induction track, the formal object is a candidate rule hypothesis. The paper gives the representative form

$$\Delta \xrightarrow{\mathcal{E}} \Gamma,\quad \Gamma \vdash_f y,$$1

A single target rule is paired with distractor rules $$\Delta \xrightarrow{\mathcal{E}} \Gamma,\quad \Gamma \vdash_f y,$$2, positive examples $$\Delta \xrightarrow{\mathcal{E}} \Gamma,\quad \Gamma \vdash_f y,$$3, negative examples $$\Delta \xrightarrow{\mathcal{E}} \Gamma,\quad \Gamma \vdash_f y,$$4, and background facts $$\Delta \xrightarrow{\mathcal{E}} \Gamma,\quad \Gamma \vdash_f y,$$5. The task is to select the unique rule consistent with all positives and none of the negatives.

In the causal track, the formal object is a DAG $$\Delta \xrightarrow{\mathcal{E}} \Gamma,\quad \Gamma \vdash_f y,$$6. A subgraph $$\Delta \xrightarrow{\mathcal{E}} \Gamma,\quad \Gamma \vdash_f y,$$7 is sampled from the Sachs network, a fictional protein XYZ is added, and new incident edges $$\Delta \xrightarrow{\mathcal{E}} \Gamma,\quad \Gamma \vdash_f y,$$8 are introduced. A linear-Gaussian SCM is then simulated under observational and do-intervention regimes to produce a table $$\Delta \xrightarrow{\mathcal{E}} \Gamma,\quad \Gamma \vdash_f y,$$9. The task is to recover $\mathcal{E}$0 from $\mathcal{E}$1 given $\mathcal{E}$2.

Across all three tracks, answers follow deterministically from the formal object before rendering. The paper identifies Prover9, Popper, and GIES as the backends supporting this auditable correctness.

3. Scientific rendering and the two difficulty axes

SciR’s rendering stage is designed to make extraction a genuine problem rather than a formatting artifact. Structured premises are split into $\mathcal{E}$3 chunks, and each chunk $\mathcal{E}$4 is rewritten by an LLM $\mathcal{E}$5 into one of eight scientific genres per track. An invertibility contract is imposed: a second LLM $\mathcal{E}$6 must recover $\mathcal{E}$7 from $\mathcal{E}$8 given the other chunks as context. Only preserved rewrites are retained, after which distractor lines are interleaved to test true extraction rather than stylized template matching (Beckmann et al., 11 Jun 2026).

The benchmark then separates two difficulty axes. The first is inference complexity. In deduction, this is controlled by depth $\mathcal{E}$9, the number of chained expansions, and width $\Gamma$0, the number of Unknown distractor trees. In induction, it is controlled by the number of distractor rules $\Gamma$1 and the number of positive examples per rule $\Gamma$2. In the causal setting, it is controlled by subgraph size times number of new edges, $\Gamma$3, together with the number of samples per environment.

The second axis is premise obfuscation. This is controlled by chunk count $\Gamma$4, style pool size, and renderer capability. In practice, the benchmark uses $\Gamma$5 chunks per task and eight genres per track, and measures a rendering gap between clean natural language $\Gamma$6 and obfuscated rendering $\Gamma$7.

The paper also specifies concrete easy and hard parameterizations. Deduction Easy uses $\Gamma$8 and corresponds to approximately $\Gamma$9 premises, whereas Deduction Hard uses $\vdash_f$0 and corresponds to approximately $\vdash_f$1 premises. Induction Easy uses $\vdash_f$2, and Induction Hard uses $\vdash_f$3. Causal Easy uses subgraph $\vdash_f$4, while Causal Hard uses $\vdash_f$5. These parameterizations are central because they permit a controlled decomposition of failure modes into extraction difficulty and inference difficulty.

4. Benchmark organization, genres, and evaluation protocol

SciR is organized into three tracks, each with Easy and Hard tiers: Deduction, Induction, and Causal. Deduction is instantiated as developmental biology syllogisms; induction as drug–drug interaction rule learning; and causal reasoning as Sachs signalling network discovery (Beckmann et al., 11 Jun 2026).

Each track is rendered through a distinct genre inventory. Deduction uses genres such as scRNA-seq report, Reactome entry, dev-bio textbook, ChIP-seq report, FACS log, spatial-transcriptomics annotation, and differentiation protocol. Induction uses FDA label, DrugBank entry, clinical case, EHR summary, PubMed abstract, in-vitro study note, pharmacoepidemiology cohort study, and pharmacy consult. Causal tasks use wet-lab notebook, LC-MS/MS proteomics report, phospho-flow screen, Results section, PD biomarker report, perturbation screen, database entry, and supplementary dataset. The genre system is part of the benchmark definition rather than a superficial wrapper, because it operationalizes premise obfuscation in track-specific scientific discourse.

The evaluation protocol samples $\vdash_f$6 tasks per model, per track, per tier, and per rendering setting. Since there are $\vdash_f$7 tracks, $\vdash_f$8 tiers, and $\vdash_f$9 rendering modes, each model is evaluated on

$$z_f \xrightarrow{\mathrm{ground}} s_f \xrightarrow{\mathcal{O}} \Delta,$$0

tasks. The paper states that the generation code is open-source, so the full dataset size is unbounded and can be scaled as needed.

SciR evaluates six base models and configurations: gpt-4o, o3-mini, deepseek-r1, llama-3.3-70b, qwen3-30b, and olmo-3.1-32b. It compares three solving paradigms: direct chain-of-thought prompting, neuro-symbolic pipelines in which an LLM formalizer hands inference to a symbolic solver, and SymbCoT$$z_f \xrightarrow{\mathrm{ground}} s_f \xrightarrow{\mathcal{O}} \Delta,$$1, in which the same formalization is produced but the LLM answers from the formal text instead of delegating to a verified solver.

5. Empirical findings and diagnostic patterns

The benchmark reports chance-normalized accuracy averaged across the three tracks. For gpt-4o CoT, accuracy is $$z_f \xrightarrow{\mathrm{ground}} s_f \xrightarrow{\mathcal{O}} \Delta,$$2 on NL Easy, $$z_f \xrightarrow{\mathrm{ground}} s_f \xrightarrow{\mathcal{O}} \Delta,$$3 on NL Hard, $$z_f \xrightarrow{\mathrm{ground}} s_f \xrightarrow{\mathcal{O}} \Delta,$$4 on Obf Easy, and $$z_f \xrightarrow{\mathrm{ground}} s_f \xrightarrow{\mathcal{O}} \Delta,$$5 on Obf Hard. For gpt-4o NS, the corresponding values are $$z_f \xrightarrow{\mathrm{ground}} s_f \xrightarrow{\mathcal{O}} \Delta,$$6, $$z_f \xrightarrow{\mathrm{ground}} s_f \xrightarrow{\mathcal{O}} \Delta,$$7, $$z_f \xrightarrow{\mathrm{ground}} s_f \xrightarrow{\mathcal{O}} \Delta,$$8, and $$z_f \xrightarrow{\mathrm{ground}} s_f \xrightarrow{\mathcal{O}} \Delta,$$9. For o3-mini CoT, they are $z_f$0, $z_f$1, $z_f$2, and $z_f$3; for o3-mini NS, $z_f$4, $z_f$5, $z_f$6, and $z_f$7. For deepseek-r1 CoT, the values are $z_f$8, $z_f$9, $s_f$0, and $s_f$1; for deepseek-r1 NS, $s_f$2, $s_f$3, $s_f$4, and $s_f$5 (Beckmann et al., 11 Jun 2026).

Several findings structure the interpretation of these numbers. First, both inference complexity and rendering obfuscation significantly degrade performance for every model, and their effects compound. The benchmark’s motivating example is gpt-4o CoT, which drops from $s_f$6 across NL Easy, NL Hard, Obf Easy, and Obf Hard. Second, the obfuscation gap also affects neuro-symbolic pipelines. The paper highlights that gpt-4o NS falls from $s_f$7 in clean natural language to $s_f$8 under obfuscation, indicating that verified inference backends do not eliminate the need for robust extraction.

Third, SymbCoT$s_f$9 trails neuro-symbolic solving, especially on induction, which the authors interpret as evidence that simply formalizing and then returning the answer generation to an LLM is inferior to a verified symbolic solve. Fourth, the two-axis setup yields a per-model extraction-vs-inference profile. The paper states that reasoning models such as deepseek-r1 and o3-mini score higher on both axes but pull further ahead on principled inference, whereas instruct models such as llama and olmo tend to be extraction-leaning or balanced.

Finally, difficulty is not uniform across tracks. Deduction is characterized as extraction-harder, causal as inference-harder, and induction as roughly balanced. This track asymmetry is important because it means aggregate accuracy alone obscures which components of scientific reasoning are actually failing.

6. Significance, interpretation, and open directions

SciR’s primary significance lies in its two-axis diagnosis of model behavior. Because the benchmark separates “not finding the premises” from “not applying the correct reasoning,” it supports a finer-grained analysis than benchmarks in which document realism and inferential complexity are entangled (Beckmann et al., 11 Jun 2026).

This also reframes the role of neuro-symbolic methods. SciR does not show that verified symbolic backends are unhelpful; on the contrary, neuro-symbolic approaches still benefit from those backends. What it does show is that the extraction and formalization stages remain bottlenecks even when the inference stage is delegated to a verified solver. A plausible implication is that progress on scientific reasoning will depend not only on stronger formal reasoning modules, but also on stronger information extraction from heterogeneous scientific discourse.

The benchmark further supports a distinction between benchmark realism and benchmark controllability. Existing science-facing benchmarks contain real scientific text but typically lack parametric ground-truth difficulty. Formal reasoning benchmarks offer controllability but do not resemble real scientific documents. SciR is designed so that both properties are present simultaneously through formal generation, scientific grounding, and multi-genre rendering. This suggests a methodological template for future benchmarks in which mechanistic correctness and realistic document surfaces are treated as coequal requirements.

The paper’s stated implications are correspondingly concrete. Future benchmarks and LLM development should jointly target robust information extraction from noisy scientific discourse and principled, mechanistic inference compatible with domain formalisms. In that sense, SciR functions not only as an evaluation dataset but also as a decomposition of the scientific-reasoning problem into auditable extraction and inference components.