- The paper introduces IsoSci, a controlled benchmark that isolates reasoning from knowledge retrieval in LLM scientific problem solving.
- It employs isomorphic cross-domain problem pairs to reveal that 91.3% of reasoning-mode improvements are driven by enhanced domain knowledge retrieval.
- Results challenge the value of reasoning toggles, showing minimal gains and benchmark-specific performance reversals among different LLM models.
IsoSci: Disentangling Reasoning from Knowledge Retrieval in LLM Scientific Problem Solving
Motivation and Background
Prevailing LLM benchmarks in science (e.g., GPQA, MMLU-STEM, SciBench) conflate two foundational capabilities: retrieval of domain-specific knowledge and application of multi-step logical reasoning. This confound prevents attribution of performance improvements to either true procedural reasoning or extended factual recall. Techniques such as chain-of-thought (CoT) prompting or reasoning-focused training are often reported to yield accuracy gains, but such aggregates do not isolate whether observed improvements are due to enhanced reasoning per se or to improved surfacing of relevant domain knowledge during prompt processing.
The necessity for precise decomposition is emphasized by the frequent observation that failure on scientific questions is ambiguous, with both insufficient knowledge retrieval and flawed reasoning able to explain model errors. Previous attempts to disentangle these factors—such as stratified item labeling [thapa-etal-2026-reasoning]—do not offer structural guarantees and cannot control for the equivalence of solution procedure.
IsoSci Benchmark Construction
IsoSci directly addresses this attribution problem via a benchmark of isomorphic, cross-domain scientific problem pairs. Each pair consists of two questions from distinct scientific domains, sharing an identical multi-step solution structure but differing exclusively in the set of required domain knowledge (i.e., formulas, constants, entities). Formally, each pair (q,q′) obeys precise properties: differing domains, identical structure type, existence of a bijection between the knowledge atoms required for each, and pairwise disjoint knowledge sets.
Five structure types capturing short-horizon (3–5 step) scientific problem solving are represented:
- Formula Recall and Substitution
- Unit Conversion Chains
- Conservation Law Application
- Proportional Reasoning
- Two-Step Causal Qualitative Chains
Dataset assembly leverages high-coverage seed pools from canonical science benchmarks, synthetic generation for underrepresented domains, LLM-assisted isomorphic partner generation, and multi-judge LLM verification for both solution procedure and domain independence. Human audits demonstrate LLM-judge decisions approach expert precision (F1=0.889); the final release contains 144 pairs (288 problems) stratified across physics, chemistry, biology, and earth science.
Methods: Decoupling Reasoning and Knowledge
The core experimental protocol evaluates multiple LLM model pairs (∣Π∣=5), encompassing both reasoning-specialized vs. standard models and “toggle” comparisons where a reasoning flag is enabled or disabled at inference time, eliminating pretraining confounds. Each model is evaluated zero-shot on prompt templates emphasizing explicit step-by-step reasoning.
To quantify attribution, the pknow metric is introduced: for each isomorphic pair, reasoning-mode gains are tracked as source-only, target-only (knowledge-dependent), or on both members (structure-invariant/true reasoning). The knowledge-dependence ratio pknow is thus the fraction of observed reasoning-mode gains that do not transfer across the isomorphic pair and must be attributed to improved domain knowledge retrieval.
Robustness is ensured via exclusion analyses, label permutation tests, and McNemar’s test for discordant outcome patterns in toggle comparisons.
Results
Dominance of Knowledge-Dependent Gains
Across all model pairs and domains, the vast majority of reasoning-mode accuracy improvements are knowledge-dependent: reasoning-mode “wins” on one member of an isomorphic pair fail to transfer to its structural twin in the other domain. The pooled estimate across all model pairs is pknow=91.3% (Wilson 95% CI [82.3%, 96.0%]); for toggle pairs, pknow ranges from 89.5% to 100% (see Table below).
Minimal Gains from Reasoning Toggles
Reasoning toggles on high-capacity models yield at most 4pp domain-stratified accuracy gains (all CIs include zero). The “reasoning” configuration confers no systematic, domain-general benefit on short-horizon problems, confirmed by McNemar’s tests on discordant pairwise item outcomes (p=1.0 for both toggle models).
Benchmark Sensitivity of Reasoning-specialized Models
Accuracy comparisons between reasoning-specialized and standard models (e.g., o3-mini vs. GPT-4o-mini) display dramatic benchmark-dependent reversals. o3-mini outperforms GPT-4o-mini on GPQA Diamond by 19pp but underperforms by 24.7pp on IsoSci and 19.2pp on MMLU-STEM. This indicates that reasoning-focused models may excel on benchmarks where knowledge retrieval is less limiting, but perform worse when exhaustive domain recall is essential.
Implications and Theoretical Impact
Epistemic Isolation of Reasoning
IsoSci’s isomorphic pair methodology provides the first controlled diagnostic isolating reasoning procedure from knowledge retrieval in LLMs. The decisive dominance of knowledge-dependent gains demonstrates that, for structured short-horizon scientific tasks, current reasoning mechanisms in SoTA LLMs effectively act as extended retrieval scaffolds rather than as engines of improved procedural inference. This challenges the canonical assumption that chain-of-thought and related reasoning modes directly enhance multi-step reasoning capabilities.
Limitations and Directions for Future Work
- IsoSci covers only short-horizon, information-complete, undergraduate-to-early-graduate procedural problems; long-horizon derivations and open-ended creative synthesis remain unexplored.
- Distillation of knowledge vs. inference may change for research-grade, symbolically intensive, or hypothesis-generating questions.
- The “reasoning toggle” paradigm relies on the vendor’s control interface and may not comprehensively modulate underlying cognitive behaviors.
Further investigation is warranted into the mechanisms by which LLMs surface and chain together knowledge atoms, and whether advances in reasoning supervision or multi-agent derivation frameworks enable more truly domain-general procedural transfer. Larger instantiations of isomorphic-pair methodology, spanning longer solution paths and richer problem structures, will be informative for both LLM development and cognitive modeling.
Conclusion
IsoSci introduces a principled methodology for evaluating LLM scientific reasoning disentangled from knowledge retrieval, and a public dataset supporting these controlled experiments. The evidence indicates that on short-horizon procedural science, reasoning-mode performance gains are almost entirely knowledge-dependent rather than structure-invariant; enabling inference-time reasoning provides, at best, marginal benefit for high-capability models in these settings. The choice of benchmark is decisive in assessing model “reasoning” competency. IsoSci and metrics such as pknow offer a new foundation for precision diagnostic evaluation of scientific reasoning, supporting deeper theoretical and practical advances in LLM development.
Reference:
"IsoSci: A Benchmark of Isomorphic Cross-Domain Science Problems for Evaluating Reasoning versus Knowledge Retrieval in LLMs" (2607.01431)