Sci-Reasoning Dataset Overview

Updated 10 January 2026

Sci-Reasoning Dataset is a structured resource capturing the intellectual synthesis behind elite AI research through detailed, annotated reasoning links.
It employs a multi-stage pipeline combining advanced LLMs with human validation to extract citation contexts and synthesize narrative links.
The dataset supports quantitative analysis of scientific progress and evaluation of AI research agents by mapping innovation patterns within top-tier papers.

The Sci-Reasoning Dataset encompasses a new genre of structured resources explicitly designed to capture, analyze, and emulate the intellectual processes underlying scientific innovation, primarily in the context of artificial intelligence research. Unlike conventional question-answer or fact retrieval corpora, the Sci-Reasoning Dataset provides mechanistic traces of how high-impact scientific contributions build upon and synthesize prior knowledge, rendering the tacit reasoning processes behind breakthroughs quantitatively accessible and actionable for both AI agents and meta-scientific inquiry (Liu et al., 8 Jan 2026).

1. Dataset Scope and Composition

The Sci-Reasoning Dataset targets the intellectual synthesis behind cutting-edge AI research, with a focus on reconstructing the reasoning trajectories connecting new work to influential predecessors. The resource is built by tracing 3,819 Oral and Spotlight papers accepted at NeurIPS, ICML, and ICLR (2023–2025), corresponding to approximately the top 1–5% of submissions in these venues. It contains:

Approximately 20,000 “lineage links,” each explicating the reasoning connection between a target paper and one of its 5–10 key predecessors.
∼100,000 structured annotation instances, covering fields such as predecessor role, reasoning relationship type, and full narrative synthesis.
Exclusions: Includes only primary research contributions; surveys, benchmark-only, and position papers are omitted.

This resource enables analysis and training of agents capable of “scientific reasoning” as it is practiced in elite AI research, rather than mere fact recall or shallow citation chaining.

2. Construction Pipeline and Annotation Schema

Dataset construction employs a multi-stage pipeline combining advanced LLMs (GPT-5) with human-in-the-loop curation:

Target Paper Identification: Extraction of all Oral and Spotlight presentations from the official proceedings of NeurIPS, ICML, and ICLR (2023–2025), filtering to only main research works.
Intellectual Lineage Extraction:
- The full text of each target paper is parsed for all in-text citations.
- Citation contexts are analyzed to determine semantic roles (e.g., “building upon,” “addresses limitation of,” “inspired by”).
- Citations are ranked by frequency and methodological diversity to select 5–10 key predecessors per paper, annotated with their function: KEY_METHODOLOGY, FOUNDATIONAL_CONCEPT, PRIMARY_BASELINE, PROBLEM_FORMULATION, ENABLING_TOOL, or INSPIRATION_BY_ANALOGY.
Reasoning Link Synthesis:
- For each target–predecessor pair, the model generates a 200–400 word narrative capturing the “intellectual synthesis,” e.g., what knowledge was transferred or recombined.
- Each link receives a relationship type (EXTENDS, COMBINES_WITH, BRIDGES_GAP_BETWEEN, ADDRESSES_LIMITATION_OF, REFRAMES_USING) and confidence score $s_{ij}\in [0,1]$ .
Human Validation:
- For any link with $s_{ij} < 0.7$ or low cross-model agreement (checked using Claude Opus 4.5, Gemini 3.0), human expert review is invoked.
- The predecessor extraction protocol achieves 89.73% recall against a gold-annotated validation set.

All records are emitted in a structured JSON schema that supports downstream graph analysis, table summarization, and supervised finetuning (Liu et al., 8 Jan 2026).

3. Taxonomy of Innovation Patterns

A key contribution is the construction of a 15-category taxonomy of “thinking patterns”—archetypal modes of scientific reasoning observed in top-tier AI research. The three most prevalent account for over half of observed innovation links:

P01: Gap-Driven Reframing (24.2%) – The problem or approach is reformulated to spotlight a previously unaddressed limitation.
P02: Cross-Domain Synthesis (18.0%) – Methods, theories, or tools from one domain are integrated or adapted into another.
P03: Representation Shift (10.5%) – Key primitives or representations are fundamentally altered (e.g., new architectures, recasting continuous as discrete).
The remaining 12 patterns cover modular pipeline composition, principled probabilistic modeling, inductive bias injection, data-centric optimization, and more.

Combination analysis reveals frequent co-occurrence: for example, Gap-Driven Reframing + Representation Shift is an especially “powerful” recipe, as is Cross-Domain Synthesis + Representation Shift. The innovation “strength” of a paper can be viewed as the sum of interpretive weights over the patterns present.

4. Dataset Format, Access, and Representative Records

The resource is distributed in multiple formats to support diverse analysis pipelines:

JSON: Each paper (e.g., “Andes2024.json”) includes metadata and an array of predecessor link records, each with all annotated fields.
CSV: A flat table of all target–predecessor pairings with fields: target_id, source_id, predecessor_role, relationship_type, narrative, confidence.
Graph Database: Neo4j-JSON for direct graph traversal.

Example link record:

{
  "target_id": "vLLM2024",
  "source_id": "PagedAttention2023",
  "predecessor_role": "KEY_METHODOLOGY",
  "relationship_type": "EXTENDS",
  "narrative": "vLLM’s continuous batching and PagedAttention KV-cache design provided the baseline memory management that Andes repurposes for token-level QoE scheduling…",
  "confidence": 0.82
}

All artifacts are accessible via GitHub and HuggingFace ((Liu et al., 8 Jan 2026); https://github.com/AmberLJC/Sci-Reasoning; https://huggingface.co/datasets/AmberLJC/Sci-Reasoning).

5. Evaluation Experiments and Empirical Findings

The dataset supports evaluation of both LLMs and “research agent” architectures on meta-scientific and ideation tasks. Two notable evaluations are:

Research Ideation: Given the set of predecessors for a (held-out) NeurIPS 2025 Oral paper, can an LLM propose at least one idea (out of $k=10$ ) matched to the real paper? Hit@10 metric: Gemini 2.5 Pro achieves 49.35%, Claude Opus 4 42.86%, GPT-5.2 38.89%, Claude Sonnet 4 29.87%.
Predecessor Extraction: On a 77-paper gold validation set, GPT-5 achieves 89.73% recall (vs. GPT-5.2 87.47%, GPT-4.1 78.00%, GPT-5-mini 68.53%).

This establishes a robust upper bound for automated reasoning link extraction and research trajectory prediction.

6. Applications and Impact

The Sci-Reasoning Dataset enables:

Quantitative studies of scientific progress, such as tracking the frequency and impact of higher-level reasoning patterns (e.g., how rapidly gap-driven reframing strategies propagate).
Training AI research agents to imitate expert-level “reasoning moves,” supporting next-generation meta-scientific automation.
Benchmarking the predictive capacity of LLMs for research ideation and direction-finding given a set of inputs.
Lineage graph construction and analysis for mapping the evolution of intellectual fields.
Systematic evaluation of innovation “recipes” – i.e., which combinations of patterns recur in the most influential works.

7. Limitations and Future Prospects

Limitations include:

Coverage restricted to top-ranked AI papers from NeurIPS, ICML, and ICLR (2023–2025); broader domain generalization will require analogous pipelines in other fields.
Pattern taxonomy and “narrative” labels, while validated, may encode modeling bias from LLM-generated summarization.
No formal causal attribution is inferred, only high-confidence narrative synthesis and structural graph data.

Future directions may extend to:

Incorporation of more granular reasoning steps (e.g., explicit stepwise abduction and hypothesis testing rather than summary narratives).
Extension to benchmark and dataset papers, as well as broader scientific domains.
RL-based agent training for longitudinal scientific innovation (“meta-research agents”).

References: (Liu et al., 8 Jan 2026)