LitXBench: A Benchmark for Extracting Experiments from Scientific Literature

Abstract: Aggregating experimental data from papers enables materials scientists to build better property prediction models and to facilitate scientific discovery. Recently, interest has grown in extracting not only single material properties but also entire experimental measurements. To support this shift, we introduce LitXBench, a framework for benchmarking methods that extract experiments from literature. We also present LitXAlloy, a dense benchmark comprising 1426 total measurements from 19 alloy papers. By storing the benchmark's entries as Python objects, rather than text-based formats such as CSV or JSON, we improve auditability and enable programmatic data validation. We find that frontier LLMs, such as Gemini 3.1 Pro Preview, outperform existing multi-turn extraction pipelines by up to 0.37 F1. Our results suggest that this performance gap arises because extraction pipelines associate measurements with compositions rather than the processing steps that define a material.

• The paper introduces LitXBench, a benchmark that groups experimental measurements by process lineage for precise extraction.
• It shows that applying categorical canonicalization during extraction improves LLM performance with significant F1 score gains.
• The study highlights the value of code-native, auditable datasets for advancing reproducible research in materials informatics.

LitXBench: Formalizing Experiment Extraction from Scientific Literature

Motivation and Problem Statement

Data-driven materials science critically depends on aggregating high-fidelity experimental results from the literature. Historically, both manual curation efforts and community-aggregated resources have failed to sufficiently capture detailed experimental procedures and downstream sample characterizations, impeding both ML model development and the reusability of scientific results. The primary source of rich experimental data remains the body of unstructured scientific literature. Automating the extraction of complete experimental narratives from scientific documents—including synthesis pathways, process conditions, and all subsequent measurement properties—thus represents a central unsolved problem in materials informatics.

This paper introduces LitXBench, a benchmark suite rigorously designed to standardize and evaluate methods for extracting experiments from scientific papers. The authors clarify a crucial distinction: experiment extraction must group measurement results by unique physical samples, as defined by their complete synthesis paths, not merely by composition. The presented LitXAlloy benchmark operationalizes this by providing a dense, meticulously annotated dataset from 19 peer-reviewed alloy papers with 1,426 measurements, supporting fine-grained evaluation and robust auditability.

Key Principles for Accurate Experiment Extraction

Traditional extraction pipelines—particularly those driven by rule-based and NER-centric approaches—are inadequate for materials informatics because they fail to resolve context-dependent phenomena (e.g., disambiguation of identically named properties with different semantics, or tracking modifications in post-synthesis annealing). The paper articulates three cardinal principles for accurate experiment extraction:

1. Processing Lineage Primacy: Measurements must be associated with their exact synthesis and processing lineage, not merely composition, to faithfully distinguish unique samples even when nominal formulas are duplicated.
2. Category Canonicalization: All categorical information must be mapped to a canonical ontology to resolve semantically divergent but lexically overlapping terms, ensuring unambiguous aggregation and comparison across studies.
3. Native Code Representation: Storing extracted benchmarks as executable, extensible code objects (here, Python) vastly improves data mutability, auditability, and validation, far beyond static text (CSV/JSON) representations.

   Figure 1: The guiding principles for extraction: processing lineage association, categorical canonicalization, and code-native data models.

The LitXAlloy Benchmark and Data Model

The LitXAlloy dataset targets one of the most challenging domains—complex multi-step alloy synthesis—and includes comprehensive annotation of all physical samples made in the studied papers. Each sample is uniquely indexed by the synthesis chain, captured as a "process string" denoting chained process events (e.g., raw_material$\rightarrow$arc_melt$\rightarrow$anneal). Measurement records include not only scalar values but also context such as method of measurement, uncertainties, and sample microstructure, each mapped to an enforced ontology.

Figure 2: Schema of materials in LitXAlloy, illustrating process-step-based identification and association of measurement types.

Processing chains are defined and reused with parameterized templates, supporting code-based inference and bulk editing. Helper functions (e.g., for converting weight-percent additions to normalized compositions) are provided and documented, allowing full provenance tracking for derived values.

Figure 3: Helper function for composition normalization, ensuring unambiguous conversion and documentation of measurement provenance.

Material synthesis groups are programmatically defined and referred to in each sample, facilitating fine-grained tracking of the effect of processing parameters on downstream properties.

Figure 4: Parametric definition of synthesis step groups and reuse across material variants.

Samples can be defined as derived from other samples (nontrivial DAG relationships), accurately reflecting both real experimental hierarchy and dependencies documented in the literature.

Figure 5: Concrete instance of process definitions, supporting both direct-from-precursor and derived-from-base sample creation.

Benchmarking Extraction Methods

The study provides a strong empirical comparison of recent and frontier LLM-driven methods (Gemini, Claude, GPT-5, Codex) as well as pipeline/agentic approaches such as KnowMat2 and LeMat-Synth. Key findings include:

• Processing Lineage Indexing Outperforms Composition-centric Methods: Methods grouping by process lineage realize up to a 0.37 improvement in overall F1 (Gemini 3.1 Pro Preview achieves 0.77–0.80 F1 in end-to-end extraction, significantly outperforming multi-turn pipelines such as KnowMat2).
• Canonicalization during Extraction: LLMs achieve higher accuracy when canonicalization is performed at extraction time, as opposed to post hoc normalization, due to access to contextual cues.
• Task Decomposition: Extraction of clean compositional data (up to 0.99 F1, Gemini 3 Flash with code output) is much easier than extraction of complete syntheses or microstructural hierarchies, which remains challenging. Targeted per-property extraction (e.g., Vickers hardness $>$0.95 F1) demonstrates more reliable performance than extracting full experimental graph objects.
• Auditability and Error Correction: Code-native datasets allow programmatic validation (e.g., unit testing of process chain logic, ontology enforcement, provenance tracking) and support rapid, transparent correction of annotation errors as models or best practices evolve.

  Figure 6: Pareto front tracing the trade-offs and performance landscape across current experiment extraction methods.

Implications for Future AI and Materials Informatics

This work demonstrates that present LLMs, while capable of extracting compositions and certain process details with high accuracy, still exhibit substantial failures in associating indirect measurement references, tracking parent-child sample relationships, and resolving semantic ambiguities in nomenclature. The experimental results underscore that solving the general experiment extraction problem for scientific literature will require continued advances in both model architectures (especially for indirect, context-dependent inference) and benchmark design (favoring code-native, ontology-aligned, and auditably maintained corpora).

The explicit use of code for benchmark storage and validation anticipates increased convergence between data curation, validation, and ML deployment. As data complexity and scale grow, these practices will likely become standard for ensuring reproducibility and community-driven curation, paralleling trends in general computational science.

Conclusion

LitXBench provides a rigorously defined, ontology-aligned, and programmatically auditable benchmark for evaluating automated experiment extraction from scientific text. The work shows that accurate experiment extraction requires grouping measurements by their processing lineages, canonicalizing all categories, and favoring code-native representations for benchmark datasets. Empirical evaluation demonstrates significant advances and persistent limitations in current LLM-based methods, highlighting both the promise of AI-driven materials informatics and the complexity of real-world experiment extraction. The adoption of code-based, editable benchmarks is projected to become foundational for robust, scalable AI-driven scientific discovery in materials science and related computational domains.

(2604.07649)