LitXBench: Experiment Extraction Benchmark

• LitXBench is a benchmarking framework that extracts full experiments from scientific papers by linking materials, synthesis processes, and measurement data.
• It uses a Python object-based schema to ensure auditability, editability, and validation of experimental extraction, setting it apart from composition-only methods.
• Its first benchmark, LitXAlloy, demonstrates high-density annotations with 1426 measurements across alloy papers, emphasizing a process-centric approach to material identity.

LitXBench is a benchmarking framework for evaluating methods that extract experiments from scientific literature, motivated by a shift from extracting isolated material properties to extracting full experiments from full papers. In its formalization, a paper contains a set of synthesized materials $M$; each material $m_i$ is created from an overview process $p_i$ and has measurements $x_i$; the experiment extraction task is to output all measurements $x_i$ for each material $m_i$, represented as tuples $(m_i, p_i, x_i)$. Its first instantiated benchmark, LitXAlloy, is a dense alloy-focused benchmark comprising 1426 total measurements from 19 alloy papers, with 101 target materials and 68 unique compositions (Chong et al., 8 Apr 2026).

1. Problem formulation and conceptual scope

LitXBench is defined around experiment extraction, not merely property extraction. This distinction is central. A composition-property tuple such as “alloy $X$ has hardness $Y$” is treated as insufficient because materials behavior depends on synthesis history, processing conditions, and characterization context. The framework therefore treats a material as identified by its processing lineage rather than by composition alone. Composition itself is modeled as one measurement inside $x_i$, not as the primary identity key (Chong et al., 8 Apr 2026).

This formulation responds to a recurring failure mode in materials information extraction: composition-centric grouping can conflate distinct processed samples that share nominal composition. The paper gives a concrete example from Zhang et al. (2019), where two materials had the same composition $m_i$0 but were sintered under different pressures. Under a composition-only representation, those samples collapse into one entity and property assignment becomes invalid. LitXBench instead evaluates whether systems recover the structured relation among synthesized materials, process histories, and measurement sets (Chong et al., 8 Apr 2026).

The extraction target is correspondingly broader than conventional materials-property IE. The framework evaluates extraction of structured experiment objects containing materials, raw materials, processing steps / process conditions, measurements, compositions, microstructure configurations, lattice parameters / structure, and dependencies between materials. This suggests a graph- or DAG-like ontology rather than a flat table of attributes. A plausible implication is that LitXBench is designed less as a mention-extraction benchmark than as a benchmark for reconstructing experimentally meaningful sample identity (Chong et al., 8 Apr 2026).

2. LitXAlloy benchmark composition

LitXAlloy, the first released benchmark within LitXBench, is built from 19 alloy papers: 18 open-access papers drawn from the MPEA dataset and 1 open-access Ni-based superalloy paper selected for its complex synthesis process and distinctive measurements. The superalloy paper was included specifically to stress process extraction and material disambiguation under more complex process structure (Chong et al., 8 Apr 2026).

The benchmark is dense rather than broad in document count. It contains 1426 total measurements / datapoints, 101 target materials, and 68 unique compositions. The paper emphasizes that this density matters: across the 18 overlapping papers, LitXAlloy averages 74.8 extracted measurements per paper versus 33.4 in MPEA, and contains 745 additional values in that overlap. This positions LitXAlloy as a compact but annotation-rich benchmark intended to stress structured reconstruction rather than document-scale coverage (Chong et al., 8 Apr 2026).

Several corpus statistics were selected to expose precisely the ambiguities that composition-centric methods mishandle. 8 of the 19 papers contain duplicate compositions, totaling 12 duplicate compositions overall. Across 6 papers, 26 materials are derived from other materials in the dataset. These properties make parent-child sample relations and process-defined material identity first-class benchmark concerns rather than corner cases (Chong et al., 8 Apr 2026).

The benchmark includes only experimental and experimentally derived measurements and explicitly excludes purely computational values such as Thermo-Calc predictions. Included values span mechanical, thermal, and physical properties, microstructural observations, lattice parameters, phase/configuration information, compositions, and experimentally derived quantities such as strain hardening exponent, fracture toughness, or Pugh ratio when derived from measured elastic constants (Chong et al., 8 Apr 2026).

3. Executable schema and object model

One of LitXBench’s most distinctive design decisions is to store entries as Python objects rather than in text serializations such as CSV or JSON. The stated rationale is threefold: auditability, editability, and validation. Code is easier to inspect and comment, easier to patch when benchmark errors are discovered, and directly compatible with compile-time and run-time validation of structural constraints (Chong et al., 8 Apr 2026).

The top-level object is an Experiment, containing raw_materials, synthesis_groups, descriptions, and output_materials. raw_materials maps names to RawMaterial objects with fields such as kind, description, and source. synthesis_groups define reusable process groups, while output_materials is a list of Material objects, each with a process, an optional name, and measurements. Materials can depend on previous materials, so the overall experiment may form a directed acyclic graph (DAG) rather than a flat list (Chong et al., 8 Apr 2026).

Process structure is encoded through named reusable groups and compact process strings such as "elements->creation" or "materialA->annealing[Temp=10]". Individual steps are represented by ProcessEvent objects with fields including kind, temperature, duration, description, source, and optionally inputs. Template variables are supported explicitly: for example, a group "annealing[Temp]" must contain the placeholder "[Temp]" in some event field, and the actual value is provided in the material’s process string. The paper characterizes this as a way to preserve strict temporal order in process history (Chong et al., 8 Apr 2026).

Measurements are represented primarily through a generic Measurement class with fields such as kind, value, unit, optional uncertainty, optional temperature, optional pressure, optional measurement_statistic, measurement_method, and source. The schema supports exact values, approximate values such as "~50", inequalities such as "\>50" or "\<50", and range-like groupings via lower/upper statistics. Specialized classes are used where necessary: CompMeasurement uses Pymatgen’s Composition, and GlobalLatticeParam / LatticeMeasurement use Pymatgen’s Lattice (Chong et al., 8 Apr 2026).

Microstructure is represented through nested Configuration objects, which may include a name, a struct such as FCC, BCC, or B2, tags such as Dendrite, Matrix, or Precipitate, a within field for nesting, and associated measurements. This allows phase-specific and nested descriptions such as dendritic composition by EDS, precipitate sizes, or intragranular precipitates within a matrix. Global notes and test conditions are stored via AlloyDescriptionGroup objects (Chong et al., 8 Apr 2026).

Canonicalization is handled through flattened enums, including MeasurementKind, RawMaterialKind, ProcessKind, CrysStruct, ConfigTag, and MeasurementMethod, plus a normalize(val, val_in_paper, source=None) wrapper to document mappings from paper terminology to canonical ontology. The validation layer enforces constraints such as no name collisions, no graph cycles, compositions summing to 100%, and alloy-specific rules such as melting events (ArcMelting, InductionMelting) must be followed by a casting event. The paper also reports a direct code-vs-JSON comparison in which code output is slightly better or roughly comparable: Claude Haiku 4.5 scored 0.63 F1 JSON vs 0.65 code, GPT-5 Mini 0.65 vs 0.67, and Gemini 3.1 Pro 0.76 vs 0.77 (Chong et al., 8 Apr 2026).

4. Benchmark creation and annotation procedure

All papers were transcribed using Mistral OCR 3. Figures were excluded from prompts and evaluation, although tables rendered as markdown remained visible to models. The benchmark annotations were created manually, and the paper reports that a single annotator performed extraction for all papers. To reduce missed values, extracted annotations were compared against MPEA as a safeguard, and LLMs were used as error-checking assistants rather than autonomous adjudicators (Chong et al., 8 Apr 2026).

Quality control was substantial. Approximately 1.1 billion tokens were spent using Claude Code with Opus 4.5/4.6 to catch benchmark errors, and additional Gemini 3.1 Pro scripts were used to identify errors and property-specific measurements. The paper states that all LLM-suggested corrections were heavily scrutinized by humans before incorporation. This emphasis on auditability is presented as a direct response to the likelihood that dense scientific IE benchmarks contain latent annotation errors even after repeated human review (Chong et al., 8 Apr 2026).

The annotation protocol includes several normalization conventions. Categorical fields are mapped to canonical identifiers; process names and property names can be wrapped in normalize(...) to document ontology alignment; compositions may be normalized with helper functions; and composition measurements prefer the highest-resolution analytical method available. Examples include mapping “Yield Strength” in a compression test to yield_strength_compression, mapping “Vacuum Arc Melting” to ProcessKind.ArcMelting, and normalizing a paper’s “plastic strain” to compressive fracture strain (Chong et al., 8 Apr 2026).

Inter-annotator agreement is not reported. Instead, the benchmark relies on a single annotator plus intensive human/LLM review. The paper also stresses residual ambiguity: papers omit key details, terminology is inconsistent, some textual descriptions are misleading, and some values remain contested. One appendix example notes that in Chen et al. (2014), table values labeled “fracture strain” appear from the stress–strain curve to actually be “strain at the ultimate point,” but because LitXBench is text-only, the ground truth remains aligned with textual evidence rather than figure-derived reinterpretation. This suggests that the benchmark privileges reproducible text-grounded extraction over post hoc scientific correction (Chong et al., 8 Apr 2026).

5. Evaluation protocol and empirical results

Models are given OCR-transcribed paper text, excluding figures, and asked to output experiments in the benchmark schema. The paper reports use of Pydantic AI and equivalent tooling. Confidence intervals are based on three runs using a 95% Student’s t-distribution confidence interval. Evaluation begins by aligning predicted and gold materials via the Hungarian algorithm to obtain a maximum bipartite match between extracted materials $m_i$1 and target materials $m_i$2 (Chong et al., 8 Apr 2026).

The overall LitXBench score is a weighted sum of four component F1 scores: Measurements with weight 0.50, Process with weight 0.20, Material with weight 0.15, and Configuration with weight 0.15. Measurement scoring compares extracted and target measurements regardless of material assignment, checking kind, value, unit, and weighted auxiliary fields such as uncertainty quantifiers, temperature, and pressure. Process scoring evaluates ordered process extraction using Levenshtein distance to define Hungarian cost. Configuration scoring uses Hungarian matching over configuration measurements and then checks parent equivalence over the configuration hierarchy (Chong et al., 8 Apr 2026).

On full experiment extraction, frontier LLMs substantially outperform prior multi-turn extraction pipelines. The paper’s headline comparison is an improvement of up to 0.37 F1 over KnowMat2 (Chong et al., 8 Apr 2026).

Model	Overall F1
KnowMat2 (GPT-5.2 High)	0.43 ± 0.29
Claude Haiku 4.5	0.65 ± 0.01
GPT-5 Mini Medium	0.67 ± 0.04
Gemini 3 Flash Preview	0.74 ± 0.05
Claude Opus 4.6	0.72 ± 0.04
GPT-5.2 High	0.72 ± 0.02
Gemini 3.1 Pro Preview	0.77 ± 0.03
Claude Code (Opus 4.6)	0.78 ± 0.00
Codex	0.72 ± 0.01
Gemini CLI (3.1 Pro Preview)	0.80 ± 0.04

The strongest reported model, Gemini CLI (3.1 Pro Preview), achieves category scores of 0.74 for Measurements, 0.84 for Process, 0.98 for Material, and 0.68 for Config. This pattern indicates that identifying the set of materials is comparatively easy, process extraction is strong, measurement extraction is harder, and microstructure/configuration extraction remains the weakest structural component (Chong et al., 8 Apr 2026).

Several auxiliary experiments sharpen the task diagnosis. Composition extraction is much easier than full experiment extraction, with best results reaching 0.99 using code output with helpers. Single-property extraction is also comparatively easy; for Gemini 3.1 Pro Preview, F1 reaches 0.91 for Ultimate Strength (T), 0.91 for Ultimate Strength (C), 0.90 for Fracture Strain (T), 0.88 for Fracture Strain (C), and 0.95 for Vickers Hardness. When models are given disordered ground-truth process conditions and measurements and asked only to organize them into materials, performance rises substantially, to 0.92 for Gemini 3.1 Pro, 0.87 for Claude Opus 4.6, and 0.85 for GPT-5.2 High. This suggests that a large portion of difficulty lies in end-to-end document extraction rather than only in post hoc clustering of already extracted values (Chong et al., 8 Apr 2026).

6. Interpretation, significance, and limitations

The paper’s central interpretive claim is that older extraction pipelines underperform because they associate measurements with compositions rather than with the processing steps that define a material. The benchmark’s error analysis supports this. In Haas et al. (2019), LeMat-Synth assigned two annealing steps, 1220°C and 950°C, to the same material when they belonged to different materials. In Jia et al. (2019), KnowMat2 extracted compressive yield strength correctly as “Yield Strength,” but a post-processing LLM canonicalized it incorrectly to tensile yield strength because contextual information had been lost. In Shi et al. (2019), even Gemini 3.1 Pro missed indirect microstructure evidence expressed through references to “the other two DPHL HEAs,” which required paper-level identity resolution across samples (Chong et al., 8 Apr 2026).

These results give LitXBench broader significance within scientific IE. It contributes a formalized experiment extraction task, a dense benchmark with structured ground truth, a material-centric representation of sample identity, integration of canonicalization into extraction rather than post-processing, and a code-based benchmark design that supports validation and audit trails. A plausible implication is that LitXBench moves materials IE from loosely defined tuple extraction toward executable structured reconstruction of experimentally meaningful sample graphs (Chong et al., 8 Apr 2026).

The framework also has clear limitations. It is text-only: figures and image-only tables are excluded, even though figures may reveal that textual labels are incomplete or misleading. Annotation errors may remain despite extensive review. Python numeric literals do not preserve significant figures. The released schema is alloy-specific, so extension to other material classes would require new canonical values, new structured measurement classes, and new class-specific validation rules. LitXAlloy is also restricted to 19 alloy papers, so generalization to polymers, ceramics, batteries, or catalysts is not established in the current benchmark (Chong et al., 8 Apr 2026).

LitXBench should also be distinguished from LitBench, a graph-centric benchmarking tool for literature tasks built around domain-specific literature subgraphs and citation graphs rather than experiment extraction from materials papers (Varvarigos et al., 10 Feb 2026). The two names are adjacent, but their targets differ fundamentally: LitXBench evaluates extraction of process-defined experimental structure from scientific articles, whereas LitBench curates and benchmarks literature-task datasets over paper-citation graphs.