SynRXN: An Open Benchmark and Curated Dataset for Computational Reaction Modeling

Abstract: We present SynRXN, a unified benchmarking framework and open-data resource for computer-aided synthesis planning (CASP). SynRXN decomposes end-to-end synthesis planning into five task families, covering reaction rebalancing, atom-to-atom mapping, reaction classification, reaction property prediction, and synthesis route design. Curated, provenance-tracked reaction corpora are assembled from heterogeneous public sources into a harmonized representation and packaged as versioned datasets for each task family, with explicit source metadata, licence tags, and machine-readable manifests that record checksums, and row counts. For every task, SynRXN provides transparent splitting functions that generate leakage-aware train, validation, and test partitions, together with standardized evaluation workflows and metric suites tailored to classification, regression, and structured prediction settings. For sensitive benchmarking, we combine public training and validation data with held-out gold-standard test sets, and contamination-prone tasks such as reaction rebalancing and atom-to-atom mapping are distributed only as evaluation sets and are explicitly not intended for model training. Scripted build recipes enable bitwise-reproducible regeneration of all corpora across machines and over time, and the entire resource is released under permissive open licences to support reuse and extension. By removing dataset heterogeneity and packaging transparent, reusable evaluation scaffolding, SynRXN enables fair longitudinal comparison of CASP methods, supports rigorous ablations and stress tests along the full reaction-informatics pipeline, and lowers the barrier for practitioners who seek robust and comparable performance estimates for real-world synthesis planning workloads.

• The paper introduces a unified benchmark that dissects CASP into five task families to standardize computational reaction modeling evaluation.
• It employs a curated, provenance-aware dataset with deterministic splitting and rigorous chemical sanity checks to ensure reproducibility.
• Baseline results show high accuracies across reaction rebalancing, atom-to-atom mapping, classification, property prediction, and synthesis planning tasks.

SynRXN: A Unified, Provenance-Aware Benchmark for Computational Reaction Modeling

Introduction

The proliferation of computer-aided synthesis planning (CASP) methods has been driven by extensive progress in molecular machine learning and the availability of curated reaction datasets. However, benchmarking CASP systems remains a challenge due to fragmented datasets, inconsistent preprocessing pipelines, and opaque evaluation protocols. "SynRXN: An Open Benchmark and Curated Dataset for Computational Reaction Modeling" (2601.01943) introduces SynRXN, a comprehensive framework to address these deficiencies. SynRXN dissects the CASP workflow into five well-defined task families: reaction rebalancing, atom-to-atom mapping, reaction classification, reaction property prediction, and synthesis planning, supplying curated, provenance-tracked corpora, transparent splitters, and rigorous evaluation scaffolding.

Figure 1: SynRXN’s architecture encompasses modular task families, each supplied with dedicated datasets, deterministic splitting routines, and tailored metrics.

Motivation and Scope

CASP frameworks integrate disparate pipeline components: extraction/curation, chemical sanity, rebalancing of reaction stoichiometry, atom-level mapping, classification, property prediction, and route planning. Each stage introduces unique error modes and evaluation demands, but the absence of harmonized benchmarks precludes fair longitudinal comparisons, systematic ablations, and robust performance estimates. SynRXN directly confronts dataset heterogeneity by enforcing deterministic build recipes and leakage-aware splits, while serving as a FAIR (Findable, Accessible, Interoperable, Reusable) [wilkinson2016fair] platform for reproducible research and rigorous benchmarking.

Corpus Assembly and Task Modularization

SynRXN is constructed from public reaction corpora, mainly patent-mined datasets such as USPTO [lowe2012], supplemented by community benchmarks. Data are versioned, metadata-rich, and curated according to explicit license terms. The framework includes automated standardization and normalization via the SynKit pipeline [phan2025synkit], ensuring uniformity in charge, aromaticity, canonicalization, and chemical-sanity filtering. All corpora record record-level provenance for transparent traceability.

Task Families

• Reaction Rebalancing: Restoration of mass and charge balance in incomplete records via perturbation sets capturing modes of violation (MNC, MOS, MBS, Complex), and validation via SynRBL [phan2024].
• Atom-to-Atom Mapping: Stratified benchmarks covering small-molecule/transformation datasets (Golden, Jaworski, USPTO_3K) and biochemical transformations (Recon3D, EColi), evaluated by ITS graph isomorphism and canonical SMILES comparison.
• Reaction Classification: Hierarchical label taxonomies at varying granularities (USPTO_TPL, Schneider, SynTemp, ECREACT), supported by stratified splits to maintain label density and ablation of mapping sensitivity.
• Reaction Property Prediction: Quantification of yields, activation barriers, and enthalpies using DRFP [probst2022] and RXNFP [schwaller2021mapping] embeddings, fed to regression models such as RandomForest or sequence-based architectures.
• Synthesis Planning: Route design benchmarks based on USPTO_50k, USPTO_MIT, and USPTO_500, equipped with evaluation tools for route-level coverage, precision, and chemical consistency metrics.

Technical Validation and Baseline Benchmarks

SynRXN’s validation pipeline is strictly automated, rejecting records that fail chemical sanity checks to preclude manual bias. Canonicalization, duplicate removal (structural and SMILES-level), and provenance preservation are core to the rebalancing and mapping splits. For rebalancing, the curated test set yields $100\%$ ground-truth success and exact-match accuracy due to manual verification. Atom-mapping benchmarks compare ensemble and individual tools (RXNMapper, Graphormer, LocalMapper, RDTool), reporting accuracies up to $97\%$ for curated sets, with significant domain variance (biochemical sets underperform relative to small organic chemistry benchmarks).

Figure 2: Automated, deterministic technical validation flow ensures stringent chemical-sanity and duplication controls for every SynRXN dataset.

Classification models (RandomForest with DRFP/RXNFP features) achieve strong weighted F1 ($>0.95$) and MCC, with stratified cross-validation confirming statistical robustness. Regression tasks display learnable property distributions (single-digit to low-tens MAE for activation energies and barriers), indicating high label integrity without significant outlier contamination. Synthesis planning metrics rigorously define coverage ($\mathcal{C}$), recognition rate (RR), top-K accuracy ($\tau_K$), max-fragment overlap, and round-trip success ($\rho$), all dependent on deterministic canonicalization and ITS-graph equality.

Data Organization and Accessibility

SynRXN’s releases are provided on Zenodo and GitHub, with versioned manifest.json files encoding checksums, row/column counts, licensing, and file paths. All datasets follow a standardized tabular format, partitioned into task-specific subdirectories (rbl, aam, class, prop, synthesis), with explicit train/validation/test splits and metadata columns (IDs, labels, property flags, mapping completeness).

Figure 3: SynRXN’s manifest-driven data organization enables modular access and exhaustive metrics coverage across five benchmark domains.

The loader supports reproducible access via Zenodo, GitHub tag, or commit. Deterministic repeated K-fold splitting (with seed control and stratification) is implemented for transparent model comparison and significance testing. End-to-end reproducibility is facilitated via CLI data rebuild workflows.

Implications and Future Prospects

SynRXN establishes the first domain-wide, verifiable scaffold for comparative CASP method development. By codifying data hygiene, transparent splitting, and leak-aware train/test separation, SynRXN removes major sources of benchmarking bias and ambiguity, enabling the field to converge on reproducible evaluation standards. High-fidelity benchmarks for mapping, classification, and property prediction will facilitate calibration and stress testing of future graph-based, transformer, and ensemble models. The modularity of SynRXN is directly extensible to novel reaction tasks, personalized synthesis planning, and advanced mechanistic analysis (e.g., transition-state theory, hybrid quantum models). The framework’s provenance-tracked builds and archived releases lower barriers for reproducibility-centric publications and collaborative method development, driving longitudinal progress in reaction informatics beyond current single-task-centric paradigms.

Conclusion

SynRXN delivers a unified, metadata-rich, and reproducible benchmark ecosystem for computational reaction modeling spanning all critical CASP components. Its leakage-aware task families, deterministic splitting, and rigorous evaluation protocols ensure fair head-to-head comparison and technical integrity for model development. SynRXN sets a durable foundation for future advances in reaction informatics, systematic benchmarking, and robust real-world synthesis planning workflows.