MOSES Benchmark for Molecular Generative Models

• MOSES benchmark is a curated platform standardizing datasets, evaluation metrics, and reference models for assessing molecular generative algorithms.
• It details a rigorous preprocessing pipeline on the ZINC Clean Leads collection, enabling both in-distribution and scaffold-novel evaluations.
• The framework supports reproducible comparisons across neural and non-neural models by quantifying trade-offs in validity, uniqueness, and novelty.

The acronym MOSES denotes separate benchmarks in three distinct research domains: molecular generative modeling, video object segmentation, and overlapping community detection in networks. In the context of molecular design, MOSES (Molecular Sets) is a rigorously curated and standardized benchmarking platform for the training and evaluation of molecular generative models, central to measuring progress in computational chemistry and AI-driven drug discovery (Polykovskiy et al., 2018, Singh, 19 May 2025). The MOSES benchmark comprises a filtered dataset, a comprehensive suite of evaluation metrics, and a set of reference models, together providing a transparent, reproducible framework for quantitative comparison of generative algorithms.

1. Dataset Construction and Preprocessing

MOSES is constructed from the ZINC Clean Leads collection, initially comprising approximately 4.6 million drug-like molecules. The preprocessing pipeline imposes physicochemical and medicinal chemistry constraints:

• Molecular weight: 250–350 Da
• Rotatable bonds: ≤7
• XlogP: ≤3.5
• Allowed atom types: {C, N, O, S, F, Cl, Br, H}
• Ring sizes: ≤8 members
• Medicinal chemistry and PAINS filters exclude reactive, unstable, or promiscuous substructures

Following filtering, the finalized corpus contains 1,936,962 molecules (canonical SMILES), with 448,854 unique Bemis–Murcko scaffolds and 58,315 unique BRICS fragments (Polykovskiy et al., 2018, Singh, 19 May 2025). The platform provides standardized splits:

• Training set: 1,584,664 molecules (~82.6%)
• Test set: 176,075 molecules (~9.1%)
• Scaffold-split test set: 176,226 molecules (~9.1%), with scaffolds absent from train/test

These splits enable unbiased assessment of both in-distribution and scaffold-novel (out-of-training-scaffold) generation, which is critical for evaluating generalization in molecular design.

2. Evaluation Metrics

MOSES establishes an explicit suite of chemically meaningful metrics to assess generative model outputs. Unless otherwise noted, these operate over generated sets $G$ and reference sets $R$ (usually the test set). All chemical validity and feature computations rely on RDKit.

Metric	Mathematical Definition / Description	Scope
Validity	$$\text{Valid}(G) = \frac{\# \text{valid molecules}}{|G|}$$	Fraction chemically valid (RDKit parse)
Uniqueness@K	$$\text{Unique@}K(G) = \frac{\# \text{unique molecules among first K}}{K}$$	Early mode collapse detection
Novelty	$$\text{Novelty}(G) = \frac{|\{g \in G: g \notin D_{\text{train}}\}|}{|G|}$$	Measures memorization
Filters	Fraction of $G$ passing MCF, PAINS, ring-size, atom-type filters	Drug-likeness compliance
Frag/ScaffSim	Cosine similarity between fragment/scaffold frequency vectors	Structural fidelity
SNN	$$\frac{1}{|G|} \sum_{x \in G} \max_{y \in R} T(x, y)$$ (Tanimoto, R=Test)	Distribution matching
IntDiv	$1 - \frac{1}{|G|^2} \sum_{x,x'} T(x, x')$	Intrinsic diversity
FCD	Fréchet distance between ChemNet embeddings:	Overall distributional distance
	$$\| \mu_g - \mu_r \|^2 + \operatorname{Tr}[\Sigma_g+\Sigma_r-2(\Sigma_g\Sigma_r)^{1/2}]$$
Property W₁	Wasserstein-1 distance for MW, logP, SA, QED histograms between $G$, test	Property distributional alignment

These metrics quantify various desiderata: chemical soundness, diversity, novelty, overfitting, fragment/scaffold preservation, and global feature distribution matching (Polykovskiy et al., 2018, Singh, 19 May 2025).

3. Baseline Models and Architectures

MOSES provides a spectrum of baseline models, encompassing both string- and graph-based, neural and non-neural generative strategies. Configurations and reference results are fixed, supporting direct, reproducible comparison:

• Non-neural: HMM (1st-order, SMILES), character-level N-gram (typically N=10–11), BRICS fragment-based combinatorial generator.
• String-based RNNs: CharRNN (GRU ×3, 512/768 units, teacher forcing on SMILES).
• Autoencoder-based: VAE (biGRU encoder, GRU decoder, 128D latent), AAE (same as VAE + adversarial discriminator, latent regularization).
• Hierarchical graph-based: Junction Tree VAE (JTN-VAE)—encodes/binds junction-tree skeleton first, then molecular graph, directly enforcing chemical validity.
• LatentGAN: An autoencoder + WGAN-GP pipeline, where the GAN is trained on autoencoder latent vectors; samples decoded to SMILES.

Performance exhibits systematic trade-offs along the axes defined by the MOSES metrics:

• Combinatorial and JTN-VAE guarantee 100% chemical validity; CharRNN and VAE approach ~97–98%, HMM substantially lower (≈8%).
• Uniqueness is high (>99.7%) for deep models, NGram also strong at moderate sample counts.
• Novelty is maximal for HMM and NGram (≈99.9%), but these suffer severely on distributional matching metrics (FCD > 5.5).
• VAE is highest on scaffold similarity (ScaffSim = 0.939), but lower on novelty (≈0.695), suggesting closer adherence to the training set (Polykovskiy et al., 2018, Singh, 19 May 2025).

4. Practical Usage and Best Practices

The MOSES platform is intended for methodological development, benchmarking, and cross-lab reproducibility. The standard workflow is:

1. Install the molsets toolchain.
2. Load the split datasets via API or direct download.
3. Train generative model(s) on the prescribed training split.
4. Generate at least 30,000 molecules; apply RDKit-based validity filters.
5. Compute all metrics (using provided scripts).
6. Repeat sampling/training with ≥3 random seeds, reporting mean ± standard deviation for stochastic robustness.
7. Compare results to the provided baseline tables to contextualize observed model behavior.

Crucial recommendations include always evaluating models on both the random test and scaffold-held-out splits, using FCD as a primary selection metric but diagnosing generation pathology using the broader interpretable metric suite, and visualizing property-histogram alignment (Polykovskiy et al., 2018, Singh, 19 May 2025).

5. Quantitative Baseline Results and Trade-off Analysis

Empirical benchmarking (means ± standard deviations reported over seeds) exposes several robust findings:

• CharRNN achieves the best overall FCD (0.073) and fragment/scaffold similarity (Frag = 1.00, Scaff = 0.924).
• VAE leads on SNN (0.626) and scaffold similarity but at the cost of lower novelty (0.695).
• Combinatorial and JTN-VAE enforce validity >99.9% but exhibit reduced exploration of novel chemical space.
• LatentGAN and adversarial models gain novelty (0.95) at the expense of FCD and stability.
• HMM, while maximally novel, is distributionally inferior (FCD ≫ 1.0, SNN ≪ 0.5).

No single model uniformly dominates. Architectures must be selected based on specific optimization priorities: distribution learning, novelty/diversity, or property preservation (Polykovskiy et al., 2018, Singh, 19 May 2025).

6. Limitations and Prospective Extensions

Several limitations are acknowledged:

• All analyses are SMILES/2D graph-based; 3D conformation, stereochemistry, and reaction information are not addressed.
• No direct assessment of downstream biological functionality (binding affinity, toxicity, ADMET, or synthetic tractability beyond heuristic SA).
• Static benchmarking; absent are iterative, active-learning-based or real-world feedback loops.
• Property metrics are chemically and contextually heuristic (QED, SA, PAINS).

Future directions suggested by the benchmark authors include incorporation of reaction-aware and 3D structure-based models, expansion of metrics to biological endpoints and retrosynthetic feasibility, adoption of dynamic (active/closed-loop) evaluation, and leveraging recent advances in transformer and diffusion-based generative models for molecular design (Singh, 19 May 2025).

7. Impact and Significance in Computational Chemistry

MOSES has established itself as a reference point for progress in deep molecular generative modeling by providing the first large-scale, standardized, and openly accessible benchmarking suite specifically tailored for distribution learning within drug-like chemical space. It enables fair, reproducible comparison of disparate generative paradigms, facilitates objective diagnosis of strengths and failure modes, and continues to drive improvements in both algorithmic innovation and evaluation methodology. Its influence is observed in its widespread adoption in the generative chemistry research community and its explicit integration into protocol sections of numerous subsequent works (Polykovskiy et al., 2018, Singh, 19 May 2025).