NovoMolGen: Transformer-Based Molecular Generation

Updated 9 July 2026

NovoMolGen is a framework for de novo molecular generation that integrates transformer-based foundation models pretrained on billions of molecules with diverse chemical representations.
It employs techniques such as sequence translation, fragment assembly, graph modeling, and 3D co-design to optimize molecular properties under explicit validity and experimental constraints.
Evaluation metrics including validity, uniqueness, and docking performance highlight both the methodological advances and challenges in aligning pretraining outcomes with practical lead optimization.

NovoMolGen denotes both a broad research program in de novo molecular generation and, more specifically, a family of transformer-based foundation models pretrained on 1.5 billion ZINC-22 molecules for molecule design (Chitsaz et al., 19 Aug 2025). In the broader sense, it covers algorithmic systems that explore a chemical space commonly described as lying between $10^{23}$ and $10^{60}$ synthesizable small molecules, with the aim of generating valid, novel structures that satisfy predefined property, scaffold, biological, or experimental constraints (Chitsaz et al., 19 Aug 2025). Recent work grouped under this label spans conditional sequence translation for lead optimization, fragment- and motif-based assembly, graph and energy-based generation, equivariant 2D–3D co-design, language-model prompting, retrieval-augmented reasoning, and multimodal conditioning on spectra, gene-expression profiles, protein pockets, and 3D conformations (Shin et al., 2020, Geng et al., 2023, Formont et al., 18 Mar 2026).

1. Scope and research landscape

Within the current literature, NovoMolGen is not a single algorithmic paradigm but a heterogeneous design space. Some systems treat molecule generation as direct sequence modeling over SMILES or SELFIES; others operate on graphs, motifs, BRICS fragments, Murcko scaffolds, or continuous 3D coordinates. Conditioning can be absent, scalar, symbolic, structural, or multimodal. Representative objectives include unconstrained distribution learning, single- and multi-property optimization, scaffold preservation, target-conditioned docking optimization, phenotype-conditioned design, and de novo reconstruction from MS/MS spectra (Chitsaz et al., 19 Aug 2025, Hataya et al., 2021, Wang et al., 3 Jan 2025).

A useful way to situate the field is by the interface through which constraints enter generation. Controlled Molecule Generator frames multi-property optimization as supervised sequence translation from a source molecule $X$ to a target molecule $Y$ under similarity and property constraints (Shin et al., 2020). Chemistry42, developed within Insilico Medicine’s Pharma.ai suite, exposes ligand-based and structure-based workflows through a cloud-native platform that combines more than 30 generative models with multi-objective scoring and reinforcement learning (Ivanenkov et al., 2021). At the other end of the spectrum, ChemGE performs population-based grammatical evolution over SMILES and emphasizes concurrency and diversity rather than differentiable property conditioning (Yoshikawa et al., 2018).

Family	Representative systems	Primary constraint interface
Sequence and LLMs	CMG, LLamol, ChatMol, NovoMolGen	Property vectors, prompts, numerical tokens
Fragment and motif systems	MiCaM, M $^4$ olGen, DGMM	Mined motifs, BRICS edits, mol-gene codes
Graph and 3D generators	GEM, MolCode, NExT-Mol, Megalodon	Substructures, topology–geometry co-design, diffusion or flows
Scientific-conditioning systems	Gx2Mol, MADGEN, Mos-Gen	Gene expression, MS/MS spectra, frozen 3D encoders

This taxonomy suggests that NovoMolGen has become less a single architecture than a unifying problem setting: the controlled exploration of chemical space under explicit validity, property, and task constraints.

2. Molecular representations and conditioning interfaces

The representational layer is a central organizing principle of NovoMolGen. The 2025 NovoMolGen foundation-model study systematically compared SMILES, SELFIES, DeepSMILES, and SAFE, together with atomwise tokenization and BPE, and reported that SMILES+BPE is a robust default, with best FCD around $0.0380$ and strong performance across unconstrained generation, PMO, and docking tasks (Chitsaz et al., 19 Aug 2025). SELFIES remains attractive because it guarantees validity by construction and is therefore used in systems such as NExT-Mol, which first samples a 1D SELFIES string and only then predicts 3D conformers (Liu et al., 18 Feb 2025).

Other systems modify representations to better match the inductive biases of their generators. ChatMol uses a simplified SELFIES representation with brackets removed and tokens space-delimited, motivated by tokenizer efficiency and by the desire to turn property-constrained molecule design into a promptable LLM task (Fan et al., 27 Feb 2025). LLamol uses SMILES with DeepChem’s BERT tokenizer and prepends numerical condition embeddings and optional token-sequence fragments as a prefix, allowing a single decoder-only transformer to support up to four simultaneous conditions (Dobberstein et al., 2023). NovoMol, by contrast, remains entirely character-level, padding SMILES to length 100 and training an encoder–decoder LSTM on one-hot vectors over a 45-character vocabulary (Rao, 2023).

Fragment and scaffold representations define a second major branch. MiCaM mines a connection-aware motif vocabulary by iteratively merging frequent adjacent subgraphs and then turning broken external bonds into explicit “*” attachment sites with preserved bond-type information (Geng et al., 2023). M $^4$ olGen decomposes molecules with BRICS, constructs a large neighbor dataset of 1,171,193 single-edit pairs, and treats generation as fragment-level add, remove, or replace operations toward precise numeric targets (Li et al., 15 Jan 2026). MADGEN uses Murcko scaffolds as an intermediate object: a contrastive retriever maps spectra to candidate scaffolds, and a spectrum-conditioned graph generator completes the undecided edges (Wang et al., 3 Jan 2025).

A third branch centers on explicit 3D structure. Mos-Gen feeds 3D conformations through a frozen Uni-Mol encoder and conditions a VAE decoder on those features, with a dedicated [SS] token to bias generation toward disulfide-containing allicin derivatives (Wang et al., 1 Jun 2026). MolCode and Megalodon jointly model topology and coordinates rather than treating 3D structure as a post hoc annotation (Zhang et al., 2023, Reidenbach et al., 23 May 2025). NExT-Mol separates the roles: a 960M-parameter SELFIES LM handles 2D-valid generation, while a Diffusion Molecular Transformer predicts conformers (Liu et al., 18 Feb 2025).

3. Learning objectives, control mechanisms, and decoding

NovoMolGen systems employ a wide range of training criteria, but several objective families recur. Maximum-likelihood sequence modeling remains common, both in SMILES LLMs and in sequence-to-sequence lead-optimization systems. CMG optimizes a Transformer encoder–decoder with a conditional log-likelihood objective and augments it with two frozen auxiliary networks: PropNet, a BiLSTM+MLP property regressor trained on 257,565 molecules with validation MSE approximately $0.0855$, and SimNet, a shared-BiLSTM similarity classifier trained on about 997,773 molecule pairs with validation accuracy approximately $0.976$ (Shin et al., 2020). The resulting joint loss, $\mathcal{L}_{\mathrm{CMG}}=\mathcal{L}_T+\lambda_p\mathcal{L}_P+\lambda_s\mathcal{L}_S$ , couples token prediction to property satisfaction and source-target similarity (Shin et al., 2020).

Variational objectives are prominent when latent interpolation or multimodal conditioning is required. Mos-Gen uses a standard ELBO with a multi-slot latent bottleneck of $10^{60}$ 0 slots, while motif awareness is introduced through token engineering and disulfide-enriched fine-tuning rather than an explicit motif regularizer (Wang et al., 1 Jun 2026). Gx2Mol uses a VAE over gene-expression profiles to extract a 64-dimensional phenotype vector that is concatenated to each LSTM input step during SMILES generation, enabling generation conditioned on target perturbation or disease-reversal signatures (Li et al., 2024). DGMM uses a teacher VAE and a distilled discrete VAE to create a quantized “mol-gene” code on which mutation and crossover are performed directly (Fang et al., 2024).

Diffusion, flow matching, and exact likelihood also play major roles. NExT-Mol applies a DDPM-style coordinate denoising objective on 3D conformers after 1D generation (Liu et al., 18 Feb 2025). Megalodon couples continuous denoising for coordinates with discrete denoising for atom types, bond types, and charges, with independent time variables for continuous and discrete tracks but joint training at every layer (Reidenbach et al., 23 May 2025). MolCode uses normalizing flows over dequantized atom types, bond types, and local spherical coordinates, yielding exact likelihood training for a joint 2D–3D autoregressive model (Zhang et al., 2023).

Preference-based and reinforcement objectives are increasingly used when exact numeric control is difficult. ChatMol performs supervised fine-tuning followed by ranking-based sequence calibration, scoring sampled candidates with RDKit or AutoDock-GPU and optimizing a combined objective $10^{60}$ 1, with $10^{60}$ 2 and $10^{60}$ 3 (Fan et al., 27 Feb 2025). A separate line of work replaces RL with Direct Preference Optimization, pairing high- and low-scoring molecules and reporting that all 20 GuacaMol tasks can be completed in 60 hours on a single A100 GPU, compared with about 400 hours for MolRL-MGPT (Hou, 2 Apr 2025). MolRGen and M $10^{60}$ 4olGen instead adopt GRPO-style policy optimization, using on-the-fly verifiable rewards and group-relative baselines rather than pairwise labels (Formont et al., 18 Mar 2026, Li et al., 15 Jan 2026).

Decoding is correspondingly varied. CMG uses modified beam re-ranking with PropNet and SimNet scores at sequence completion (Shin et al., 2020). LLamol uses temperature sampling at $10^{60}$ 5 and relies on Stochastic Context Learning to remain robust to missing conditions (Dobberstein et al., 2023). ChatMol augments numerical tokens with sinusoidal numeric embeddings and reports that Pearson correlation between instructed values and generated properties increased by up to $10^{60}$ 6 (Fan et al., 27 Feb 2025). These systems collectively indicate that control in NovoMolGen is increasingly achieved through a combination of conditioning interface, auxiliary evaluators, and structured decoding rather than through the generator alone.

4. Topology–geometry co-design and 3D molecular generation

A major development in NovoMolGen has been the move from 2D-valid string or graph generation toward explicit 3D structure generation. NExT-Mol embodies a staged design: a 960M-parameter SELFIES LM, pretrained on 1.8 billion ZINC-15 molecules and about 90 billion SELFIES tokens, samples a 100%-valid molecule; a diffusion model then predicts its conformers (Liu et al., 18 Feb 2025). On GEOM-DRUGS, the system reports a 26% relative improvement in 3D FCD for de novo 3D generation and a 13% average relative gain for conditional 3D generation on QM9-2014 (Liu et al., 18 Feb 2025). The transfer of pretrained 1D token embeddings into the 3D diffusion model also improved conformer prediction on unseen scaffolds (Liu et al., 18 Feb 2025).

MolCode addresses the same problem by modeling 2D topology and 3D geometry within a single equivariant flow-based factorization. It represents a molecule as $10^{60}$ 7, uses SphereNet or EGNN encoders plus bond-aware global attention, and generates atoms, bonds, and local spherical coordinates autoregressively (Zhang et al., 2023). On QM9 random generation, MolCode reports 99.95% validity and 98.75% uniqueness; in structure-based drug design, it reports a 61.8% high-affinity ratio, defined as the fraction of protein pockets for which at least one generated ligand outperforms the reference ligand in Vina score (Zhang et al., 2023).

Megalodon pushes co-design further by jointly denoising continuous coordinates and discrete atom, bond, and charge variables with a fused transformer trunk and a lightweight equivariant coordinate-update layer (Reidenbach et al., 23 May 2025). On GEOM Drugs, diffusion Megalodon reports Atom Stability $10^{60}$ 8, Molecule Stability $10^{60}$ 9, and Connected Validity $X$ 0, while its energetics benchmark shows median $X$ 1 kcal/mol, compared with $X$ 2 for EQGAT-diff and $X$ 3 for SemlaFlow (Reidenbach et al., 23 May 2025). The larger 40.6M-parameter model generates up to 49x more valid large molecules than EQGAT-diff in the $X$ 4-atom regime (Reidenbach et al., 23 May 2025).

Mos-Gen illustrates a different use of 3D structure: not generic conformer realism, but motif-aware generation in a low-label application domain. A frozen Uni-Mol encoder pretrained on about 18 million unlabeled molecules conditions a VAE with a GRU decoder and an explicit [SS] token; a separate Uni-Mol classification head, fine-tuned on about 200 labeled compounds, provides the predictive screen (Wang et al., 1 Jun 2026). In prospective validation, 14 compounds were synthesized, 8 had no prior literature record, 7 of 9 predicted positives were active, and none of the 5 predicted negatives showed activity, corresponding to a hit rate of approximately 78% among predicted positives (Wang et al., 1 Jun 2026).

System	Core mechanism	Reported result
NExT-Mol	SELFIES LM plus 3D diffusion	26% relative improvement in 3D FCD on GEOM-DRUGS
MolCode	Equivariant flow-based 2D–3D co-design	99.95% validity; 98.75% uniqueness
Megalodon	Joint continuous/discrete denoising co-design	Up to 49x more valid large molecules
Mos-Gen	Frozen Uni-Mol plus motif-aware VAE	78% hit rate among predicted positives

These systems indicate a shift in NovoMolGen from chemistry-valid generation toward geometry-valid and energetically plausible generation, with explicit 3D structure now treated as a first-class object rather than a downstream annotation.

5. Benchmarks, metrics, and reported performance regimes

Evaluation in NovoMolGen is fragmented because the field spans distribution learning, constrained optimization, docking, and multimodal reconstruction. Nevertheless, a relatively stable metric vocabulary has emerged: validity, uniqueness, novelty, similarity, diversity, FCD, scaffold or fragment similarity, conformer metrics such as COV and AMR, docking score, success rate under hard constraints, and, in reasoning-oriented settings, top- $X$ 5 reward and diversity-aware top- $X$ 6 (Chitsaz et al., 19 Aug 2025, Formont et al., 18 Mar 2026).

Controlled multi-property optimization is exemplified by CMG. Under the multi-objective setting $X$ 7, $X$ 8, $X$ 9, and $Y$ 0, CMG reports a success rate of 6.98% on all 2,365 samples, versus 3.56% for VJTNN; in the single-objective PlogP benchmark, CMG reports $Y$ 1 with diversity $Y$ 2 (Shin et al., 2020). ChatMol reports 94.5% success and diversity $Y$ 3 for a single-property logP targeting task, while in the multi-objective ESR1 binding task it reports a best $Y$ 4 of $Y$ 5, QED $Y$ 6, SA $Y$ 7, and the highest overall performance, surpassing previous methods by 4.76% (Fan et al., 27 Feb 2025).

The 2025 NovoMolGen foundation models establish a distinct benchmark regime based on large-scale pretraining followed by PMO and docking evaluation. The best SMILES+BPE configuration reports a PMO total score of 16.70 across 23 tasks, compared with 16.30 for f-RAG and 14.01 for REINVENT, while docking on parp1 yields a best docking score of $Y$ 8 and a hit rate of 90.24% (Chitsaz et al., 19 Aug 2025). A key finding is that pretraining loss and FCD correlate weakly with downstream PMO quality, with reported correlation $Y$ 9, $^4$ 0 (Chitsaz et al., 19 Aug 2025).

Reasoning-based evaluation is codified most explicitly in MolRGen, which defines both raw top- $^4$ 1 and a diversity-aware top- $^4$ 2 score under fingerprint-based similarity constraints (Formont et al., 18 Mar 2026). In that setting, the RL-fine-tuned RL-Mistral model reaches top-1 scores of 0.64, 0.67, and 0.68 for $^4$ 3 rollouts, but performs poorly on top-10 and top-25 because of diversity collapse (Formont et al., 18 Mar 2026). This result is important because it separates optimization quality from exploration quality.

System	Benchmark regime	Reported result
CMG	Multi-objective success rate	6.98% vs 3.56% for VJTNN
ChatMol	ESR1 multi-objective binding	$^4$ 4; 4.76% higher overall performance
NovoMolGen	PMO and docking	PMO total 16.70; parp1 hit rate 90.24%
DPO + curriculum	GuacaMol	Total 18.235; Perindopril MPO 0.883
MolRGen RL-Mistral	Reward-based top-1	0.64 at $^4$ 5; 0.68 at $^4$ 6

The heterogeneity of these numbers suggests that “state of the art” in NovoMolGen is benchmark-specific. Distributional fidelity, exact numeric control, docking performance, and diversity under repeated sampling do not necessarily improve together.

6. Application domains and experimentally grounded workflows

NovoMolGen is increasingly evaluated in domain-specific workflows rather than purely abstract benchmarks. In mosquito insecticide discovery, Mos-Gen targets disulfide-containing allicin derivatives and uses a motif-aware generative–discriminative pipeline. Fourteen compounds were synthesized by a one-step method; 8 of the 14 had no prior literature record, 7 of 9 predicted positives were active, and none of the 5 predicted negatives showed activity (Wang et al., 1 Jun 2026). The absence of activity among predicted negatives supports the framework’s high specificity on the tested set, although broader validation was explicitly noted as necessary (Wang et al., 1 Jun 2026).

In kinase-directed design, NovoMol fine-tunes an oral-bioavailability-oriented SMILES LSTM on 450 known tyrosine-kinase inhibitors and then generates PDGFR $^4$ 7 candidates (Rao, 2023). Out of 311 generated candidates, 44% outperformed imatinib’s reported binding affinity of $^4$ 8 kcal/mol in AutoDock Vina, and the best candidate reached $^4$ 9 kcal/mol with QED $0.0380$0 (Rao, 2023). DGMM pursues a different route to lead optimization by evolving a discrete “mol-gene” representation under docking-based competition. In the CHK1 case study, the generated molecules CHK1-F-1, CHK1-F-2, and CHK1-F-3 reached docking scores of $0.0380$1, $0.0380$2, and $0.0380$3 kcal/mol, respectively, while preserving interactions with residues Glu17 and Cys87 identified as central in the study’s structural analysis (Fang et al., 2024).

Phenotype- and assay-conditioned design extend NovoMolGen beyond property vectors. Gx2Mol conditions a SMILES LSTM on a VAE-derived latent representation of gene-expression profiles from LINCS L1000 and CREEDS (Li et al., 2024). It generated 1,322 molecules, of which 1,171 were valid, 972 were unique, and 1,167 were novel, corresponding to 88.6% validity, 83.0% uniqueness, and 99.7% novelty (Li et al., 2024). MADGEN addresses a structurally different problem—annotating dark chemical space from MS/MS spectra—but does so with a two-stage scaffold-retrieval and scaffold-conditioned generation framework that is unmistakably within the NovoMolGen toolkit: on NIST23, predictive retrieval yields top-1 accuracy 10.3%, whereas oracle retrieval lifts top-1 accuracy to 49.0% (Wang et al., 3 Jan 2025).

At platform scale, Chemistry42 integrates de novo generation with medicinal chemistry rules, 2D/3D assessment, and cloud-native execution (Ivanenkov et al., 2021). It supports both LBDD and SBDD workflows, interoperates with PandaOmics for target discovery, and reports a standard full-model experiment time of about 72 hours (Ivanenkov et al., 2021). Although that paper does not provide a campaign-specific numerical case study, it shows how NovoMolGen methods have become components of end-to-end discovery systems rather than isolated generators.

7. Limitations, evaluation caveats, and anticipated directions

Several limitations recur across the literature. The first is reliance on surrogate evaluators. CMG depends on PropNet, SimNet, and a DRD2 proxy, and explicitly notes that miscalibration or out-of-distribution errors in these predictors can lead to over-optimization of inaccurate targets (Shin et al., 2020). M$0.0380$4olGen relies on RDKit oracles for QED, LogP, and MW and on a pretrained DimeNet++ predictor for HOMO and LUMO; the paper notes that oracle fidelity directly constrains precise targeting (Li et al., 15 Jan 2026). MolRGen extends this problem to docking and ADMET-style reward functions, where programmatic verification is convenient but biologically imperfect (Formont et al., 18 Mar 2026).

A second limitation is the weak alignment between pretraining metrics and downstream utility. The NovoMolGen foundation-model study shows that language-model loss and FCD improve steadily during pretraining, yet PMO and docking gains saturate early and correlate weakly with pretraining metrics (Chitsaz et al., 19 Aug 2025). This suggests that generic molecular syntax and distribution learning do not fully capture the functional semantics required for docking, MPO, or medicinal chemistry heuristics.

A third limitation is diversity collapse under strong optimization pressure. MolRGen demonstrates this most explicitly: GRPO-trained RL-Mistral is highly effective at top-1 optimization but poor at top-10 and top-25 because it repeatedly emits similar high-reward molecules (Formont et al., 18 Mar 2026). ChatMol addresses a related issue from another angle, observing that LLMs struggle with numeric compliance and that explicit numerical embeddings improve adherence to prompt values (Fan et al., 27 Feb 2025). Together these results suggest that reasoning ability, numeric control, and diversity must be optimized jointly.

Finally, many papers identify missing components rather than algorithmic failures. Chemistry42 does not specify exact docking or physics engines, retrosynthetic route planning, or detailed campaign-level quantitative outcomes in its platform paper (Ivanenkov et al., 2021). ChatMol does not include explicit PAINS filters or retrosynthesis planning, although the framework is described as flexible enough to incorporate them (Fan et al., 27 Feb 2025). NExT-Mol notes remaining generalization gaps on fully unseen scaffolds and points to stronger cross-modal alignment, classifier-free or particle guidance, and ligand–protein or ligand–RNA conditioning as future directions (Liu et al., 18 Feb 2025). Megalodon identifies quadratic edge-feature cost and higher flow-matching relaxation energies as continuing efficiency and fidelity bottlenecks (Reidenbach et al., 23 May 2025).

Across these works, proposed directions are remarkably consistent: uncertainty-aware property predictors, exploration-aware RL objectives, richer motif or fragment libraries, explicit synthesis and safety constraints, stronger 1D–3D alignment, and broader multimodal conditioning (Shin et al., 2020, Formont et al., 18 Mar 2026, Reidenbach et al., 23 May 2025). This suggests that the next stage of NovoMolGen will likely be defined less by a single winning generator class than by tighter integration between generation, verification, and domain-specific scientific priors.