---
title: 'NovoMolGen: Transformer-Based Molecular Generation'
url: https://www.emergentmind.com/topics/novomolgen
type: topic
---

# NovoMolGen: Transformer-Based Molecular Generation

NovoMolGen denotes both a broad research program in *de novo* molecular generation and, more specifically, a family of transformer-based foundation models pretrained on 1.5 billion ZINC-22 molecules for molecule design [2508.13408]. In the broader sense, it covers algorithmic systems that explore a chemical space commonly described as lying between $10^{23}$ and $10^{60}$ synthesizable small molecules, with the aim of generating valid, novel structures that satisfy predefined property, scaffold, biological, or experimental constraints [2508.13408]. Recent work grouped under this label spans conditional sequence translation for lead optimization, fragment- and motif-based assembly, graph and energy-based generation, equivariant 2D–3D co-design, language-model prompting, retrieval-augmented reasoning, and multimodal conditioning on spectra, gene-expression profiles, protein pockets, and 3D conformations [2010.13908][2302.01129][2603.18256].

## 1. Scope and research landscape

Within the current literature, NovoMolGen is not a single algorithmic paradigm but a heterogeneous design space. Some systems treat molecule generation as direct sequence modeling over SMILES or SELFIES; others operate on graphs, motifs, BRICS fragments, Murcko scaffolds, or continuous 3D coordinates. Conditioning can be absent, scalar, symbolic, structural, or multimodal. Representative objectives include unconstrained distribution learning, single- and multi-property optimization, scaffold preservation, target-conditioned docking optimization, phenotype-conditioned design, and *de novo* reconstruction from MS/MS spectra [2508.13408][2102.04600][2501.01950].

A useful way to situate the field is by the interface through which constraints enter generation. Controlled Molecule Generator frames multi-property optimization as supervised sequence translation from a source molecule $X$ to a target molecule $Y$ under similarity and property constraints [2010.13908]. Chemistry42, developed within Insilico Medicine’s Pharma.ai suite, exposes ligand-based and structure-based workflows through a cloud-native platform that combines more than 30 generative models with multi-objective scoring and reinforcement learning [2101.09050]. At the other end of the spectrum, ChemGE performs population-based grammatical evolution over SMILES and emphasizes concurrency and diversity rather than differentiable property conditioning [1804.02134].

| Family | Representative systems | Primary constraint interface |
|---|---|---|
| Sequence and language models | CMG, LLamol, ChatMol, NovoMolGen | Property vectors, prompts, numerical tokens |
| Fragment and motif systems | MiCaM, M$^4$olGen, DGMM | Mined motifs, BRICS edits, mol-gene codes |
| Graph and 3D generators | GEM, MolCode, NExT-Mol, Megalodon | Substructures, topology–geometry co-design, diffusion or flows |
| Scientific-conditioning systems | Gx2Mol, MADGEN, Mos-Gen | Gene expression, MS/MS spectra, frozen 3D encoders |

This taxonomy suggests that NovoMolGen has become less a single architecture than a unifying problem setting: the controlled exploration of chemical space under explicit validity, property, and task constraints.

## 2. Molecular representations and conditioning interfaces

The representational layer is a central organizing principle of NovoMolGen. The 2025 NovoMolGen foundation-model study systematically compared SMILES, SELFIES, DeepSMILES, and SAFE, together with atomwise tokenization and BPE, and reported that SMILES+BPE is a robust default, with best FCD around $0.0380$ and strong performance across unconstrained generation, PMO, and docking tasks [2508.13408]. SELFIES remains attractive because it guarantees validity by construction and is therefore used in systems such as NExT-Mol, which first samples a 1D SELFIES string and only then predicts 3D conformers [2502.12638].

Other systems modify representations to better match the inductive biases of their generators. ChatMol uses a simplified SELFIES representation with brackets removed and tokens space-delimited, motivated by tokenizer efficiency and by the desire to turn property-constrained molecule design into a promptable LLM task [2502.19794]. LLamol uses SMILES with DeepChem’s BERT tokenizer and prepends numerical condition embeddings and optional token-sequence fragments as a prefix, allowing a single decoder-only transformer to support up to four simultaneous conditions [2311.14407]. NovoMol, by contrast, remains entirely character-level, padding SMILES to length 100 and training an encoder–decoder LSTM on one-hot vectors over a 45-character vocabulary [2312.01527].

Fragment and scaffold representations define a second major branch. MiCaM mines a connection-aware motif vocabulary by iteratively merging frequent adjacent subgraphs and then turning broken external bonds into explicit “*” attachment sites with preserved bond-type information [2302.01129]. M$^4$olGen decomposes molecules with BRICS, constructs a large neighbor dataset of 1,171,193 single-edit pairs, and treats generation as fragment-level add, remove, or replace operations toward precise numeric targets [2601.10131]. MADGEN uses Murcko scaffolds as an intermediate object: a contrastive retriever maps spectra to candidate scaffolds, and a spectrum-conditioned graph generator completes the undecided edges [2501.01950].

A third branch centers on explicit 3D structure. Mos-Gen feeds 3D conformations through a frozen Uni-Mol encoder and conditions a VAE decoder on those features, with a dedicated `[SS]` token to bias generation toward disulfide-containing allicin derivatives [2606.01846]. MolCode and Megalodon jointly model topology and coordinates rather than treating 3D structure as a post hoc annotation [2304.12436][2505.18392]. NExT-Mol separates the roles: a 960M-parameter SELFIES LM handles 2D-valid generation, while a Diffusion Molecular Transformer predicts conformers [2502.12638].

## 3. Learning objectives, control mechanisms, and decoding

NovoMolGen systems employ a wide range of training criteria, but several objective families recur. Maximum-likelihood sequence modeling remains common, both in SMILES language models and in sequence-to-sequence lead-optimization systems. CMG optimizes a Transformer encoder–decoder with a conditional log-likelihood objective and augments it with two frozen auxiliary networks: PropNet, a BiLSTM+MLP property regressor trained on 257,565 molecules with validation MSE approximately $0.0855$, and SimNet, a shared-BiLSTM similarity classifier trained on about 997,773 molecule pairs with validation accuracy approximately $0.976$ [2010.13908]. The resulting joint loss,
$\mathcal{L}_{\mathrm{CMG}}=\mathcal{L}_T+\lambda_p\mathcal{L}_P+\lambda_s\mathcal{L}_S$,
couples token prediction to property satisfaction and source-target similarity [2010.13908].

Variational objectives are prominent when latent interpolation or multimodal conditioning is required. Mos-Gen uses a standard ELBO with a multi-slot latent bottleneck of $K=16$ slots, while motif awareness is introduced through token engineering and disulfide-enriched fine-tuning rather than an explicit motif regularizer [2606.01846]. Gx2Mol uses a VAE over gene-expression profiles to extract a 64-dimensional phenotype vector that is concatenated to each LSTM input step during SMILES generation, enabling generation conditioned on target perturbation or disease-reversal signatures [2412.19422]. DGMM uses a teacher VAE and a distilled discrete VAE to create a quantized “mol-gene” code on which mutation and crossover are performed directly [2406.12910].

Diffusion, flow matching, and exact likelihood also play major roles. NExT-Mol applies a DDPM-style coordinate denoising objective on 3D conformers after 1D generation [2502.12638]. Megalodon couples continuous denoising for coordinates with discrete denoising for atom types, bond types, and charges, with independent time variables for continuous and discrete tracks but joint training at every layer [2505.18392]. MolCode uses normalizing flows over dequantized atom types, bond types, and local spherical coordinates, yielding exact likelihood training for a joint 2D–3D autoregressive model [2304.12436].

Preference-based and reinforcement objectives are increasingly used when exact numeric control is difficult. ChatMol performs supervised fine-tuning followed by ranking-based sequence calibration, scoring sampled candidates with RDKit or AutoDock-GPU and optimizing a combined objective
$\mathcal{L}_{\mathrm{total}}=\gamma_1\mathcal{L}_{ce}+\gamma_2\mathcal{L}_{rank}$,
with $\gamma_1=0.1$ and $\gamma_2=10.0$ [2502.19794]. A separate line of work replaces RL with Direct Preference Optimization, pairing high- and low-scoring molecules and reporting that all 20 GuacaMol tasks can be completed in 60 hours on a single A100 GPU, compared with about 400 hours for MolRL-MGPT [2504.01389]. MolRGen and M$^4$olGen instead adopt GRPO-style policy optimization, using on-the-fly verifiable rewards and group-relative baselines rather than pairwise labels [2603.18256][2601.10131].

Decoding is correspondingly varied. CMG uses modified beam re-ranking with PropNet and SimNet scores at sequence completion [2010.13908]. LLamol uses temperature sampling at $0.8$ and relies on Stochastic Context Learning to remain robust to missing conditions [2311.14407]. ChatMol augments numerical tokens with sinusoidal numeric embeddings and reports that Pearson correlation between instructed values and generated properties increased by up to $0.49$ [2502.19794]. These systems collectively indicate that control in NovoMolGen is increasingly achieved through a combination of conditioning interface, auxiliary evaluators, and structured decoding rather than through the generator alone.

## 4. Topology–geometry co-design and 3D molecular generation

A major development in NovoMolGen has been the move from 2D-valid string or graph generation toward explicit 3D structure generation. NExT-Mol embodies a staged design: a 960M-parameter SELFIES LM, pretrained on 1.8 billion ZINC-15 molecules and about 90 billion SELFIES tokens, samples a 100%-valid molecule; a diffusion model then predicts its conformers [2502.12638]. On GEOM-DRUGS, the system reports a 26% relative improvement in 3D FCD for *de novo* 3D generation and a 13% average relative gain for conditional 3D generation on QM9-2014 [2502.12638]. The transfer of pretrained 1D token embeddings into the 3D diffusion model also improved conformer prediction on unseen scaffolds [2502.12638].

MolCode addresses the same problem by modeling 2D topology and 3D geometry within a single equivariant flow-based factorization. It represents a molecule as $(V,A,R)$, uses SphereNet or EGNN encoders plus bond-aware global attention, and generates atoms, bonds, and local spherical coordinates autoregressively [2304.12436]. On QM9 random generation, MolCode reports 99.95% validity and 98.75% uniqueness; in structure-based drug design, it reports a 61.8% high-affinity ratio, defined as the fraction of protein pockets for which at least one generated ligand outperforms the reference ligand in Vina score [2304.12436].

Megalodon pushes co-design further by jointly denoising continuous coordinates and discrete atom, bond, and charge variables with a fused transformer trunk and a lightweight equivariant coordinate-update layer [2505.18392]. On GEOM Drugs, diffusion Megalodon reports Atom Stability $0.999$, Molecule Stability $0.977$, and Connected Validity $0.927$, while its energetics benchmark shows median $\Delta E_{\mathrm{relax}}=3.17$ kcal/mol, compared with $6.36$ for EQGAT-diff and $32.96$ for SemlaFlow [2505.18392]. The larger 40.6M-parameter model generates up to 49x more valid large molecules than EQGAT-diff in the $>72$-atom regime [2505.18392].

Mos-Gen illustrates a different use of 3D structure: not generic conformer realism, but motif-aware generation in a low-label application domain. A frozen Uni-Mol encoder pretrained on about 18 million unlabeled molecules conditions a VAE with a GRU decoder and an explicit `[SS]` token; a separate Uni-Mol classification head, fine-tuned on about 200 labeled compounds, provides the predictive screen [2606.01846]. In prospective validation, 14 compounds were synthesized, 8 had no prior literature record, 7 of 9 predicted positives were active, and none of the 5 predicted negatives showed activity, corresponding to a hit rate of approximately 78% among predicted positives [2606.01846].

| System | Core mechanism | Reported result |
|---|---|---|
| NExT-Mol | SELFIES LM plus 3D diffusion | 26% relative improvement in 3D FCD on GEOM-DRUGS |
| MolCode | Equivariant flow-based 2D–3D co-design | 99.95% validity; 98.75% uniqueness |
| Megalodon | Joint continuous/discrete denoising co-design | Up to 49x more valid large molecules |
| Mos-Gen | Frozen Uni-Mol plus motif-aware VAE | 78% hit rate among predicted positives |

These systems indicate a shift in NovoMolGen from chemistry-valid generation toward geometry-valid and energetically plausible generation, with explicit 3D structure now treated as a first-class object rather than a downstream annotation.

## 5. Benchmarks, metrics, and reported performance regimes

Evaluation in NovoMolGen is fragmented because the field spans distribution learning, constrained optimization, docking, and multimodal reconstruction. Nevertheless, a relatively stable metric vocabulary has emerged: validity, uniqueness, novelty, similarity, diversity, FCD, scaffold or fragment similarity, conformer metrics such as COV and AMR, docking score, success rate under hard constraints, and, in reasoning-oriented settings, top-$k$ reward and diversity-aware top-$k$ [2508.13408][2603.18256].

Controlled multi-property optimization is exemplified by CMG. Under the multi-objective setting $\delta=0.4$, $\Delta\mathrm{PlogP}\ge 1.0$, $\mathrm{QED}\ge 0.9$, and $\mathrm{DRD2}>0.5$, CMG reports a success rate of 6.98% on all 2,365 samples, versus 3.56% for VJTNN; in the single-objective PlogP benchmark, CMG reports $3.92\pm1.88$ with diversity $0.545$ [2010.13908]. ChatMol reports 94.5% success and diversity $0.703$ for a single-property logP targeting task, while in the multi-objective ESR1 binding task it reports a best $K_D$ of $0.25$, QED $0.53$, SA $2.74$, and the highest overall performance, surpassing previous methods by 4.76% [2502.19794].

The 2025 NovoMolGen foundation models establish a distinct benchmark regime based on large-scale pretraining followed by PMO and docking evaluation. The best SMILES+BPE configuration reports a PMO total score of 16.70 across 23 tasks, compared with 16.30 for f-RAG and 14.01 for REINVENT, while docking on parp1 yields a best docking score of $-14.90$ and a hit rate of 90.24% [2508.13408]. A key finding is that pretraining loss and FCD correlate weakly with downstream PMO quality, with reported correlation $r=0.376$, $p=0.358$ [2508.13408].

Reasoning-based evaluation is codified most explicitly in MolRGen, which defines both raw top-$k$ and a diversity-aware top-$k$ score under fingerprint-based similarity constraints [2603.18256]. In that setting, the RL-fine-tuned RL-Mistral model reaches top-1 scores of 0.64, 0.67, and 0.68 for $n_r=1,5,10$ rollouts, but performs poorly on top-10 and top-25 because of diversity collapse [2603.18256]. This result is important because it separates optimization quality from exploration quality.

| System | Benchmark regime | Reported result |
|---|---|---|
| CMG | Multi-objective success rate | 6.98% vs 3.56% for VJTNN |
| ChatMol | ESR1 multi-objective binding | $K_D=0.25$; 4.76% higher overall performance |
| NovoMolGen | PMO and docking | PMO total 16.70; parp1 hit rate 90.24% |
| DPO + curriculum | GuacaMol | Total 18.235; Perindopril MPO 0.883 |
| MolRGen RL-Mistral | Reward-based top-1 | 0.64 at $n_r=1$; 0.68 at $n_r=10$ |

The heterogeneity of these numbers suggests that “state of the art” in NovoMolGen is benchmark-specific. Distributional fidelity, exact numeric control, docking performance, and diversity under repeated sampling do not necessarily improve together.

## 6. Application domains and experimentally grounded workflows

NovoMolGen is increasingly evaluated in domain-specific workflows rather than purely abstract benchmarks. In mosquito insecticide discovery, Mos-Gen targets disulfide-containing allicin derivatives and uses a motif-aware generative–discriminative pipeline. Fourteen compounds were synthesized by a one-step method; 8 of the 14 had no prior literature record, 7 of 9 predicted positives were active, and none of the 5 predicted negatives showed activity [2606.01846]. The absence of activity among predicted negatives supports the framework’s high specificity on the tested set, although broader validation was explicitly noted as necessary [2606.01846].

In kinase-directed design, NovoMol fine-tunes an oral-bioavailability-oriented SMILES LSTM on 450 known tyrosine-kinase inhibitors and then generates PDGFR$\alpha$ candidates [2312.01527]. Out of 311 generated candidates, 44% outperformed imatinib’s reported binding affinity of $-9.4$ kcal/mol in AutoDock Vina, and the best candidate reached $-12.9$ kcal/mol with QED $0.686$ [2312.01527]. DGMM pursues a different route to lead optimization by evolving a discrete “mol-gene” representation under docking-based competition. In the CHK1 case study, the generated molecules CHK1-F-1, CHK1-F-2, and CHK1-F-3 reached docking scores of $-10.92$, $-11.25$, and $-11.49$ kcal/mol, respectively, while preserving interactions with residues Glu17 and Cys87 identified as central in the study’s structural analysis [2406.12910].

Phenotype- and assay-conditioned design extend NovoMolGen beyond property vectors. Gx2Mol conditions a SMILES LSTM on a VAE-derived latent representation of gene-expression profiles from LINCS L1000 and CREEDS [2412.19422]. It generated 1,322 molecules, of which 1,171 were valid, 972 were unique, and 1,167 were novel, corresponding to 88.6% validity, 83.0% uniqueness, and 99.7% novelty [2412.19422]. MADGEN addresses a structurally different problem—annotating dark chemical space from MS/MS spectra—but does so with a two-stage scaffold-retrieval and scaffold-conditioned generation framework that is unmistakably within the NovoMolGen toolkit: on NIST23, predictive retrieval yields top-1 accuracy 10.3%, whereas oracle retrieval lifts top-1 accuracy to 49.0% [2501.01950].

At platform scale, Chemistry42 integrates *de novo* generation with medicinal chemistry rules, 2D/3D assessment, and cloud-native execution [2101.09050]. It supports both LBDD and SBDD workflows, interoperates with PandaOmics for target discovery, and reports a standard full-model experiment time of about 72 hours [2101.09050]. Although that paper does not provide a campaign-specific numerical case study, it shows how NovoMolGen methods have become components of end-to-end discovery systems rather than isolated generators.

## 7. Limitations, evaluation caveats, and anticipated directions

Several limitations recur across the literature. The first is reliance on surrogate evaluators. CMG depends on PropNet, SimNet, and a DRD2 proxy, and explicitly notes that miscalibration or out-of-distribution errors in these predictors can lead to over-optimization of inaccurate targets [2010.13908]. M$^4$olGen relies on RDKit oracles for QED, LogP, and MW and on a pretrained DimeNet++ predictor for HOMO and LUMO; the paper notes that oracle fidelity directly constrains precise targeting [2601.10131]. MolRGen extends this problem to docking and ADMET-style reward functions, where programmatic verification is convenient but biologically imperfect [2603.18256].

A second limitation is the weak alignment between pretraining metrics and downstream utility. The NovoMolGen foundation-model study shows that language-model loss and FCD improve steadily during pretraining, yet PMO and docking gains saturate early and correlate weakly with pretraining metrics [2508.13408]. This suggests that generic molecular syntax and distribution learning do not fully capture the functional semantics required for docking, MPO, or medicinal chemistry heuristics.

A third limitation is diversity collapse under strong optimization pressure. MolRGen demonstrates this most explicitly: GRPO-trained RL-Mistral is highly effective at top-1 optimization but poor at top-10 and top-25 because it repeatedly emits similar high-reward molecules [2603.18256]. ChatMol addresses a related issue from another angle, observing that LLMs struggle with numeric compliance and that explicit numerical embeddings improve adherence to prompt values [2502.19794]. Together these results suggest that reasoning ability, numeric control, and diversity must be optimized jointly.

Finally, many papers identify missing components rather than algorithmic failures. Chemistry42 does not specify exact docking or physics engines, retrosynthetic route planning, or detailed campaign-level quantitative outcomes in its platform paper [2101.09050]. ChatMol does not include explicit PAINS filters or retrosynthesis planning, although the framework is described as flexible enough to incorporate them [2502.19794]. NExT-Mol notes remaining generalization gaps on fully unseen scaffolds and points to stronger cross-modal alignment, classifier-free or particle guidance, and ligand–protein or ligand–RNA conditioning as future directions [2502.12638]. Megalodon identifies quadratic edge-feature cost and higher flow-matching relaxation energies as continuing efficiency and fidelity bottlenecks [2505.18392].

Across these works, proposed directions are remarkably consistent: uncertainty-aware property predictors, exploration-aware RL objectives, richer motif or fragment libraries, explicit synthesis and safety constraints, stronger 1D–3D alignment, and broader multimodal conditioning [2010.13908][2603.18256][2505.18392]. This suggests that the next stage of NovoMolGen will likely be defined less by a single winning generator class than by tighter integration between generation, verification, and domain-specific scientific priors.

Source: https://www.emergentmind.com/topics/novomolgen