MolRGen: Unified Molecular Generation

• MolRGen is a comprehensive framework that defines de novo molecular generation and property prediction as prompt-completion tasks using reasoning-based LLMs.
• It employs reinforcement learning with reward shaping and diversity-aware metrics to balance multi-objective optimization and chemical exploration.
• The approach integrates interpretable substructure extraction and conditional SMILES generation, enabling flexible and scalable drug-design pipelines.

Searching arXiv for recent and related papers on MolRGen to ground the article. arxiv_search query: "MolRGen de novo molecular generation reasoning models LLamol rationale substructures" MolRGen denotes a line of research in de novo molecular generation, but in the materials considered here it refers primarily to the unified training-and-evaluation framework introduced in "MolRGen: A Training and Evaluation Setting for De Novo Molecular Generation with Reasonning Models" (Formont et al., 18 Mar 2026). In that formulation, MolRGen is a large-scale benchmark and dataset for training and evaluating reasoning-based LLMs on de novo molecular generation and molecular property prediction without requiring supervised molecule pairs. The label also appears in an earlier substructure-based context as a summary name for the rationale-driven graph-generation method in "Multi-Objective Molecule Generation using Interpretable Substructures" (Jin et al., 2020). In practical workflow discussions, conditional SMILES generators such as "LLamol: A Dynamic Multi-Conditional Generative Transformer for De Novo Molecular Design" are presented as building blocks for MolRGen pipelines (Dobberstein et al., 2023).

1. Terminological scope and research context

The 2026 MolRGen framework is explicitly designed to study reasoning-based LLMs on de novo molecular generation, defined as generating entirely novel drug-like compounds, and on downstream property-prediction tasks (Formont et al., 18 Mar 2026). Its central motivation is that prior approaches either emphasize evaluation alone or depend on training setups with ground-truth labels such as molecule pairs with known property modifications, whereas such supervision is unavailable in de novo molecular generation.

An earlier usage appears in the integrated summary of the 2020 paper "Multi-Objective Molecule Generation using Interpretable Substructures" (Jin et al., 2020), where MolRGen is used for a method also referred to as RationaleRL. There, the emphasis is not on reasoning LLMs, but on extracting interpretable substructures, called molecular rationales, and composing molecules through a mixture-of-experts graph generative model.

These usages are related by a common objective—multi-constraint generation of novel molecules—but they operate at different methodological levels. The 2026 MolRGen work formalizes a benchmark, reward environment, and RL training setting for LLMs (Formont et al., 18 Mar 2026), whereas the 2020 rationale-based work specifies a concrete graph-generative architecture for satisfying multiple property constraints (Jin et al., 2020). A plausible implication is that the later benchmark can be viewed as an evaluation and training substrate broad enough to compare a wide range of molecular generation strategies, including structured generators and LLM-based systems.

2. Core problem formulations in the 2026 framework

MolRGen defines two tasks in a prompt-completion paradigm: de novo molecular generation and molecular property prediction (Formont et al., 18 Mar 2026).

For de novo molecular generation, the prompt is

$q = \{ (\nu_i, \rho_i, x_i, \sigma_i) \}_{i=1}^n,$

where $\nu_i$ denotes an objective type such as "maximize" or "above threshold", $\rho_i$ is a molecular property or docking target, $x_i$ is a reference molecule or protein pocket for docking, and $\sigma_i$ is a margin or threshold. A model $\pi$ samples a completion $o \sim \pi(\cdot \mid q)$, from which a SMILES string is parsed and scored by per-property rewards $r_i(q,o) \in [0,1]$. The overall reward is the geometric mean,

$$r(q,o) = \Bigl( \prod_{i=1}^n r_i(q,o) \Bigr)^{1/n},$$

so only candidates that optimize all objectives receive high reward (Formont et al., 18 Mar 2026).

For molecular property prediction, each prompt is

$q = (\rho, task, x, \sigma),$

where $\nu_i$0 is the property to predict, $\nu_i$1, $\nu_i$2 is the query molecule in SMILES form, and $\nu_i$3 is the standard deviation of training labels for normalization. The model again samples a completion, from which a numeric answer is extracted. The reward is the predictive score: classification accuracy or normalized $\nu_i$4, converted to a $\nu_i$5 scale, or a Spearman correlation penalized by the fraction of valid predictions (Formont et al., 18 Mar 2026).

This formulation is notable because no supervised molecular pairs are required. All supervision comes from the black-box reward computed at generation time. Within the stated setting, this is the central difference between MolRGen and training regimes that rely on known high-scoring exemplars or explicit property-modification pairs.

3. Dataset construction and benchmark design

MolRGen represents molecules as canonical SMILES and couples molecular descriptors with docking-based objectives (Formont et al., 18 Mar 2026). For de novo generation, docking targets are drawn from SAIR, containing approximately 1 million predicted co-structures over approximately 5 thousand proteins, and SIU, which contains experimentally resolved pockets. After filtering for high-confidence structures and clustering ligand-contact residues through IoU-based hierarchical clustering, the framework identifies approximately 4.5 thousand distinct protein pockets. Each pocket is cleaned via Meeko and stored as a PDB.

Twelve classical molecular descriptors are computed via RDKit, including QED, SA, LogP, MW, TPSA, HBA, HBD, and rotatable bonds (Formont et al., 18 Mar 2026). Prompt generation samples 1 to 3 objectives per prompt and over-samples drug-like properties such as QED and SA to encourage feasibility. The generation benchmark contains 49 thousand training prompts and 1 thousand prompts each for validation and test, with splits constructed so that no protein pocket appears across splits.

For property prediction, MolRGen aggregates 27 public ADMET benchmarks from sources including TDC, Polaris, ASAP, Biogen, and AstraZeneca, producing 52 thousand prompts (Formont et al., 18 Mar 2026). Regression labels are log-transformed when ranges are broad, classification labels remain binary, and each prompt stores the standard deviation $\nu_i$6 for normalization.

The benchmark design ties molecular generation to realistic reward computation rather than to fixed gold outputs. This suggests a research emphasis on policy quality under black-box objectives, not merely on imitation of existing molecules.

4. Training protocol, reward shaping, and diversity-aware evaluation

MolRGen trains LLMs with reinforcement learning. Models receive a system message encouraging chain-of-thought reasoning followed by the user prompt. For de novo generation, a pre-trained LLM such as Mistral-Small-24B is fine-tuned with a Group Relative Policy Optimization objective (Formont et al., 18 Mar 2026). In each batch, for prompts $\nu_i$7, the method samples $\nu_i$8 rollouts, computes rewards, forms centered advantages

$\nu_i$9

normalizes them across the batch, and applies a clipped importance-weighted log-likelihood loss analogous to PPO/GRPO.

Reward functions are smooth transformations of property values into $\rho_i$0. For "above threshold" objectives, the framework may use a sigmoid transformation of the gap between the property value and the threshold; docking scores are inverted and rescaled; invalid SMILES receive reward $\rho_i$1 (Formont et al., 18 Mar 2026). The geometric-mean aggregation is chosen over an arithmetic mean specifically to penalize failure on any individual objective.

A central contribution is the diversity-aware top-$\rho_i$2 score. Standard top-$\rho_i$3 evaluation measures the average best-$\rho_i$4 aggregate reward among multiple samples but ignores structural redundancy. MolRGen therefore introduces a diversity-aware variant that greedily selects $\rho_i$5 molecules from the rollouts under a similarity cap $\rho_i$6, typically based on Tanimoto similarity over ECFP6 fingerprints, always choosing the highest-reward candidate whose maximum similarity to earlier selections is below the threshold (Formont et al., 18 Mar 2026). The result is a metric that jointly rewards high quality and exploration.

This evaluation choice addresses a common misconception in generative modeling: strong top-1 performance does not necessarily indicate broad chemical exploration. In MolRGen, diversity-sensitive evaluation is treated as necessary rather than optional because a model can otherwise optimize reward by collapsing onto a narrow mode.

5. Empirical findings, failure modes, and open problems

MolRGen compares several reasoning-oriented and chemically specialized LLMs, including MiniMax-M2, Qwen3-30B, Qwen3-Next-80B, GPT-OSS-120B, R1-Llama-70B, R1-Qwen-32B, Gemma-3-27B, Llama-3.3-70B, ChemDFM-R-14B, Ether0-24B, ChemDFM-v2.0-14B, and the newly trained RL-Mistral-24B (Formont et al., 18 Mar 2026).

On the standard top-1 metric with $\rho_i$7, naive GRPO-fine-tuned Mistral far outperformed all baselines, reaching top-1 $\rho_i$8 versus the next best at approximately $\rho_i$9 (Formont et al., 18 Mar 2026). However, when either $x_i$0 or the number of rollouts increased, RL-Mistral’s performance collapsed: top-10 was approximately $x_i$1 versus MiniMax’s approximately $x_i$2 at $x_i$3. Pretrained models such as MiniMax-M2 or the Qwen3 series required 3 to 5 times more samples to match RL-Mistral’s top-1, but then improved steadily up to $x_i$4.

The diversity-aware analysis clarifies this pattern. Under weak diversity constraints around $x_i$5, MiniMax-M2 remained best; under strong constraints around $x_i$6, ChemDFM-R overtook MiniMax, indicating better exploration of distinct chemical neighborhoods (Formont et al., 18 Mar 2026). RL-Mistral, despite very high top-1, scored near-zero under even moderate diversity thresholds, which the paper identifies as evidence of mode collapse.

On property prediction, Qwen3-Next-80B and Qwen3-30B outperformed specialized models on classification tasks, with accuracies up to approximately $x_i$7, whereas ChemDFM-R and RL-Mistral were around approximately $x_i$8 (Formont et al., 18 Mar 2026). On regression tasks measured by normalized Spearman correlation, most models were around $x_i$9 to $\sigma_i$0, and only ChemDFM-R and RL-Mistral occasionally exceeded $\sigma_i$1 on a few properties. Many completions failed to parse into valid numeric answers, highlighting an output-formatting problem as well as a modeling problem.

The appendix-level analysis attributes the collapse to GRPO’s exponentiated-advantage updates, which concentrate posterior mass on a single or narrow cluster of high-reward molecules (Formont et al., 18 Mar 2026). Proposed remedies include a coverage regularizer penalizing low support over greedy similarity clusters and a group-level bonus

$\sigma_i$2

intended to favor under-represented clusters. These are explicitly framed as future work.

The broader limitations stated for MolRGen are also methodological. Docking scores and RDKit proxies only approximate real binding and drug-likeness; experiments should be extended to wet-lab validation; RL fine-tuning at scale remains expensive; and future work is expected to investigate coverage bonuses, synthesizability constraints, and extensions to reaction-planning tasks (Formont et al., 18 Mar 2026).

6. Earlier MolRGen as rationale-based multi-objective generation

In the 2020 rationale-based formulation, MolRGen operates by extracting interpretable substructures and expanding them into full molecules through graph generation (Jin et al., 2020). For each property $\sigma_i$3, the objective is to find a connected subgraph $\sigma_i$4 of a positive molecule such that $\sigma_i$5 and $\sigma_i$6. Because exhaustive subgraph enumeration is intractable, the method uses Monte Carlo Tree Search. Each state is a connected subgraph; each action deletes one peripheral bond or ring; and the search uses a PUCT rule with visit counts, total values, mean values, and immediate rewards.

Single-property rationales are collected into vocabularies $\sigma_i$7. Multi-property rationales are then constructed by taking one rationale from each $\sigma_i$8, computing their Maximum Common Substructure, superposing them so that the MCS aligns, and discarding merged candidates that fail any property threshold (Jin et al., 2020). The final multi-property rationale vocabulary is denoted $\sigma_i$9.

Given a rationale $\pi$0, the method expands it into a molecule through a latent-variable VAE,

$\pi$1

The encoder is a Message-Passing Network over the graph with the rationale marked; the decoder generates the molecule in BFS order. At each step it decides whether to add a new atom, predicts the atom type if an addition occurs, and then predicts bond types to frontier atoms (Jin et al., 2020).

For multi-objective optimization, the choice of rationale is treated as a latent expert with categorical distribution $\pi$2, giving the mixture model

$\pi$3

Learning uses an entropy-regularized expected-reward objective over $\pi$4, or equivalently a penalized negative-reward objective with property weights and an entropy term that encourages exploration over rationales (Jin et al., 2020).

Implementation details reported in the summary include pre-training on 1.02 million $\pi$5 pairs from ChEMBL, MCTS with 20 iterations per positive molecule, $\pi$6, and maximum subgraph size $\pi$7 (Jin et al., 2020). The extraction process yielded approximately 6,299 rationales for GSK3$\pi$8 and approximately 417 for JNK3. Multi-property vocabularies contained 9,719 two-constraint rationales for GSK3$\pi$9+JNK3 and 61 four-constraint rationales for GSK3$o \sim \pi(\cdot \mid q)$0+JNK3+QED+SA.

Evaluation sampled 5,000 molecules and reported success, novelty, and diversity. On the dual-constraint Alzheimer’s task GSK3$o \sim \pi(\cdot \mid q)$1+JNK3, MolRGen achieved $o \sim \pi(\cdot \mid q)$2 success, $o \sim \pi(\cdot \mid q)$3 novelty, and $o \sim \pi(\cdot \mid q)$4 diversity, compared with REINVENT at $o \sim \pi(\cdot \mid q)$5, $o \sim \pi(\cdot \mid q)$6, and $o \sim \pi(\cdot \mid q)$7 (Jin et al., 2020). On the four-constraint task GSK3$o \sim \pi(\cdot \mid q)$8+JNK3+QED+SA, MolRGen reached $o \sim \pi(\cdot \mid q)$9 success, $r_i(q,o) \in [0,1]$0 novelty, and $r_i(q,o) \in [0,1]$1 diversity, while GCPN, GVAE-RL, and JT-VAE collapsed to less than $r_i(q,o) \in [0,1]$2 success. An ablation without rationales also failed under multi-constraint settings, which the summary interprets as evidence for the importance of interpretable substructures.

7. Relationship to conditional SMILES generation and pipeline integration

The LLamol work is explicitly presented as practical advice for integrating a generative transformer into a MolRGen pipeline (Dobberstein et al., 2023). LLamol is an 8-layer, approximately 15 million-parameter decoder-only Transformer derived from LLaMA 2, with masked multi-head self-attention using 8 heads, rotary positional embeddings, a SwiGLU feed-forward network with $r_i(q,o) \in [0,1]$3, RMSNorm, and 10% dropout. Its embedding dimension is $r_i(q,o) \in [0,1]$4, and it uses a vocabulary of 591 SMILES tokens.

Conditioning is implemented by prepending up to $r_i(q,o) \in [0,1]$5 context tokens to the SMILES input: three learned numeric-property embeddings and up to 50 SMILES tokens from a fragment. The generative objective models

$r_i(q,o) \in [0,1]$6

with standard autoregressive factorization, and the loss is mean cross-entropy over SMILES tokens only, excluding context tokens (Dobberstein et al., 2023).

A distinctive training procedure, Stochastic Context Learning, drops available conditions stochastically during each batch with $r_i(q,o) \in [0,1]$7, allowing the model to handle any subset of its conditions, including fully unconditional generation (Dobberstein et al., 2023). The training dataset, OrganiX13, contains approximately 13.1 million SMILES collected from ZINC 15, PubChemQC, ChEMBL, QM9, PC9, ZINC250k, RedDB, OPV, and CEP, with preprocessing that removes salts, molecules above 256 tokens, and parsing failures, while computing surrogate properties logP, scaled molecular weight $r_i(q,o) \in [0,1]$8, and SAScore via RDKit.

At temperature $r_i(q,o) \in [0,1]$9, LLamol reports approximately $$r(q,o) = \Bigl( \prod_{i=1}^n r_i(q,o) \Bigr)^{1/n},$$0 validity, approximately $$r(q,o) = \Bigl( \prod_{i=1}^n r_i(q,o) \Bigr)^{1/n},$$1 novelty, and approximately $$r(q,o) = \Bigl( \prod_{i=1}^n r_i(q,o) \Bigr)^{1/n},$$2 uniqueness for unconditional generation over 20 thousand samples (Dobberstein et al., 2023). In single-condition settings, the reported mean absolute deviations include $$r(q,o) = \Bigl( \prod_{i=1}^n r_i(q,o) \Bigr)^{1/n},$$3 for logP over $$r(q,o) = \Bigl( \prod_{i=1}^n r_i(q,o) \Bigr)^{1/n},$$4, $$r(q,o) = \Bigl( \prod_{i=1}^n r_i(q,o) \Bigr)^{1/n},$$5 for SAScore over $$r(q,o) = \Bigl( \prod_{i=1}^n r_i(q,o) \Bigr)^{1/n},$$6, and $$r(q,o) = \Bigl( \prod_{i=1}^n r_i(q,o) \Bigr)^{1/n},$$7 for MW over $$r(q,o) = \Bigl( \prod_{i=1}^n r_i(q,o) \Bigr)^{1/n},$$8. In multi-condition settings, pairwise combinations achieve validity at least $$r(q,o) = \Bigl( \prod_{i=1}^n r_i(q,o) \Bigr)^{1/n},$$9, novelty at least $q = (\rho, task, x, \sigma),$0, and uniqueness at least $q = (\rho, task, x, \sigma),$1, while token-sequence conditioning gives substructure match rates typically between $q = (\rho, task, x, \sigma),$2 and $q = (\rho, task, x, \sigma),$3, remaining above $q = (\rho, task, x, \sigma),$4 even under an additional numeric condition (Dobberstein et al., 2023).

Within the broader MolRGen landscape, LLamol represents a different design point from the 2026 reasoning benchmark and from the 2020 rationale-based graph generator. It is a unified conditional SMILES generator that supports unconditional, single-, and multi-property generation within one model. A plausible implication is that MolRGen, as a research area, now spans at least three complementary layers: benchmark-and-reward settings for reasoning LLMs, interpretable substructure-based graph generation, and flexible conditional SMILES generation.