Mol-RL: Molecular Reinforcement Learning

• Mol-RL is a family of methods that integrates reinforcement learning, relational learning, and representation learning for molecular design and optimization.
• It spans approaches from sequence-based SMILES generation and graph-editing to quantum-circuit design and simulation-guided transition state discovery.
• Applications include de novo molecule generation, lead optimization with constrained oracle budgets, and structure-aware editing using retrieval and chemical priors.

Mol-RL is an umbrella label for several research programs centered on molecules and sequential decision-making, but the term is not used uniformly. In its most common sense, it denotes molecular reinforcement learning: an agent generates, edits, or optimizes molecules under oracle feedback, often with objectives such as target-property satisfaction, structural preservation, docking affinity, or sample-efficient lead optimization. In other papers, closely related labels denote molecular relational learning over molecule pairs or reaction-aware molecule representation learning rather than reinforcement learning in the strict sense. Taken together, the literature presents Mol-RL as a family of methods that ranges from SMILES-level policy gradients and graph-edit optimization to 3D atom placement, transition-path sampling, quantum-circuit design for molecular Hamiltonians, and modular LLM systems for molecule-pair reasoning (Thomas et al., 27 Jan 2025, Li et al., 15 Jan 2026, Wang et al., 14 Apr 2026, Simm et al., 2020, Zhang et al., 2020, Chen et al., 1 Jun 2025).

1. Terminological scope and canonical formulations

The most stable usage of Mol-RL refers to reinforcement learning applied to molecular design. In this setting, the state is typically a partial molecule, a current edited molecule, or a task-conditioned optimization context; the action is a token, fragment edit, graph edit, atom placement, or whole-molecule proposal; and the reward is computed from chemistry oracles, docking engines, RDKit descriptors, or quantum chemistry. Representative formulations include sequence-generation MDPs for chemical LLMs, constrained molecular editing, phenotype-guided molecular generation, and sample-efficient lead optimization (Thomas et al., 27 Jan 2025, Zhuang et al., 26 May 2025, Guo et al., 25 Sep 2025, Wang et al., 14 Apr 2026).

A second usage broadens the label beyond reinforcement learning proper. ModuLM is proposed as a unified framework for LLM-based Molecular Relational Learning and is explicitly introduced as “MRL / Mol-RL,” with tasks defined over molecular pairs such as drug-drug interaction, solute-solvent interaction, and chromophore-solvent interaction (Chen et al., 1 Jun 2025). MolR goes further and states that the query “Mol-RL” should be interpreted there as molecule representation learning via reaction-aware learning, with no reinforcement-learning component (Wang et al., 2021). This suggests that “Mol-RL” is semantically overloaded across subcommunities.

Sense of “Mol-RL”	Optimized object	Representative papers
Molecular reinforcement learning	Molecules, edits, or trajectories	(Thomas et al., 27 Jan 2025, Li et al., 15 Jan 2026, Wang et al., 14 Apr 2026)
Molecular relational learning	Molecule-pair relations	(Chen et al., 1 Jun 2025)
Molecule representation learning	Reaction-aware embeddings	(Wang et al., 2021)

For reinforcement-learning papers, the dominant problem families are de novo generation, source-conditioned editing, exact target matching, and oracle-budgeted lead optimization. For the broader usages, the common thread is still molecule-centered learning under structured inductive bias, but the training signal is no longer necessarily an RL reward.

2. Sequence and graph formulations for molecular generation and editing

A canonical Mol-RL formulation treats molecular generation as sequence policy optimization. In the REINFORCE study on chemical LLMs, the state is a partially built SMILES prefix, the action is the next token, the trajectory is the completed molecular string, and the return is effectively terminal-only molecular reward. That paper systematically studies baselines, hill-climbing, experience replay, KL regularization, entropy regularization, diversity filters, and random network distillation, and argues that REINFORCE remains a strong default for sequence-based molecular design when the policy is pretrained and rewards are terminal (Thomas et al., 27 Jan 2025). RL-MolGAN stays within the same sequence-policy family but embeds it in a discrete Transformer GAN: the generator is a policy over SMILES tokens, the reward mixes discriminator realism with property scores for QED, logP, and SA, and Monte Carlo tree search provides rollout-based intermediate value estimates (Li et al., 17 Mar 2025).

Several later systems move from unconstrained generation toward structured optimization. ExMolRL fine-tunes a phenotype-conditioned SMILES generator with policy gradient, using a scalarized reward $Reward(s)=Dock(s)\times QED(s)$, together with ranking loss, prior-likelihood regularization, and entropy maximization. Its central claim is that a phenotype-guided prior can preserve alignment with desired transcriptional effects while RL adds target specificity through docking rewards (Guo et al., 25 Sep 2025). Token-Mol uses a token-only transformer that encodes 2D structure, 3D torsion information, and scalar properties in one autoregressive representation, but applies RL only to pocket-based molecular generation. Its RL stage is vanilla REINFORCE over completed token sequences, with reward based on Vina thresholding, invalidity handling, and a QED bonus (Wang et al., 2024).

Graph-based editing introduces a different control regime. MolEditRL defines molecular editing as source-conditioned generation of a target graph under natural-language instructions and similarity constraints. Stage 1 is a discrete graph diffusion editor pretrained to reconstruct target molecules conditioned on source structure and instruction; Stage 2 is an RL fine-tuning stage that treats the reverse denoising process as an MDP, uses terminal reward from RDKit/TDC property oracles, and regularizes the policy toward the pretrained editor with a KL term. The result is a source-preserving, structure-aware editing policy rather than unconstrained generation from scratch (Zhuang et al., 26 May 2025).

Across these papers, Mol-RL is not merely reward maximization over chemical desirability. It includes terminal-reward sequence policies, adversarially regularized generators, phenotype-conditioned priors, and source-conditioned graph editors. A recurring theme is that chemical validity and realism are rarely left to reward alone; they are stabilized through pretraining, priors, masking, or structure-aware parameterization.

3. Retrieval, memory, and LLM reasoning as Mol-RL control mechanisms

A major recent direction replaces free-form generation with explicit reasoning and local refinement. M$^4$olGen formulates inverse molecular design under explicit numeric constraints as a two-stage process. Stage I is a retrieval-augmented, multi-agent prototype generator that finds a molecule near the feasible region. Stage II is a fragment-level optimizer trained with GRPO that applies add, remove, and replace actions over BRICS fragments to minimize weighted distance to target properties, using the objective

$$E(m)=\sum_{i \in \{\mathrm{QED},\mathrm{LogP},\mathrm{MW}\}} w_i\,|p_i(m)-p_{i,\mathrm{tgt}}|.$$

The system is evaluated on exact target matching for QED/LogP/MW and HOMO/LUMO, and the paper emphasizes that RL is used for local, controllable refinement rather than global search from scratch (Li et al., 15 Jan 2026).

MolMem reframes lead optimization as a multi-turn, memory-augmented decision process under a hard oracle budget. Its state includes the lead, optimization history, and optionally retrieved memory; its action is a full SMILES proposal; and it is trained with PPO and dense step-wise rewards. Two external memories are central: a Static Exemplar Memory built from 2.8M ChEMBL molecules for cold-start grounding, and an Evolving Skill Memory that distills successful edit trajectories into reusable textual strategies. Under a budget of 500 oracle calls per lead and a similarity threshold $\gamma=0.4$, the paper reports 90% success on single-property tasks and 52% on multi-property tasks (Wang et al., 14 Apr 2026).

Reference-guided Policy Optimization addresses a different failure mode in LLM-based molecular optimization. RePO starts from the observation that answer-only SFT collapses reasoning and that GRPO-style RLVR suffers from sparse feedback under similarity constraints. It therefore combines a GRPO-style exploration term over sampled reasoning-and-answer trajectories with an answer-level reference-guidance term conditioned on the model’s own sampled reasoning prefix: $\log \pi_\theta(m_{\mathrm{ref}}\mid q,t_i).$ The method does not supervise the intermediate reasoning tokens directly; instead, it uses gradient masking so that reference guidance applies only to the final answer. The paper reports consistent gains over SFT and GRPO baselines on $\text{SR} \times \text{Similarity}$ and argues that this balances exploration of new edits with exploitation of reference molecules (Li et al., 6 Mar 2026).

These systems mark a shift from “molecule as output string” to “molecule as target of explicit reasoning, retrieval, and controlled local improvement.” A plausible implication is that Mol-RL increasingly relies on auxiliary structures—retrieval sets, external memories, answer-level demonstrations, and fragment vocabularies—to make reward optimization tractable under sparse or expensive feedback.

4. Three-dimensional, quantum-mechanical, and simulation-grounded Mol-RL

A distinct branch of Mol-RL operates directly in continuous physical space. “Reinforcement Learning for Molecular Design Guided by Quantum Mechanics” formulates molecular construction in Cartesian coordinates. The state is $s_t=(\mathcal{C}_t,\beta_t)$, where $\mathcal{C}_t$ is the current canvas of placed atoms and $\beta_t$ is the remaining bag; the action is $a_t=(e_t,x_t)$, a hybrid discrete-continuous choice of element and 3D position; and the reward is the negative interaction-energy increment

$^4$0

The environment, MolGym, includes single-bag, multi-bag, and solvation tasks, and uses PM6 via Sparrow as a fast semi-empirical quantum chemistry oracle (Simm et al., 2020).

RL$^4$1, or “Deep Reinforcement Learning of Transition States,” applies RL not to molecular generation but to rare-event molecular simulation. The method formulates transition-state discovery as a shooting game: a virtual player chooses a configuration $^4$2, launches paired trajectories with opposite momenta, and wins if one trajectory reaches reactant and the other reaches product. The value function $^4$3 is interpreted as transition-path probability, while the policy is a bias potential $^4$4 that shapes the shooting distribution

$^4$5

This lets RL concentrate sampling near the separatrix and use the learned value function as a mechanistic map for SN2, alanine dipeptide isomerization, and Claisen rearrangement (Zhang et al., 2020).

A further extension moves from molecular structures to molecular quantum simulation workflows. “Reinforcement learning of quantum circuit architectures for molecular potential energy curves” defines a geometry-conditioned MDP in which the state encodes the current wavefunction amplitudes, normalized construction step, and a Gaussian embedding of bond distance $^4$6; the action comprises a discrete gate choice and a continuous rotation angle; and Soft Actor-Critic learns a policy that maps molecular Hamiltonian instances to quantum circuits across an entire potential energy curve. The paper positions this as a molecular RL method because the optimized object is a geometry-dependent ansatz family for molecular ground states, not the molecule itself (Krumtünger et al., 20 Nov 2025).

These 3D and quantum-mechanical variants enlarge Mol-RL beyond graph or sequence generation. The optimized object may be a 3D structure, a shooting distribution in configuration space, or a circuit ansatz for a molecular Hamiltonian, but the common principle is still sequential control under molecularly meaningful rewards.

5. Relational and representation-learning senses of “Mol-RL”

In some work, Mol-RL refers not to reinforcement learning but to adjacent molecule-centered learning paradigms. ModuLM defines Molecular Relational Learning as learning over molecular pairs and presents a modular LLM framework that supports 1D strings, 2D graphs, and 3D conformations; 8 graph encoders; 11 conformation encoders; 7 interaction layers; 2 alignment modules; and 7 LLM backbones, enabling dynamic construction of over 50,000 model configurations. Its tasks are drug-drug interaction, solute-solvent interaction, and chromophore-solvent interaction, and its central claim is infrastructural: fair, modular comparison of LLM-based pairwise molecular models (Chen et al., 1 Jun 2025).

MolR explicitly treats “Mol-RL” as molecule representation learning and uses chemical reactions as supervision. Its central constraint is that the sum of reactant embeddings equals the sum of product embeddings: $^4$7 With GNN encoders and a contrastive minibatch objective, this reaction-aware equivalence induces a structured latent space and implicitly captures reaction templates. The paper reports strong gains on reaction prediction, molecule property prediction, and graph-edit-distance prediction, while emphasizing that the contribution is representation learning rather than reinforcement learning (Wang et al., 2021).

This terminological divergence matters because it changes what “RL” denotes: reward-based control in one case, relational learning or representation learning in another. A plausible implication is that encyclopedia usage should treat Mol-RL as a contested umbrella rather than a single settled acronym.

6. Evaluation criteria, recurrent design patterns, and recurrent limitations

Across the reinforcement-learning papers, evaluation almost always couples objective satisfaction to some form of structural or distributional control. Exact target matching in M$^4$8olGen is measured by per-property mean absolute error, aggregated error, diversity, and uniqueness rather than by unconstrained desirability optimization (Li et al., 15 Jan 2026). MolMem defines constrained success under a similarity threshold and a fixed 500-oracle-call budget, and reports Success Rate, Similarity, and Relative Improvement (Wang et al., 14 Apr 2026). RePO uses $^4$9 as its main metric because similarity-preserving optimization and task success must be balanced rather than optimized independently (Li et al., 6 Mar 2026). MolEditRL evaluates validity, $$E(m)=\sum_{i \in \{\mathrm{QED},\mathrm{LogP},\mathrm{MW}\}} w_i\,|p_i(m)-p_{i,\mathrm{tgt}}|.$$0, $$E(m)=\sum_{i \in \{\mathrm{QED},\mathrm{LogP},\mathrm{MW}\}} w_i\,|p_i(m)-p_{i,\mathrm{tgt}}|.$$1, and FCD, again tying reward optimization to structural fidelity (Zhuang et al., 26 May 2025).

A second recurring pattern is the coexistence of reward maximization with explicit regularization toward chemical priors. In sequence-based chemical LLMs, prior-preserving reward shaping and KL regularization are central tools for preventing over-exploitation and reward hacking (Thomas et al., 27 Jan 2025). ExMolRL uses prior-likelihood regularization and entropy maximization in addition to docking and QED reward (Guo et al., 25 Sep 2025). MolEditRL regularizes the RL-updated diffusion policy toward the pretrained editor with a KL term (Zhuang et al., 26 May 2025). M$$E(m)=\sum_{i \in \{\mathrm{QED},\mathrm{LogP},\mathrm{MW}\}} w_i\,|p_i(m)-p_{i,\mathrm{tgt}}|.$$2olGen bounds edit complexity primarily through hop budget and mentions optional similarity regularization, while MolMem uses exemplar retrieval, skill memory, and dense rewards to reduce wasteful exploration under expensive oracle feedback (Li et al., 15 Jan 2026, Wang et al., 14 Apr 2026).

A third theme is the prevalence of proxy oracles and the limitations they introduce. Token-Mol’s RL stage uses docking-derived Vina reward with a QED threshold bonus (Wang et al., 2024). ExMolRL relies on LeDock and QED, and explicitly notes the risk of reward hacking and the indirect nature of phenotype preservation through prior regularization rather than an explicit phenotype reward (Guo et al., 25 Sep 2025). MolEditRL uses RDKit/TDC-based property oracles and leaves the reward decomposition only partially specified (Zhuang et al., 26 May 2025). M$$E(m)=\sum_{i \in \{\mathrm{QED},\mathrm{LogP},\mathrm{MW}\}} w_i\,|p_i(m)-p_{i,\mathrm{tgt}}|.$$3olGen notes reliance on computed proxy properties and a narrow evaluated property set as limitations (Li et al., 15 Jan 2026). In simulation-grounded work, the same issue appears as a fidelity–cost trade-off: MolGym uses PM6 because DFT is too slow for RL-scale reward queries (Simm et al., 2020).

Taken together, these papers show that Mol-RL is not a single algorithmic template. It is a research area organized around a recurring set of tensions: exact objective control versus exploration, structural preservation versus aggressive optimization, proxy rewards versus physical fidelity, and one-shot generation versus multi-step reasoning, memory, or simulation. That diversity is the defining feature of the field as it appears in the recent literature.