Optimization of Molecules via Deep Reinforcement Learning (1810.08678v3)

Abstract: We present a framework, which we call Molecule Deep $Q$-Networks (MolDQN), for molecule optimization by combining domain knowledge of chemistry and state-of-the-art reinforcement learning techniques (double $Q$-learning and randomized value functions). We directly define modifications on molecules, thereby ensuring 100\% chemical validity. Further, we operate without pre-training on any dataset to avoid possible bias from the choice of that set. Inspired by problems faced during medicinal chemistry lead optimization, we extend our model with multi-objective reinforcement learning, which maximizes drug-likeness while maintaining similarity to the original molecule. We further show the path through chemical space to achieve optimization for a molecule to understand how the model works.

• The paper introduces MolDQN, a framework that employs value function learning for stable and sample-efficient molecular optimization.
• It demonstrates enhanced single-property and constrained optimization by learning from scratch without relying on pre-trained data.
• The method successfully integrates multi-objective reinforcement learning to balance drug-likeness with structural similarity in molecule design.

Optimization of Molecules via Deep Reinforcement Learning

The paper "Optimization of Molecules via Deep Reinforcement Learning" presents a framework known as Molecule Deep $Q$-Networks (MolDQN) for optimizing molecular structures. By leveraging advancements in reinforcement learning—specifically double $Q$-learning and randomized value functions—this approach integrates foundational chemistry knowledge to ensure chemically valid molecular modifications.

Key Contributions

MolDQN distinguishes itself from prior work through three primary aspects:

1. Value Function Learning: Unlike the majority, which utilize policy gradient methods, MolDQN employs value function learning. This method typically offers more stability and sample efficiency in applicable scenarios.
2. Learning From Scratch: Eschewing pre-training on existing datasets avoids biases, potentially leading to a broader exploration of chemical space and discovery of molecules with enhanced properties.
3. Multi-Objective Optimization: By integrating multi-objective reinforcement learning, MolDQN allows users to prioritize various objectives, such as maximizing drug-likeness while maintaining structural similarity to target molecules.

Methodology

MolDQN models molecular modification as a Markov decision process (MDP), with states representing molecular structures and actions including atom additions, bond modifications, and chemical rule enforcement. Each molecule undergoes valid modifications, ensuring generated outputs are chemically valid.

Key implementation details include:

• State Space Definition: The state includes the molecule and the number of modification steps.
• Action Space Control: Only chemically valid actions are allowed, systematized by domain-specific constraints.
• Reward Structure: Rewards are based on the desired molecular properties.

Results

The model's performance was evaluated against established methods like JT-VAE, ORGAN, and GCPN on molecule property optimization tasks, including penalized logP and QED. Noteworthy findings include:

• Single Property Optimization: MolDQN showed superior results in optimizing penalized logP and equivalent prowess for QED compared to others. The method notably managed to achieve these without pre-training, emphasizing its robust exploration capability.
• Constrained Optimization: The approach excelled at improving molecular properties while maintaining high structural similarity, yielding statistically significant improvements over GCPN.
• Multi-Objective Optimization: MolDQN demonstrated flexibility by optimizing multiple objectives concurrently, effectively balancing property enhancement and structural fidelity.

Implications and Future Work

MolDQN presents a promising technique for molecular design, particularly in drug discovery. Its ability to learn effectively from scratch, coupled with a multi-objective focus, could significantly streamline molecular optimization tasks currently requiring substantial time and resources.

Future research could explore more advanced function approximators and refine hyperparameters to further enhance model performance. Additionally, the integration of experimentally verified predictive models could substantially elevate MolDQN's utility in practical applications.

Thus, the paper contributes a significant optimization tool, expanding the methodological toolkit available to chemists and machine learning practitioners involved in molecular design.