GraphXForm: Graph transformer for computer-aided molecular design with application to extraction (2411.01667v1)

Abstract: Generative deep learning has become pivotal in molecular design for drug discovery and materials science. A widely used paradigm is to pretrain neural networks on string representations of molecules and fine-tune them using reinforcement learning on specific objectives. However, string-based models face challenges in ensuring chemical validity and enforcing structural constraints like the presence of specific substructures. We propose to instead combine graph-based molecular representations, which can naturally ensure chemical validity, with transformer architectures, which are highly expressive and capable of modeling long-range dependencies between atoms. Our approach iteratively modifies a molecular graph by adding atoms and bonds, which ensures chemical validity and facilitates the incorporation of structural constraints. We present GraphXForm, a decoder-only graph transformer architecture, which is pretrained on existing compounds and then fine-tuned using a new training algorithm that combines elements of the deep cross-entropy method with self-improvement learning from LLMing, allowing stable fine-tuning of deep transformers with many layers. We evaluate GraphXForm on two solvent design tasks for liquid-liquid extraction, showing that it outperforms four state-of-the-art molecular design techniques, while it can flexibly enforce structural constraints or initiate the design from existing molecular structures.

GraphXForm: Graph Transformer for Computer-Aided Molecular Design with Application to Extraction

The paper presents GraphXForm, a graph-based molecular design methodology leveraging the transformer architecture for tasks in chemical design, specifically focusing on solvent design for liquid-liquid extraction. The methodology addresses limitations found in string-based molecular representation, such as SMILES or SELFIES, in terms of ensuring chemical validity and embedding structural constraints during chemical compound generation.

Core Contributions

The primary contribution of this research is the integration of graph-based molecular representations with transformer architectures to enable the generation of molecular structures that inherently satisfy chemical validity. Unlike string-based models, the graph-based method allows for seamless incorporation of structural constraints from the beginning, ensuring more feasible and usable chemical designs.

Key features of GraphXForm include:

• Molecular Graph Iterative Modification: The paper proposes a graph transformer model, GraphXForm, which modifies the molecular graph by adding atoms and bonds iteratively. This approach naturally ensures the chemical validity of generated molecules.
• Decoder-Only Architecture with Novel Training: The design includes a decoder-only graph transformer architecture, pretrained on existing molecules and fine-tuned using a novel training algorithm combining self-improvement learning and elements from LLMing. This facilitates the stable fine-tuning of transformers, even for deep models with multiple layers.
• Empirical Evaluation on Solvent Design: The paper benchmarks GraphXForm against four state-of-the-art molecular design models across two solvent design tasks. The results indicate that GraphXForm not only outperforms these comparative techniques in solvent design but also demonstrates flexibility in enforcing structural constraints and leveraging existing molecular designs.

Evaluation and Numerical Results

GraphXForm has been applied to solvent design tasks, specifically targeting liquid-liquid extraction processes which are critical in industries like biotechnology. The performance of GraphXForm was evaluated against the objective functions based on activity coefficients at infinite dilution for two distinct tasks: the separation of isobutanol (IBA) from water and a process involving 3,5-dimethoxybenzaldehyde (DMBA) and (R)-3,3’,5,5’-tetramethoxy-benzoin (TMB).

Results from the experiments show that GraphXForm consistently outperforms existing methods in both tasks, achieving higher maximal and mean objective values across multiple runs and significantly improving the capability of enforcing structural constraints. Notably, GraphXForm derived more chemically feasible solvent structures, accommodating specified constraints such as ring sizes and bond types.

Theoretical and Practical Implications

The theoretical implications center around the ability to effectively combine transformer architectures with graph-based representations in deep learning. This approach not only broadens the application scope of transformers in generative tasks but also ensures chemical design validity, which is often challenging with conventional string-based methods.

On the practical side, GraphXForm opens avenues for more efficient and chemically valid molecular design processes in fields like drug discovery and material science. The flexibility to impose structural constraints and initiate designs from preexisting structures ushers in a user-friendly and adaptive design framework, enhancing its utility in real-world chemical engineering applications.

Conclusion and Future Directions

GraphXForm showcases the successful blending of deep learning methodologies with chemical design needs, underlining the applicability of graph-former models in generating viable chemical structures. It demonstrates that operating directly on molecular graphs revolutionizes the flexibility and validity of molecule-generation processes. Future developments could expand the model’s capabilities by incorporating additional chemical elements and states, thus broadening its application. Furthermore, integrating GraphXForm into larger LLMs can facilitate its translation into a more intuitive user interface for constraint specification, thus streamlining the workflow of chemical researchers and engineers.