SMI-Editor: Edit-based SMILES Language Model with Fragment-level Supervision (2412.05569v2)

Abstract: SMILES, a crucial textual representation of molecular structures, has garnered significant attention as a foundation for pre-trained LLMs (LMs). However, most existing pre-trained SMILES LMs focus solely on the single-token level supervision during pre-training, failing to fully leverage the substructural information of molecules. This limitation makes the pre-training task overly simplistic, preventing the models from capturing richer molecular semantic information. Moreover, during pre-training, these SMILES LMs only process corrupted SMILES inputs, never encountering any valid SMILES, which leads to a train-inference mismatch. To address these challenges, we propose SMI-Editor, a novel edit-based pre-trained SMILES LM. SMI-Editor disrupts substructures within a molecule at random and feeds the resulting SMILES back into the model, which then attempts to restore the original SMILES through an editing process. This approach not only introduces fragment-level training signals, but also enables the use of valid SMILES as inputs, allowing the model to learn how to reconstruct complete molecules from these incomplete structures. As a result, the model demonstrates improved scalability and an enhanced ability to capture fragment-level molecular information. Experimental results show that SMI-Editor achieves state-of-the-art performance across multiple downstream molecular tasks, and even outperforming several 3D molecular representation models.

• The paper introduces an edit-based framework that replaces token-level SMILES prediction with fragment-level supervision to address key limitations in molecular modeling.
• It leverages chemical substructures using BRICS and RMCF algorithms to generate meaningful training distortions for improved learning.
• Experimental results demonstrate state-of-the-art performance in molecular property prediction, highlighting enhanced scalability and semantic understanding.

SMI-EDITOR: An Edit-Based SMILES LLM with Fragment-Level Supervision

This essay examines the innovative methodology presented in the discussed paper, which addresses fundamental limitations in the pre-training of SMILES-based LLMs (LMs). SMILES, or Simplified Molecular Input Line Entry System, is a text representation for describing the structure of chemical molecules. The prevalent practice has been to apply natural language processing strategies such as Masked LLMing (MLM), which have demonstrated effectiveness in textual data but exhibit significant shortcomings when applied to SMILES.

Current Challenges in SMILES LLMs

The research identifies three primary issues facing traditional SMILES LMs:

1. Individual Token Focus: Most models predominantly predict isolated tokens within broken SMILES sequences. This approach restricts the model's ability to understand complex molecular semantics that extend beyond simple atom or bond-level tokens.
2. Rapid Saturation: Due to the simplicity of predicting isolated tokens, most SMILES LMs quickly reach a high level of prediction accuracy, limiting their capacity to generalize or improve significantly with additional training.
3. Train-Inference Mismatch: Models are often trained on corrupted SMILES strings containing masked tokens, which do not reflect valid SMILES encountered during inference, leading to performance discrepancies.

The SMI-EDITOR Approach

To overcome these challenges, the authors propose SMI-EDITOR, an edit-based SMILES LM featuring fragment-level supervision. This approach involves deliberately distorting molecules by removing substructures and requiring the model to predict and reconstruct those missing parts.

Key Contributions and Methodologies

1. Fragment-Level Supervision: Instead of focusing on token prediction, SMI-EDITOR introduces the concept of fragment-level training. This involves using chemically meaningful substructures instead of isolated tokens as the primary unit of learning. The model harnesses expert knowledge to fragment molecules and subsequently distorts them for learning tasks, drawing on BRICS and RMCF algorithms for effective fragmentation.
2. Edit-Based Pre-Training: By using an edit-based framework, SMI-EDITOR operates with intact SMILES inputs. It uses a learning process centered on predicting the necessary editorial changes (deletion and insertion) needed to reconcile a given incomplete SMILES with its original structure, enabling the model to effectively mitigate the train-inference mismatch.
3. Superior Scalability and Performance: Comprehensive experimental results indicate that SMI-EDITOR outperforms existing SMILES LMs and even several 3D molecular representation models across a variety of molecular property prediction tasks. Notably, it achieves state-of-the-art performance, emphasizing its utility and efficacy in capturing the intricacies of molecular data.

Implications and Future Directions

SMI-EDITOR's approach to molecular modeling represents a meaningful shift from traditional token-based models, illustrating that fragments—representing functional groups and substructures—hold the key to richer semantic understanding in molecular data. Future research could explore expanding this framework to encompass additional data types such as 3D conformational information, potentially integrating SMI-EDITOR with other modalities to address even more complex molecular tasks.

Moreover, the edit-based framework promises broader applicability in sequence-based modeling beyond SMILES, such as textual data with latent hierarchical structures. Future developments could aim at improving the alignment between the molecular representation and its topology, optimizing computation for larger datasets, and investigating reinforcement learning techniques for better fragment selection and assembly in complex molecular configurations.

In conclusion, SMI-EDITOR's introduction of fragment-level supervision in SMILES modeling is a promising advancement that suggests substantial improvements in the performance and utility of LMs in chemical informatics. This paradigm could set a new standard for the field, encouraging the exploration of more semantically dense neural architectures that recognize the intricacies of molecular structure.