Chemical Language Model Overview

Updated 19 December 2025

Chemical language models are Transformer-based neural sequence models that interpret and generate chemical structures using discrete representations like SMILES and SELFIES.
They leverage precise tokenization and canonicalization strategies to convert molecular data into consistent, valid inputs for deep learning architectures.
Applications include property prediction, inverse design, and chemical fingerprinting, accelerating research in molecular modeling and materials discovery.

A chemical LLM (CLM) is a neural sequence model, typically Transformer-based, designed to infer, represent, generate, or reason about molecules, polymers, and materials by mapping their discrete representations—such as SMILES, SELFIES, or 3D tokenizations—into contextual embeddings and output sequences. These models, by analogy to natural LLMs, treat chemical structures as languages with their own vocabulary, grammar, and semantics. They form the methodological basis for property prediction, inverse design, structure elucidation, cross-modal generation (e.g., text, image, coordinates), and protein or polymer engineering across chemistry and materials science.

1. Input Formalisms and Tokenization Strategies

Chemical LLMs operate over structured chemical notations, requiring highly controlled tokenization protocols to map molecules into suitable sequences for Transformer architectures. Several encoding schemes are prominent:

SMILES/PSMILES: Linear string representations of molecules or polymers; for example, polymers utilize “PSMILES” with specialized end-group markers [*]. Canonicalization, including repeated-unit collapse and unique ring formation, is critical to standardize representations and minimize notation-induced ambiguity. Tokenization can be character-based, atom-level (e.g., splitting into elemental symbols and bond tokens), or substructural (byte-pair encoding to capture common motifs) (Kuenneth et al., 2022, Lee et al., 2022).
SELFIES and PSELFIES: Grammar-constrained, invertible representations. SELFIES guarantee decodability into valid molecules, making them robust for unconstrained generative modeling, while PSELFIES extend this approach to polymers by marking cleavage points and converting structures for sequence-to-sequence modeling (Sahu et al., 21 Oct 2025, Flam-Shepherd et al., 2023).
3D Direct Representations: CLMs can operate on 3D structures by serializing coordinates (XYZ/PDB/CIF), fragments, or compressed run-length encodings. Tokens comprise atomic types and quantized spatial data, facilitating direct generative modeling of molecules, materials, and protein pockets in three dimensions, while handling coordinate precision and ensuring invertibility (Flam-Shepherd et al., 2023, Jiang et al., 14 Aug 2025).
Heterogeneous Embeddings: A growing trend is the simultaneous encoding of fragment sequences, molecular graphs, and conformations, producing composite vector representations via adaptive fusion, often optimized by Q-learning to mitigate bias and maximize downstream task performance (Lv et al., 30 Dec 2024).

Rigorous canonicalization and incorporation of stereochemistry are essential. Variational studies show that inconsistencies in SMILES linearization and chiral labeling propagate through encoder–decoder CLMs, altering latent representations and reducing prediction/reconstruction fidelity unless systematic data preprocessing is employed (Kikuchi et al., 11 May 2025).

2. Transformer Architectures and Learning Objectives

CLMs primarily employ Transformer encoder, decoder, or encoder–decoder architectures tuned for the unique challenges of chemical data:

Encoder-only (BERT-style): Used to construct dense chemical fingerprints via masked language modeling (MLM), in which a fraction of tokens are replaced with [MASK]. The model predicts missing values, and average-pooled last-layer embeddings yield property-agnostic fingerprints for downstream regression/classification tasks (Kuenneth et al., 2022, Ross et al., 2021).
Decoder-only (GPT-style/autoregressive): Suitable for sequence generation. Models are trained to maximize the likelihood of the current token conditioned on the preceding tokens (next-token prediction), often for tasks such as de novo molecule/protein generation, reaction prediction, or 3D atom-by-atom structure assembly (Flam-Shepherd et al., 2023, Flam-Shepherd et al., 2023).
Encoder–Decoder (T5/Seq2Seq): These architectures are designed for complex sequence-to-sequence tasks, including property-conditioned generation (input: target property values; output: molecule or polymer sequence) and cross-modal applications (image-to-text, SMILES-to-description) (Sahu et al., 21 Oct 2025, Lv et al., 30 Dec 2024, Livne et al., 2023).

Pretraining objectives include masked language modeling, span corruption, and next-token autoregression; some CLMs incorporate bilingual objectives to align natural language text and chemical tokens for joint reasoning (Lee et al., 2022, Edwards et al., 18 May 2025). For protein–ligand or multimodal tasks, cross-attention mechanisms integrate protein sequence embeddings (from a protein LLM) with ligand or molecule embeddings to drive target-aware generation (Singh et al., 20 Mar 2025, Jiang et al., 14 Aug 2025).

3. Applications: Fingerprinting, Property Prediction, and Inverse Design

CLMs are deployed across a wide spectrum of chemical informatics applications:

Chemical Fingerprinting: Models such as polyBERT use Transformer encodings of canonicalized polymer sequences to rapidly produce fixed-length fingerprints for downstream multitask regression (e.g., 29 polymer properties across ∼35,000 samples at R²=0.80), outperforming hand-engineered descriptors at over two orders of magnitude faster inference speeds (Kuenneth et al., 2022).
Property Prediction: SMI-TED289M, MoLFormer, nach0, and ChemLLM represent families of large-scale pre-trained encoder–decoder or encoder-only models that deliver state-of-the-art accuracy on molecular property benchmarks including quantum chemistry (QM9/QM8), ADMET, and solubility, both as frozen feature extractors and with end-to-end fine-tuning (Soares et al., 24 Jul 2024, Ross et al., 2021, Livne et al., 2023, Zhang et al., 10 Feb 2024).
Inverse Design and Generative Modeling: Decoders trained with property or protein conditioning (e.g., Chem42, polyT5, mCLM) enable de novo generation of molecules or polymers with user-defined functionality, structure, or binding affinity (Singh et al., 20 Mar 2025, Sahu et al., 21 Oct 2025, Edwards et al., 18 May 2025). CLMs can generate target-aware ligands with superior chemical validity, target-specificity, and binding affinity by fusing atomic-level molecule representations with sequence-encoded protein context (via cross-attention with Prot42 embeddings) (Singh et al., 20 Mar 2025).
Structural Elucidation: Encoder–decoder CLMs map high-dimensional spectroscopic data (IR, UV, NMR) directly to SMILES via vision transformer encoding and ChemBERTa-style decoding, achieving rapid, accurate end-to-end elucidation of organic compounds (Tan, 13 Oct 2024).
3D Model Generation: Both unconditional and protein-conditioned CLMs have been shown to produce valid, unique, and novel molecules, materials (e.g., perovskites), and protein binding site structures as raw atomic coordinate sequences, matching or exceeding graph-based generative methods (Flam-Shepherd et al., 2023, Jiang et al., 14 Aug 2025).
Cross-modal and Multimodal Learning: Heterogeneous encodings (HME) and multimodal LLMs (ChemMLLM) tightly integrate textual, graphical, and image modalities for captioning, structure generation, and caption-to-molecule/image tasks, outperforming previous approaches in both chemical and linguistic metrics (Lv et al., 30 Dec 2024, Tan et al., 22 May 2025).

4. Model Evaluation, Performance, and Benchmarking

CLM performance is assessed using a suite of both chemistry-specific and general benchmarking tasks:

Classification and Regression: Evaluation on MoleculeNet, ADMET, and quantum property tasks uses standard metrics (ROC-AUC, RMSE, MAE, R²). CLMs such as SMI-TED289M and MoLFormer deliver peak ROC-AUC over 91% (BBBP), RMSE=0.611 (ESOL), and MAE=1.32×10⁻³ (QM9), matching or exceeding leading GNNs (Soares et al., 24 Jul 2024, Ross et al., 2021).
Generative Benchmarks: Metrics include chemical validity, uniqueness, novelty, and distributional similarity (Fréchet ChemNet Distance, FCD; Earth Mover’s Distance for property histograms). Foundation fragment models (FragAtlas-62M) achieve 99.90% chemical validity and introduce >20% novel, practically relevant scaffolds (Ho et al., 23 Sep 2025).
Task Suites: Comprehensive benchmarks (ChemBench, ChemEval) encompass name conversion, property prediction, reaction prediction, retrosynthesis, yield, and multi-step reasoning. General-purpose CLMs, when instruction-tuned on chemistry datasets, can approach or match specialized models (e.g., ChemLLM vs GPT-4 on six of nine core chemistry tasks; ChemEval documents systematic trade-offs between broad and domain-specialized models) (Zhang et al., 10 Feb 2024, Huang et al., 21 Sep 2024).
Empirical Structural Validation: For protein and polymer generation, CLMs are validated against experimentally measured properties, secondary and tertiary structure predictions (AlphaFold2), and practical synthesis, confirming that generative outputs fall within the chemical space and property ranges observed in training sets (Flam-Shepherd et al., 2023, Sahu et al., 21 Oct 2025).

5. Chemical LLM Variants and Scaling

The field has developed numerous architectural variants tailored for scale, efficiency, and cross-domain generalization:

Parameter Scaling: Model families range from ∼40 M (FragAtlas-62M) to multi-billion parameter LLaMA/InternLM2 derivatives, using decoder-only, encoder-only, and encoder–decoder backbones (Ho et al., 23 Sep 2025, Zhang et al., 10 Feb 2024, Flam-Shepherd et al., 2023).
Mixture-of-Experts (MoE): SMI-TED8×289M combines multiple expert networks with sparse activation, yielding consistent performance gains in both classification and regression under fixed parameter budgets (Soares et al., 24 Jul 2024).
Parameter-efficient Tuning and Modularization: Adapter-based strategies (ChemLML) allow frozen text/molecule model backbones to be connected via lightweight cross-attention modules, achieving near-baseline performance at ∼5 M trainable parameters (Deng et al., 26 Oct 2024). Low-Rank Adaptation (LoRA), as in SmileyLlama, provides scalable, prompt-aligned fine-tuning with minimal overhead (Cavanagh et al., 3 Sep 2024).
Cross-modal and Multimodal Extensions: Joint frameworks (nach0, ChemMLLM, HME) integrate chemical language with both linguistic and multimodal inputs, enabling chemically guided captioning, text-conditioned structure/image generation, property-aware molecular design, and vice versa (Livne et al., 2023, Tan et al., 22 May 2025, Lv et al., 30 Dec 2024).
Block-based and Modular Tokenization: mCLM departs from atom-based tokenization, instead leveraging functional building blocks and GNN-based representations for code-switched bilingual modeling (text ↔ blocks), enabling single-block edit optimization under synthesis constraints (Edwards et al., 18 May 2025).

6. Best Practices, Limitations, and Future Directions

Robust CLM pipelines require careful attention to both representation and evaluation:

Preprocessing and Standardization: Curation of canonicalized, stereochemically rich input is critical; nearly half of surveyed studies failed to specify canonicalization steps, undermining reproducibility. Both grammatical and stereochemical SMILES inconsistency can significantly reduce translation and structure prediction accuracy, although property prediction is more resilient due to label-driven feature selection (Kikuchi et al., 11 May 2025).
Validity and Synthesis Constraints: Guaranteeing chemical and synthetic validity (SMILES/SELFIES-invertibility, block-based assembly) is central, with block-tokenization and SELFIES representations conferring deterministic guarantees (Sahu et al., 21 Oct 2025, Edwards et al., 18 May 2025, Flam-Shepherd et al., 2023). However, further constraints (synthetic accessibility, 3D feasibility) may require additional filters or 3D validation.
Multimodal/3D Reasoning: Directly tokenizing 3D structures is possible via reversible text encodings and discrete quantization, allowing CLMs to capture geometry alongside sequence; these models exhibit strong performance on both conformation generation and structure-based design (Jiang et al., 14 Aug 2025, Flam-Shepherd et al., 2023). Nonetheless, scaling to larger systems and integrating domain priors (e.g., SE(3) equivariance) remain open challenges.
Few-Shot and Compositional Generalization: Embedding spaces with compositional structure (e.g., group-contribution linearity) facilitate algebraic manipulation and efficient extrapolation to new molecular families (Soares et al., 24 Jul 2024). Cross-modal and agentic AI frameworks further democratize complex query pipelines, enabling natural language interaction with CLMs for design, prediction, and elucidation tasks (Sahu et al., 21 Oct 2025).
Benchmarking and Standardization: Emerging community benchmarks (ChemEval, ChemBench) provide multi-level and multi-task testbeds spanning nomenclature to synthesis planning, highlighting both strengths and persistent limitations of current models—especially on advanced mechanistic and multistep tasks (Huang et al., 21 Sep 2024).

Current trends emphasize the integration of chemical, physical, and textual modalities, parameter scaling combined with parameter-efficient adapters or MoE, and rigorous evaluation protocols. Limitations persist in handling long sequences, representing complex materials (e.g., MOFs, full proteins), and explicit 3D or multi-objective optimization. Developing architectures with built-in invariances, expanding multi-modal capacities, and advancing joint chemical–linguistic reasoning remain key directions.

Key References:

polyBERT for polymer property prediction and fingerprinting (Kuenneth et al., 2022)
Chem42 for protein-conditioned, target-aware molecular generation (Singh et al., 20 Mar 2025)
Atom-by-atom SELFIES generation for proteins and hybrid macromolecules (Flam-Shepherd et al., 2023)
MoLFormer, SMI-TED289M, mCLM for large-scale molecular property prediction and generation (Ross et al., 2021, Soares et al., 24 Jul 2024, Edwards et al., 18 May 2025)
nach0, ChemLLM, ChemEval for multi-task, instruction-tuned, and benchmarked chemical LLMs (Livne et al., 2023, Zhang et al., 10 Feb 2024, Huang et al., 21 Sep 2024)
ChemLML and HME for modular adapters, heterogeneous molecular encoding, and cross-modal modeling (Deng et al., 26 Oct 2024, Lv et al., 30 Dec 2024)
FragAtlas-62M for fragment-based generative modeling (Ho et al., 23 Sep 2025)
Chem3DLLM, ChemMLLM for 3D, multimodal, and image-based chemical representation and generation (Jiang et al., 14 Aug 2025, Tan et al., 22 May 2025)