Rethinking Molecular Text Representations for LLMs: An Empirical Study

Abstract: LLMs are increasingly used for molecular tasks, but it remains unclear which molecular representation to use. We present a systematic benchmark evaluating LLM molecular competence across nine representations and eight chemical tasks. We benchmark 16 LLMs across five model families, including reasoning and non-reasoning variants, chemistry-specialized LLMs, and closed frontier models. Performance is strongly representation-dependent and no single representation wins across tasks, though CML is the best, followed by MolJSON, InChI, and then canonical SMILES. Explicit structured text representations (CML and MolJSON) dominate structural tasks; IUPAC dominates semantic tasks, winning molecule retrieval for all 16 LLMs; and SMILES variants are rarely optimal despite their prevalence in pretraining. Chemistry-specialized models perform well with SMILES at the cost of large degradations with structured text representations, suggesting SMILES-only evaluation rewards specialization that does not generalize. Using LLM-as-a-judge, we find that IUPAC produces the highest fraction of correct molecule generations. A mechanistic study via tokenization audits, linear probes and attention shows that representations are encoded differently inside the model; for example, structured representations require higher attention across the molecular span. Our results argue against representation-invariant evaluation and motivate task-aware representation routing for LLM-based chemistry.

• The paper demonstrates that LLM performance is highly sensitive to the chosen molecular text representation, with no single encoding universally optimal across tasks.
• The analysis shows that structured formats like CML, MolJSON, and IUPAC names excel in specific tasks such as atom counting and molecule retrieval compared to traditional SMILES.
• The study reveals that domain-adaptive post-training on SMILES may cause over-specialization and catastrophic forgetting, underscoring the need for multi-representation benchmarks.

Empirical Analysis of Molecular Text Representations for LLMs

Problem Motivation and Benchmark Construction

This paper provides a comprehensive evaluation of the dependence of LLM performance on the textual representation of molecules across a wide spectrum of chemical tasks. The core motivation arises from the observation that convention, rather than empirical evidence, has dictated representation choice in the molecular AI literature—typically favoring the SMILES line notation—despite the existence of numerous encodings (e.g., SMILES variants, InChI, SELFIES, IUPAC names, CML, MolJSON). The authors introduce MolRepBench: a benchmark spanning eight tasks (comprising structural, semantic, retrieval, and generation objectives) and nine molecular string representations. The evaluation encompasses 16 LLMs, including open-weight, reasoning-ablated, chemistry-specialized, and frontier closed models.

Experimental Design

The investigation is executed over eight tasks, each selected to stress distinct cognitive and generative demands on LLMs. These tasks include atom counting, functional group identification, physicochemical property estimation, molecule retrieval, isomer and tautomer discrimination, protonation state recognition, and caption-to-molecule generation. Each task is paired with carefully curated molecules from ChEBI-20 and ZINC250K under strict measures to minimize pretrain and test contamination. The key feature is the cross-product of tasks, models, and representations, allowing for robust measurement of the interaction between these axes using rigorous evaluation metrics (e.g., exact-match, macro-F1, Spearman’s ρ, Tanimoto similarity, FCD).

Representation Dependence: Quantitative Outcomes

The central finding is that LLM performance is far from representation-invariant. No representation is universally optimal; the efficacy of a representation is intrinsically task-dependent.

• Structured representations (CML, MolJSON) outperform compact notations and natural language formalisms on tasks requiring explicit graph or atom-wise enumeration (e.g., atom counting, isomer and tautomer discrimination). CML achieves the highest aggregate win count, followed by MolJSON and InChI.
• IUPAC names decisively dominate molecule retrieval and caption-to-molecule generation tasks. Their morphemic alignment with compositional descriptions yields stronger semantic and generative alignment, as evidenced by the highest fraction of correct molecules in LLM-as-judge evaluations.

Figure 1: Proportion of error modes in molecule generation across various representations as determined by Gemini 3 Flash.

Figure 2: Gemini 3 Flash's scoring (from 1 to 5) across representations in the molecule generation task, based on faithfulness, chemical reasoning, and overall quality.

• SMILES variants, while abundant in pretraining corpora and traditionally used in molecular AI, are rarely optimal outside a small subset of tasks. Canonical SMILES achieves high performance only in property regression and some generation settings, and even there, validity does not guarantee topological correctness or semantic fidelity.
• For protonation state recognition, only SMILES and InChI, which explicitly encode charge, enable robust discrimination, consistent with their syntax.

Impact of Specialty Model Post-Training

Evaluation of chemistry-specialized LLMs (e.g., ChemDFM variants, Ether0) reveals a specificity-generalizability trade-off. These models show strong improvements on SMILES-based tasks, particularly generation (e.g., ChemDFM-R-14B yields a 0.98 validity rate and leading Tanimoto/FCD scores on canonical SMILES), but their performance collapses with explicit, structured, or less-seen representations (e.g., CML, MolJSON, InChI).

This supports the conclusion that domain-adaptive post-training on SMILES induces catastrophic forgetting for other representations, and that reporting only SMILES-based evaluation results systematically overstates the models’ molecular generalization.

Generative Evaluation and Error Analysis

Caption-to-molecule generation most starkly differentiates representations: whereas compact SMILES and structured representations are prone to syntax errors and hallucinations, IUPAC delivers the highest rate of coherent, faithful, and correct outputs, as adjudicated by the Gemini 3 Flash LLM-judge.

LLM-judging exposes systematic error modes—stereochemistry mistakes, substituent loss, scaffold confusion, and hallucination—which are radically amplified with SELFIES and DeepSMILES. Notably, reasoning-tuned models consistently dampen hallucination rates across all representations (e.g., OLMo-3.1-32B-Think halves hallucination compared to its non-reasoning counterpart).

Figure 3: Reasoning models reduce hallucination.

Mechanistic Interpretability: Tokenization, Probing, and Attention

Mechanistic analysis with Qwen3-4B provides a structural explanation for empirical trends:

• Tokenization audit reveals >4× variability in token length across representations (SMILES < IUPAC < SELFIES < InChI < CML/MolJSON), with explicit formats imposing extreme sequence length penalties.
• Linear probing of transformer activations recapitulates representation effects: while pre-trained SMILES and MolJSON internally encode molecular features most linearly, structured representations (MolJSON, CML) invoke higher molecule-spanning attention, highlighting their requirement for persistent aggregation over long contexts.

Figure 4: Token count distribution across 8 representations using Qwen3's tokenizer.

Figure 5: Linear probing performance of Qwen3-4B across all representations for molecular weight, log P, and TPSA prediction.

Figure 6: Attention between the last token and the molecule.

Figure 7: Attention within the molecule.

Theoretical and Practical Implications

This study provides compelling evidence against the practice of representation-invariant evaluation for chemistry LLMs. The non-trivial interaction between representation, model architecture, and chemical task means the choice of molecular text encoding becomes an algorithmic ingredient in downstream applications. Moreover, over-specialization (e.g., SMILES-centric fine-tuning) hinders model robustness and generalization.

Practical implications include:

• The need for task-aware representation routing infrastructure for molecular LLM pipelines.
• Adoption of heterogeneous evaluation reporting (multi-representation benchmarks) as standard.
• The necessity of LLM-as-judge or similarly nuanced qualitative error analysis to diagnose mode collapse, proxy task solutions, and hallucinations.
• Caution in deploying domain-specialized models for open-world molecular tasks, particularly those requiring interoperation between human, natural-language, and graph-based formats.

Outlook and Future Directions

The findings motivate multiple future research vectors: pretraining strategies with balanced exposure to multiple representations; fusion architectures capable of explicit cross-representation reasoning; refined interpretability tools for decoding which chemical signals are (or are not) learned by LLMs; and extended evaluation to multimodal (text/graph/3D) molecular data. The results also have direct relevance for medicinal chemistry, automated molecule design, and the construction of reliable AI–human interfaces in the chemical sciences.

Conclusion

The empirical evidence presented in this benchmark decisively demonstrates that LLM-driven chemical reasoning and generation is sensitive to both model training protocol and the surface molecular representation. The downstream choice of molecular string encoding cannot be decoupled from system performance and generalization; explicit consideration and reporting of representation effects is required for rigor in both research and deployment of molecular LLMs.