- The paper introduces MolJSON, a structured JSON encoding of molecular graphs, and benchmarks it against SMILES, SELFIES, InChI, IUPAC names, and MOL V2000 across 78,045 questions.
- GPT-5 reached 95.3% accuracy on constrained generation and 98.5% on shortest-path reasoning with MolJSON, compared with 64.0% and 92.2% using SMILES.
- MolJSON reduced GPT-5 output length by about 1.8× versus SMILES and remained robust on fused-ring molecules, but its support for stereochemistry and broader chemistry remains untested.
Motivation and scope
LLM-based systems for chemistry require a text serialisation of molecular graphs, and prior work has predominantly used SMILES strings or IUPAC names without systematically justifying that choice. This paper, by Runcie, Imrie, and Deane (University of Oxford), addresses this gap in two ways: it introduces MolJSON, a structured JSON schema for molecular graphs designed specifically for LLM input and output, and it presents a systematic benchmark comparing MolJSON against five established formats—SMILES, SELFIES, InChI, IUPAC names, and MOL V2000. The central claim is strong: the choice of molecular representation materially determines LLM performance on structure-centric tasks, and existing cheminformatics formats may currently be the limiting factor for LLM-based chemistry systems.
The MolJSON representation
MolJSON encodes a molecule as two explicit arrays: an "atoms" array of unique identifiers paired with element symbols, and a "bonds" array of atom-identifier pairs with bond orders (including 1.5 for aromatic bonds). Two optional sparse fields—"charges" for non-zero formal charges and "aromatic_n_h" for explicit hydrogens on aromatic nitrogens—handle valence edge cases; all other hydrogens are implicit under standard valence rules. Molecular identity is defined directly by the atom and bond sets, independent of ordering, which eliminates both the graph linearisation required by SMILES/SELFIES and the rule-based nomenclature mapping required by IUPAC names. A practical advantage is native compatibility with the structured output modes now offered by major LLM APIs, which matters for agentic pipelines requiring machine-parseable responses.
Benchmark design
The evaluation used 78,045 algorithmically generated questions across three tasks, answered by GPT-5-nano, GPT-5-mini, GPT-5, and Claude Haiku 4.5 (224,055 total responses):
- Translation: pairwise conversion between representations, evaluated by canonical SMILES equivalence via RDKit (with OPSIN for IUPAC outputs), using molecules drawn from PubChem deposits between October and December 2025 to reduce training-data contamination risk.
- Shortest-path reasoning: counting bonds along the shortest path between two halogen atoms, adapted from ChemIQ, isolating input-representation quality.
- Constrained generation: producing a molecule satisfying constraints on halogen placement, inter-halogen shortest paths, ring counts, sizes, and topology, isolating output-representation quality.
This design deliberately isolates structural manipulation from applied chemistry knowledge; the authors acknowledge it does not evaluate downstream chemistry performance, though they argue the tasks assess prerequisite skills.
Headline results
Across all models and tasks, MolJSON consistently outperformed SMILES and IUPAC as both input and output format:
| Task |
Format |
GPT-5 accuracy |
| IUPAC → output |
SMILES |
43.7% |
| IUPAC → output |
MolJSON |
71.0% |
| Constrained generation |
SMILES |
64.0% |
| Constrained generation |
IUPAC |
76.3% |
| Constrained generation |
MolJSON |
95.3% |
| Shortest-path (input) |
IUPAC |
82.7% |
| Shortest-path (input) |
SMILES |
92.2% |
| Shortest-path (input) |
MolJSON |
98.5% |
Notably, translation accuracy depended primarily on the output representation: translations into MolJSON achieved roughly twice the accuracy of translations into SMILES or IUPAC. On shortest-path tasks, MolJSON was also ~1.8× more token-efficient than SMILES (1,021 vs. 1,854 average output tokens for GPT-5), consistent with the model avoiding an error-prone implicit graph-parsing step. These gains were achieved without any fine-tuning on MolJSON, despite SMILES and IUPAC being ubiquitous in training corpora—a result that implies these legacy formats are intrinsically misaligned with current LLMs rather than merely under-trained.
Error analysis
Systematic failure modes correlated with molecular size and ring complexity. For acyclic molecules, accuracy declined with heavy-atom count when outputting SMILES or IUPAC but remained robust for MolJSON, indicating parsing is reliable while generation is not. Fused ring systems emerged as a dominant failure mode for IUPAC: on shortest-path questions with fused systems, GPT-5 accuracy dropped from 96.9% (non-fused) to 66.2% with IUPAC input, whereas MolJSON held at 98.5% in both regimes. In constrained generation, GPT-5 produced fused bicyclic molecules at 91.5% accuracy with MolJSON versus 41.5% with SMILES. SMILES outputs also produced the highest proportion of syntactically invalid molecules among the three focused formats.
Among the secondary formats, MOL V2000 showed strongly asymmetric behaviour (81.7% to MolJSON, largely trivial reformatting, versus 22.1% in reverse, dominated by counts-line errors); SELFIES suffered from refusals and malformed tokens; and InChI generation was essentially non-functional, with only 36 valid strings produced out of 2,100 attempts, none correct.
A mechanistic hypothesis
An appendix analysis offers a possible explanation for MolJSON's output advantage: GPT-5 spontaneously chose rich, semantically meaningful atom identifiers (e.g., "C_acyl", "O_carb") in roughly 22% of IUPAC-to-MolJSON translations, behaving as functional "graph tokens" within chain-of-thought reasoning. The permissive identifier scheme may allow the model to externalise intermediate graph structure directly, whereas SMILES and IUPAC force serialisation through traversal or nomenclature rules. This hypothesis is plausible but untested—the paper does not establish causality between identifier richness and accuracy.
Limitations and open questions
Several caveats bear directly on the conclusions. First, the benchmark excludes stereochemistry, salts, isotopes, radicals, and inorganic elements, so MolJSON's advantage on more chemically diverse inputs is unestablished; the schema itself does not yet support stereochemistry or coordinates. Second, contamination cannot be fully excluded: the PubChem date filter postdates only the GPT-5 releases, not other data sources. Third, the evaluation covers four proprietary models at low reasoning effort; whether the findings generalise to open-weight models, higher reasoning budgets, or fine-tuned chemistry models remains open. Fourth, the tasks isolate structural manipulation rather than applied chemistry, and the claim that identified limitations will persist downstream is an expectation, not a measurement. Finally, the authors themselves frame the schema as provisional, noting design choices such as directed "source"/"target" bond keys that may be suboptimal for undirected chemical bonds.
Conclusion
This paper provides systematic evidence that molecular representation choice is a first-order determinant of LLM performance on structure-centric chemistry tasks. MolJSON, an explicit JSON-encoded molecular graph, outperformed SMILES, IUPAC, and four other established formats across translation, graph reasoning, and constrained generation—despite lacking any presence in model training data—and was markedly more robust to size- and ring-complexity-dependent failure modes. The results argue that LLM-based chemistry systems should adopt explicit graph schemas over legacy serialisations, and leave open how such schemas should be extended to stereochemistry and validated on applied chemistry workloads.