SMILES-Specific Vocabulary

• SMILES-specific vocabulary is the set of tokens used to represent molecular structures via SMILES, balancing chemical granularity and full coverage.
• Tokenization strategies, including fixed-length, atom-wise, BPE, and glyph-based methods, offer complementary trade-offs in chemical interpretability and model efficiency.
• Integrating these vocabularies into machine learning models improves property prediction and molecule generation by reducing tokenization errors and ensuring comprehensive chemical representation.

SMILES-specific vocabulary refers to the set of string tokens or subunits used by computational models to represent, parse, and learn from molecules encoded as SMILES (Simplified Molecular Input Line Entry System) strings. Constructing an effective SMILES vocabulary is a central topic in chemoinformatics, molecular machine learning, and chemical language modeling, as it directly controls the coherence and informativeness of downstream molecular representations, the capacity to capture chemical substructures, and ultimately model performance on property prediction, synthesis planning, and molecule generation tasks.

1. Tokenization Strategies for SMILES

Several tokenization paradigms have emerged for segmenting SMILES strings into syntactic or chemically meaningful units:

• Fixed-length and bracket-aware tokenization: The “Learning to SMILE(S)” methodology separates bracketed groups (e.g., [Na+]) as atomic tokens. All other substrings are split into overlapping 2-character tokens (“N(”, “(c”, etc.), providing local chemical context but minimal explicit chemical semantics. No explicit distinction is made between single- and two-character atoms or other SMILES constructs (e.g., bonding or ring closures), and special tokens such as [UNK] or [PAD] are absent (Jastrzębski et al., 2016).
• Atom-wise closed vocabulary: This approach, used by ChemBERTa-SmilesTokenizer and similar models, treats each bracketed atom as a token, e.g., [13C@@H+], with canonical SMILES atoms, bonds, ring closures, and stereochemical markers each assigned their own entries. The combinatorial explosion of possible bracket tokens (due to isotopes, charges, stereochemistry) leads to very large but still incomplete coverage for practical modeling (Wadell et al., 2024).
• Subword segmentation with merge algorithms: Byte Pair Encoding (BPE), WordPiece, and Unigram (SentencePiece) algorithms are adapted from natural language processing and segment SMILES strings into frequent substrings or “chemical words.” BPE merges the most frequent adjacent pairs until a target vocabulary size is reached, often yielding tokens that correspond to functional groups or common substructures such as “c1ccccc1” (benzene ring) or “C(=O)” (carbonyl group) (Temizer et al., 2022, Kalamkar et al., 18 Nov 2025). Unigram approaches select probabilistically segmented substrings by fitting a unigram LLM and pruning low-likelihood units.
• Open-vocabulary and glyph-based methods: The Smirk tokenizer decomposes all SMILES into atomic “glyphs,” extracting every symbol needed to describe elements, isotopes, formal charges, chiralities, rings, and brackets (totaling 167 tokens). This ensures full coverage of the OpenSMILES grammar. Smirk-GPE applies BPE at the glyph level to reduce sequence length while maintaining this completeness (Wadell et al., 2024).
• Fragment- and graph-based encodings: Frameworks like t-SMILES operate on fragment-level vocabulary, using a combination of standard SMILES tokens and a minimal set of structural markers (e.g., empty node &, sibling separator ,, dummy atom *), reflecting the hierarchical composition of molecular graphs (Wu et al., 2023).

2. Vocabulary Construction Principles

The construction of a SMILES-specific vocabulary involves choices about what constitutes a “token” and which tokens are included:

• Data-driven token selection: Most contemporary workflows mine large chemical corpora (e.g. PubChem, ChEMBL, Enamine) for frequent, chemically meaningful substrings using merge-based algorithms. The stopping criterion for new merges can be frequency (e.g., only add a merged subunit if it appears more than τ times, with τ=3 in (Kalamkar et al., 18 Nov 2025)), or based on corpus likelihood maximization (WordPiece).
• Special tokens and case-sensitivity: Some tokenizers include special markers for sequence boundaries (e.g., <SMILES>, <EOS>, <CLS>, <SEP>, <MASK>, [UNK]). In most chem-specific schemes, case sensitivity is preserved to distinguish, for instance, “Cl” from “CL” (Jastrzębski et al., 2016).
• Coverage and completeness: Vocabulary coverage is quantified as the fraction of unique SMILES primitives (e.g., all legal atoms, rings, stereochemistry) represented without fallback to unknown tokens. Some closed-vocabulary tokenizers cover only 46–74% of OpenSMILES primitives, while open-vocabulary schemes like Smirk and Smirk-GPE achieve 100% coverage (Wadell et al., 2024).
• Vocabulary size tradeoffs: Typical vocabulary sizes range from thousands (BPE/WordPiece on chemical corpora) to hundreds (glyph-based open-vocab), to low tens (classical SMILES), or O(50–100) for structurally bracketed languages like SELFIES. Larger vocabularies capture more substructure but may fragment ring motifs or rare tokens; smaller vocabularies reduce coverage and dilute chemical context (Temizer et al., 2022, Wu et al., 2023).

3. Embedding and Integration into Models

Embeddings for SMILES-specific vocabularies are implemented as follows:

• One-hot encoding: In early CNN and RNN approaches, each token is mapped directly to a one-hot vector of dimensionality |V| (vocabulary size), feeding into convolutional or recurrent layers. No pre-trained embeddings are used (Jastrzębski et al., 2016).
• Learned embeddings in Transformer and LLMs: Contemporary LLMs (e.g., Llama-3 derivatives) extend their embedding matrices to accommodate new chemistry tokens, initializing new vectors as the mean of the existing base embeddings. Downstream continued pretraining uses the cross-entropy loss over the expanded vocabulary (Kalamkar et al., 18 Nov 2025).
• Adapters for grammar and type: Approaches that inject grammatical knowledge via adapters (e.g., K-adapters) augment embedding spaces with representations that encode connectivity (syntactic parse) and token type, improving understanding of SMILES grammar (Lee et al., 2022).
• BPE/compressiveness: BPE or merged vocabularies reduce sequence length (“token fertility”) for SMILES by capturing frequent substructures as single tokens, decreasing computational burden in sequence modeling while maintaining expressiveness (Kalamkar et al., 18 Nov 2025, Wadell et al., 2024).

4. Evaluation Metrics and Empirical Assessment

Performance and quality of a SMILES-specific vocabulary are assessed using several metrics:

• Coverage: The proportion of SMILES primitives representable without unknown tokens, as defined by OpenSMILES. For instance, Smirk and Smirk-GPE attain 100% coverage, while closed-vocabulary tokenizers range from 46% to 74% (Wadell et al., 2024).
• Fertility: Median number of tokens per SMILES (sequence length). Lower fertility correlates with higher modeling efficiency; for example, optimized vocabularies reduced median sequence length from 41 to 10 tokens (Smirk, BPE, vocabulary extension schemes) (Kalamkar et al., 18 Nov 2025).
• Language modeling cross-entropy: Add-one-smoothed n-gram cross-entropy (measured in nats) is used as an intrinsic metric for tokenization quality, strongly correlated with downstream molecular property prediction accuracy (Wadell et al., 2024).
• Downstream property prediction: RMSE or F1-score for regression/classification tasks on benchmarks such as Lipophilicity, FreeSolv, HIV, and BBBP. Encoders using complete or optimized vocabularies (e.g., Smirk, Smirk-GPE, BPE-extended LLMs) match or outperform closed-vocabulary models (Wadell et al., 2024, Kalamkar et al., 18 Nov 2025).
• Substructure interpretability: Tokenization strategies are assessed for their ability to recover chemically meaningful fragments or “chemical words,” which can be linked to pharmacophores or known binding moieties via statistical scoring (e.g., TF-IDF) (Temizer et al., 2022).

5. Comparative Taxonomy of SMILES Vocabularies

SMILES and Derivatives

Representation	Atom/Bond Tokens	Branch/Ring/Other Tokens	Approx. Vocabulary Size
SMILES	atoms, bonds	(, ), 1–9, @, =, #, etc.	~32
DeepSMILES	as SMILES	simplified branching, repeat-count rings	~30
SELFIES	bracketed units	bracketed structural tokens	50–100
TSSA (t-SMILES)	as SMILES	&, ,	~36
TSDY	as TSSA	* (dummy atom)	~38
TSID	as TSSA	[n*] (numbered dummy atoms)	45–50
Smirk	atomic glyphs	brackets, digits, all primitives	167
Smirk-GPE	BPE-merged glyphs	as above	~2 300

This taxonomy highlights that vocabulary design impacts both expressiveness and modeling constraints: open-vocabulary schemes guarantee completeness and avoid OOV masking, while heavily merged BPE vocabularies achieve short, chemically relevant tokens but may risk partial motif fragmentation in rare cases.

6. Functional Applications and Practical Guidelines

SMILES-specific vocabularies enable a variety of practical workflows:

• Property prediction and generative modeling: Tokenization strategies directly influence LLM and encoder/decoder models for molecular property regression, classification, and molecule generation (Wu et al., 2023, Kalamkar et al., 18 Nov 2025, Lee et al., 2022).
• Pharmacophore discovery and interpretability: Subword units with chemically meaningful boundaries correspond to functional groups or binding motifs, which can be ranked and identified using TF-IDF or similar relevance scores. These units assist in ligand-based design and scaffold hopping (Temizer et al., 2022).
• Model stability and robustness: Full-coverage vocabularies (e.g., Smirk, Smirk-GPE) eliminate unknown tokens, which is essential for representing less common coordination complexes, isotopologues, and chiralities. This broadens the chemical space accessible to AI models (Wadell et al., 2024).
• Best practices: Guidelines emphasize: (1) bracketed atom or charge groups as atomic tokens or decomposed glyphs; (2) using merge algorithms to learn corpus-specific merged tokens; (3) ensuring coverage and minimizing sequence length while maintaining semantic integrity; (4) integrating grammatical adapters for models aiming at interpretable chemical grammar (Kalamkar et al., 18 Nov 2025, Lee et al., 2022).

A plausible implication is that ongoing vocabulary refinement is crucial as chemical LLMs are deployed in broader material and drug discovery settings, where novel chemotypes and rare SMILES symbols become prevalent.

7. Perspectives and Evolving Challenges

Persistent challenges in SMILES vocabulary development include:

• Balance of granularity and chemical completeness: Large substructure-based vocabularies (e.g., BPE 8k–32k) can represent common fragments efficiently but may split critical motifs or fail on rare primitives. Fully open-vocabulary schemes scale better but modestly increase sequence length and learning complexity (Wadell et al., 2024, Temizer et al., 2022).
• Tokenization bottleneck in large LLMs: General-domain tokenizers fragment SMILES into semantically incomplete pieces, prompting targeted vocabulary extension workflows that augment LLMs with chemistry-specific tokens and continued pretraining (Kalamkar et al., 18 Nov 2025).
• Grammar-injection and structural modeling: Augmenting tokenizers or models with explicit grammar, connectivity, and type information (using adapters or explicit parse trees) can improve structure-awareness and property prediction (Lee et al., 2022).
• Evaluation standards: Coverage, fertility, n-gram cross-entropy, and error metrics for downstream tasks form an emerging set of best practices for robust comparison and benchmarking (Wadell et al., 2024).

In summary, SMILES-specific vocabulary design has advanced from simple string segmentation to sophisticated, coverage-guaranteed systems integrating data-driven substructure discovery, open-vocabulary modeling, and explicit chemical grammar knowledge. These vocabularies underpin the ongoing progress in molecular AI, bridging cheminformatics and modern language modeling (Jastrzębski et al., 2016, Temizer et al., 2022, Kalamkar et al., 18 Nov 2025, Lee et al., 2022, Wu et al., 2023, Wadell et al., 2024).