CAMT5: Context-Aware Molecular T5

• The paper introduces CAMT5, a text-to-molecule model that replaces atom-level tokenization with chemically meaningful motif tokens to capture global structural context.
• It employs an importance-based training strategy that weights larger fragments, reducing pre-training token usage to just 2% compared to BioT5.
• Experimental results show CAMT5 achieves higher exact match scores and 100% validity on datasets like ChEBI-20 and PCDes, highlighting its robust performance.

Searching arXiv for the specified paper and cited baselines. {"^^^^^^^^5query5 (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^^^^^", "^^^^^^^^5max_results5 ^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55query5^^^^} {"^^^^^^^^5query5 Text-to-Molecule Models with Context-Aware Tokenization\"", "^^^^^^^^5max_results5 5} {"^^^^^^^^5query5 arXiv text-to-molecule MolT5", "^^^^^^^^5max_results5 ^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55query5^^^^} Context-Aware Molecular T5 (CAMT5) is a text-to-molecule model that adapts a standard Transformer-based encoder–decoder T5 backbone to molecular generation by replacing atom-level tokenization with a substructure-level, or motif-level, vocabulary and by introducing an importance-based training strategy that prioritizes chemically salient fragments such as ring systems and non-single-bond components. In "Training Text-to-Molecule Models with Context-Aware Tokenization" (&&&^^^^5query5^^^^&&&), CAMT5 is presented as a response to a central limitation of prior text-to-molecule systems: atom-level tokenizations primarily model local connectivity and thereby limit the ability to capture global structural context within molecules. The model targets description-to-molecule generation and is evaluated on ChEBI-^^^^5max_results5query5^^^^ and PCDes, where it is reported to outperform MolT5 and BioT5 while using only ^^^^5max_results5^^^^% of the training tokens employed by BioT5 (&&&^^^^5query5^^^^&&&).

^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^. Conceptual basis and problem setting

CAMT5 is formulated for the setting in which a natural-language description PRESERVED_PLACEHOLDER_^^^^5query5^^^^ is mapped to a molecular output sequence. The central architectural decision is to predict a sequence of motif tokens PRESERVED_PLACEHOLDER_^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^ rather than atom tokens. In the encoder, the input text is mapped through a shared embedding layer and PRESERVED_PLACEHOLDER_^^^^5max_results5^^^^ Transformer blocks to contextual hidden states; in the decoder, motif tokens are generated autoregressively according to

PRESERVED_PLACEHOLDER_^^^^5query5^^^^

with a vocabulary size PRESERVED_PLACEHOLDER_^^^^5ti:\5^^^^ for ChEBI-^^^^5max_results5query5^^^^ and PCDes (&&&^^^^5query5^^^^&&&).

The motivating contrast is between atom-level and motif-level representations. Atom-level encodings such as SMILES and SELFIES assign one token to each atom and bracket or branch symbol. According to the paper, such tokenizations often require grammar constraints to guarantee validity and mainly emphasize local connectivity. CAMT5 instead assigns one token to a chemically meaningful fragment: all atoms participating in any ring system, all atoms connected by a non-single bond, or a singleton atom otherwise. This redefinition shifts the modeling unit from isolated local symbols to substructures that are intended to encode broader molecular semantics (&&&^^^^5query5^^^^&&&).

A common misconception is that CAMT5 is merely a backbone substitution. The paper’s formulation indicates otherwise: the backbone remains standard T5, while the primary innovations are the substructure-level tokenization and the importance-based training scheme. This suggests that CAMT5 is best understood as a token-space and objective-space intervention rather than as a novel Transformer architecture in the narrow sense.

^^^^5max_results5^^^^. Substructure tokenization and molecular linearization

The formal object underlying CAMT5 tokenization is a molecular graph

$$G=(V,E),\quad V=\{v_1,\dots,v_{|V|}\},\quad E\subseteq V\times V.$$

The vertex set is partitioned into disjoint subsets $\{V_i\}_{i=1}^n$ such that each induced subgraph $M_i=(V_i,E_i)$ is either a ring, a non-single-bond connected component, or a singleton. From these fragments, the method constructs a motif tree

$$\mathcal T(G)=(\mathcal V,\mathcal E),\quad \mathcal V=\{M_1,\dots,M_n\},$$

where $(M_i,M_j)\in\mathcal E$ if and only if there is at least one bond in PRESERVED_PLACEHOLDER_^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55query5^^^^ between PRESERVED_PLACEHOLDER_^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^ and PRESERVED_PLACEHOLDER_^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55max_results5^^^^. By construction,

PRESERVED_PLACEHOLDER_^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55query5^^^^

and PRESERVED_PLACEHOLDER_^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55ti:\5^^^^ (&&&^^^^5query5^^^^&&&).

Tokenization proceeds in three steps. First, the method identifies substructures, specifically rings and non-single-bond clusters. Second, it builds the motif tree over those fragments plus singletons. Third, it linearizes the motif tree by a depth-first search (DFS), producing the motif-token sequence. During emission, CAMT5 records the order indices of fragmented bonds that point to children or parent tokens, stereochemical markers such as E/Z and R/S on incident edges, and implicit hydrogens when a fragmented bond is not later re-attached (&&&^^^^5query5^^^^&&&).

The extraction rules are explicit. Ring finding uses a standard cycle detection algorithm, exemplified in the paper by Johnson’s algorithm. Non-single-bond components are computed as connected components of the subgraph

PRESERVED_PLACEHOLDER_^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT555^^^^

The paper’s high-level pseudocode describes the procedure as extracting rings and multi-bond components, adding singleton atoms, building inter-fragment edges, running DFS on the fragment graph, and annotating each emitted fragment (&&&^^^^5query5^^^^&&&).

An important property claimed for this representation is non-degeneracy at the level of valid reconstruction: because each token PRESERVED_PLACEHOLDER_^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^6 is a unified, non-degenerate fragment, any linear sequence PRESERVED_PLACEHOLDER_^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^7 produced by the decoder is guaranteed to map back to a valid molecular graph. In the paper’s framing, this distinguishes CAMT5 from tokenizations that can fail due to grammar constraints or that admit multiple token-level realizations of equivalent molecular content (&&&^^^^5query5^^^^&&&).

^^^^5query5^^^^. Importance-weighted pre-training and supervised fine-tuning

CAMT5 uses a two-phase training pipeline: masked language modeling pre-training on unlabeled molecules and text, followed by supervised text-to-molecule fine-tuning. The distinctive mechanism is the use of a motif importance score

PRESERVED_PLACEHOLDER_^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^8

which for PRESERVED_PLACEHOLDER_^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^9 motifs is instantiated as

PRESERVED_PLACEHOLDER_^^^^5max_results5query5^^^^

Here PRESERVED_PLACEHOLDER_^^^^5max_results5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^ is the atom count of motif PRESERVED_PLACEHOLDER_^^^^5max_results5max_results5^^^^ (&&&^^^^5query5^^^^&&&).

These weights serve two stated purposes. During masked-language-model pre-training, each token’s loss is weighted by PRESERVED_PLACEHOLDER_^^^^5max_results5query5^^^^. In addition, when initializing new motif embeddings rather than re-using atom embeddings, tokens with high PRESERVED_PLACEHOLDER_^^^^5max_results5ti:\5^^^^ are sampled more frequently so that their embedding subspace is better tuned. The design rationale is that larger fragments, including rings and multi-bond assemblies, should receive greater emphasis during representation learning (&&&^^^^5query5^^^^&&&).

The pre-training objective is

PRESERVED_PLACEHOLDER_^^^^5max_results55^^^^

where PRESERVED_PLACEHOLDER_^^^^5max_results5^^^^6 is the usual cross-entropy for predicting a masked token given the remaining unmasked tokens. The fine-tuning objective reverts to the standard autoregressive cross-entropy,

PRESERVED_PLACEHOLDER_^^^^5max_results5^^^^7

with no additional weighting in the supervised stage (&&&^^^^5query5^^^^&&&).

The paper attributes CAMT5’s sample efficiency to two factors: motif sequences are typically ^^^^5query5^^^^× shorter than atom sequences, and the weighted pre-training objective focuses learning on chemically salient fragments. Empirically, Figure ^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^ is reported to show that CAMT5 reaches the BioT5 baseline performance using only ^^^^5max_results5^^^^% of the pre-training tokens. This suggests that the tokenization change and the weighting scheme are intended to operate jointly: compression alone reduces sequence length, while importance weighting biases optimization toward structurally consequential units (&&&^^^^5query5^^^^&&&).

^^^^5ti:\5^^^^. Experimental regime and reported performance

The pre-training corpus combines text from the C^^^^5ti:\5^^^^ corpus with molecular data from ZINC-^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT555^^^^, amounting to ^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^.6 billion motif tokens. Fine-tuning is performed on description-to-molecule datasets comprising ChEBI-^^^^5max_results5query5^^^^ with ^^^^5max_results5^^^^6 ^^^^5ti:\5query5 (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^/^^^^5query5^^^^  train/val/test pairs and PCDes with ^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55query5^^^^ 5^^^^5query5query5^^^^/ (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^ 5^^^^5query5query5^^^^/ train/val/test pairs, together with +^^^^5query5ti:\5^^^^ ^^^^5query5query5query5^^^^ extra pairs from PubChem for fair alignment of BioT5 and MolT5 (&&&^^^^5query5^^^^&&&).

The optimization settings are specified in detail. Pre-training uses AdamW with learning rate PRESERVED_PLACEHOLDER_^^^^5max_results5^^^^8, warm-up for ^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^,^^^^5query5query5query5^^^^ steps, cosine annealing to PRESERVED_PLACEHOLDER_^^^^5max_results5^^^^9, batch size ^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^6×^^^^5ti:\5^^^^ GPUs, ^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55query5query5^^^^ k steps, and gradient clipping at ^^^^5query5query5^^^^. Fine-tuning uses batch size ^^^^5ti:\5^^^^8 on a single RTX ^^^^5query5query5max_results5query5^^^^ for 5^^^^5query5^^^^ k steps, learning rate in PRESERVED_PLACEHOLDER_^^^^5query5query5^^^^, cosine scheduling, clipping at ^^^^5query5query5^^^^, and maximum length 5^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55max_results5^^^^ (&&&^^^^5query5^^^^&&&).

The principal reported results are summarized below.

Dataset / model	Key reported result	Additional note
ChEBI-^^^^5max_results5query5^^^^ / CAMT5_base (^^^^5max_results5^^^^86 M)	Exact ^^^^5query5^^^^.^^^^5ti:\5max_results5max_results5	Validity ^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^.^^^^5query5query5query5^^^^
ChEBI-^^^^5max_results5query5^^^^ / MolT5_base	Exact ^^^^5query5^^^^.^^^^5query5max_results5	Validity ^^^^5query5^^^^.95^^^^5query5
ChEBI-^^^^5max_results5query5^^^^ / BioT5_base	Exact ^^^^5query5^^^^.^^^^5query5ti:\5ti:\5	Validity ^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^.^^^^5query5query5query5^^^^
PCDes / CAMT5_base	Exact ^^^^5query5^^^^.^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^96	Compared against MolT5_base ^^^^5query5^^^^.^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT555arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^ and BioT5_base ^^^^5query5^^^^.^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55query5max_results5^^^^

On ChEBI-^^^^5max_results5query5^^^^, Table A reports five metrics: Exact match, MACCS Tanimoto, RDK Tanimoto, Morgan Tanimoto, and Validity. CAMT5_base (^^^^5max_results5^^^^86 M parameters) yields Exact ^^^^5query5^^^^.^^^^5ti:\5max_results5max_results5 MACCS ^^^^5query5^^^^.88^^^^5max_results5 RDK ^^^^5query5^^^^.8^^^^5query5ti:\5 Morgan ^^^^5query5^^^^.7^^^^5ti:\5max_results5 and Validity ^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^.^^^^5query5query5query5^^^^, compared with MolT5_base at ^^^^5query5^^^^.^^^^5query5max_results5 ^^^^5query5^^^^.8^^^^5ti:\5 ^^^^5query5^^^^.797, ^^^^5query5^^^^.7^^^^5max_results5query5 and ^^^^5query5^^^^.95^^^^5query5 and BioT5_base at ^^^^5query5^^^^.^^^^5query5ti:\5ti:\5 ^^^^5query5^^^^.8^^^^5ti:\5max_results5 ^^^^5query5^^^^.77^^^^5query5 ^^^^5query5^^^^.66^^^^5ti:\5 and ^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^.^^^^5query5query5query5^^^^. The paper also states that even the small variant (^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55query5query5^^^^ M) outperforms both baselines. On PCDes, CAMT5_base achieves Exact ^^^^5query5^^^^.^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^96, compared with ^^^^5query5^^^^.^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT555arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^ for MolT5_base and ^^^^5query5^^^^.^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55query5max_results5^^^^ for BioT5_base (&&&^^^^5query5^^^^&&&).

The efficiency claim is framed in token-budget terms. Because motif sequences are approximately ^^^^5query5^^^^× shorter, pre-training on ^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^.6 B motif tokens corresponds to only ^^^^5max_results5^^^^% of the 69 B SELFIES tokens used by BioT5. Figure ^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^ (right) is reported to show that with pre-training tokens less than or equal to 5^^^^5query5^^^^ M, CAMT5 already matches or exceeds baselines trained on hundreds of millions of tokens (&&&^^^^5query5^^^^&&&).

5. Confidence-based ensemble inference

Although CAMT5 is described as the single best model on average, the paper notes that some individual prompts are better served by MolT5 or BioT5. To exploit this complementarity, it defines a per-candidate confidence score and selects the candidate with highest confidence across models (&&&^^^^5query5^^^^&&&).

For a model PRESERVED_PLACEHOLDER_^^^^5query5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^ generating a sequence PRESERVED_PLACEHOLDER_^^^^5query5max_results5^^^^ on prompt PRESERVED_PLACEHOLDER_^^^^5query5query5^^^^, confidence is defined as

PRESERVED_PLACEHOLDER_^^^^5query5ti:\5^^^^

The inference procedure is correspondingly simple: each model generates a candidate, invalid candidates are filtered out, the mean log-probability is computed for each valid candidate, and the candidate with maximal confidence is returned (&&&^^^^5query5^^^^&&&).

The reported gains are nontrivial. On ChEBI-^^^^5max_results5query5^^^^, the ensemble of {MolT5, BioT5, CAMT5} raises Exact from ^^^^5query5^^^^.^^^^5ti:\5query5query5^^^^ to ^^^^5query5^^^^.^^^^5ti:\5query5max_results5^^^^ and RDK from ^^^^5query5^^^^.8^^^^5ti:\5query5^^^^ to ^^^^5query5^^^^.86^^^^5query5 On PCDes, Exact increases from ^^^^5query5^^^^.^^^^5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55^^^^96 to ^^^^5query5^^^^.^^^^5max_results5arXiv (Kim et al., 30 Aug 2025) Training Text-to-Molecule Models with Context-Aware Tokenization BioT5 MolT55query5^^^^. Figure ^^^^5ti:\5^^^^ is described as showing cases in which CAMT5’s confidence dips and another model attains both higher confidence and better chemical fidelity. A plausible implication is that the three systems retain partially distinct error surfaces even when evaluated on the same task, making simple confidence-based selection effective despite the absence of a more elaborate meta-learner (&&&^^^^5query5^^^^&&&).

6. Interpretation, limitations, and prospective extensions

The paper offers three reasons why context-aware tokenization helps. First, global context is better represented because rings and multi-bond fragments carry resonance and stereochemical information that atom sequences must infer indirectly. Second, motif tokens reduce sequence length by approximately ^^^^5query5^^^^×–5×, which is argued to improve gradient flow and reduce exposure bias. Third, each motif has a unique interpretation, avoiding SELFIES degeneracies or SMILES grammar failures (&&&^^^^5query5^^^^&&&).

These claims delineate CAMT5’s position within text-to-molecule modeling. The model does not replace sequence generation with graph generation; instead, it redefines the sequential alphabet so that the sequence itself is structurally denser. This suggests a broader methodological lesson: in molecular language modeling, a substantial portion of performance may depend not only on the backbone and objective, but also on the granularity at which molecular structure is serialized.

The paper also identifies several limitations and extensions. CAMT5 is said to focus on token space, and future work can integrate it with diffusion models such as LDMol, GFlowNets, or multi-task pre-training frameworks. On the representational side, more nuanced fragment schemes, including BRICS or learned graph cuts, could be explored, although initial ablations in Table 9 are reported to show that the simple ring-plus-multibond choice is already optimal. The discussion also notes a misuse concern: as with any generative chemistry model, safeguards are needed against designer toxins (&&&^^^^5query5^^^^&&&).

Two misconceptions are explicitly countered by the paper’s design and results. One is that atom-level tokenization is an essentially neutral preprocessing choice; CAMT5 treats tokenization as a core modeling decision with measurable effects on sample efficiency and exact generation quality. The other is that substructure tokenization must necessarily require complex or learned fragment discovery; the reported ablations indicate that a simple hand-specified decomposition into rings, non-single-bond connected components, and singletons is sufficient to achieve the model’s best reported performance.