FragAtlas-62M: GPT-2 for Fragment Generation

• The paper presents a GPT-2 based fragment generator trained on 62M SMILES from ZINC-22, achieving a 99.90% validity rate and rediscovery of known fragments.
• FragAtlas-62M is a specialized foundation model that generates de novo chemical fragments with high descriptor fidelity, supporting practical fragment-based drug discovery workflows.
• The model’s architecture, featuring 6 transformer layers and 42.7M parameters, balances moderate scale with fragment domain specialization for rapid library expansion and hit identification.

Searching arXiv for the primary paper and a related source to support disambiguation. FragAtlas-62M is a specialized foundation chemical LLM for comprehensive fragment-based drug discovery, trained on the complete ZINC-22 fragment subset comprising over 62 million molecules. Built as a GPT-2–based autoregressive model with 42.7M parameters, it is explicitly oriented toward fragment chemical space rather than whole-molecule generation. Its reported behavior combines very high chemical validity, close alignment to the training distribution across descriptor and fingerprint analyses, substantial rediscovery of known fragments, and a nontrivial fraction of novel structures with practical relevance (Ho et al., 23 Sep 2025).

1. Scope and scientific positioning

Fragment-based drug discovery (FBDD) uses low-molecular-weight fragments as starting points for hit discovery and lead optimization. Within that setting, FragAtlas-62M is positioned as a fragment-focused foundation model rather than a general chemical LLM. The motivation given for the model is that many existing chemical LLMs are trained on full-sized drug-like molecules and are therefore optimized for whole-molecule design, whereas fragment-specific generative tools have generally been smaller in scale and often oriented toward linking or screening rather than comprehensive de novo fragment generation (Ho et al., 23 Sep 2025).

The defining claim of FragAtlas-62M is scale at the fragment level. It is trained on the complete ZINC-22 fragment subset and is described as using the largest fragment dataset to date. The intended consequence is broad coverage of fragment chemical space together with a reusable generator that preserves realistic fragment property distributions while still yielding novel structures. This suggests a shift from fragment-generation systems designed for narrow downstream tasks toward a more general-purpose base model for fragment library expansion, candidate proposal, and downstream FBDD workflows.

2. Training corpus and preprocessing regime

The training data are the ZINC-22 2D fragments defined by heavy atom bins H08 to H16 inclusive and LogP bins M500 to P240 inclusive. In the notation provided, this corresponds to heavy atom counts $H \in \{8, 9, \dots, 16\}$, with LogP spanning bins from $L \le -5.00$ up to $L \ge 2.40$. After preprocessing, the corpus contains 62,015,589 SMILES strings (Ho et al., 23 Sep 2025).

Preprocessing preserves representational diversity while separating generation-time evaluation from training-time representation. Exact string duplicates are removed from the training set, but non-canonical SMILES are preserved during training. Canonicalization is applied only at evaluation time for validity checking and for counting unique structures and overlaps with ZINC. The reported SMILES statistics are an average length of $31.54 \pm 5.85$ characters, a range of 8–86 characters, and an interquartile range of 27–35 characters. Total training volume is approximately 2 billion SMILES tokens.

The data characterization emphasizes distributional comparisons between generated and training fragments rather than a single exhaustive descriptor inventory. The explicitly named descriptors in the provided details are QED, LogP, molecular weight, H-bond donors, heavy atom count, Topological Polar Surface Area, Synthetic Accessibility score, H-bond acceptors, rotatable bonds, ring count, and Natural Product score. The paper states that validation is performed across 12 descriptors and three fingerprint methods.

3. Architecture and optimization

FragAtlas-62M uses a decoder-only GPT-2 configuration with 6 transformer layers, 12 attention heads, 768-dimensional embeddings, a context length of 128 tokens, and a character-level tokenizer with vocabulary size 42. The parameter count is 42.7M. The model generates SMILES autoregressively, token by token, by learning the conditional distribution $p_{\theta}(x_t \mid x_{<t})$ and minimizing next-token cross-entropy (Ho et al., 23 Sep 2025):

$$\mathcal{L}(\theta) = - \sum_{t=1}^{T} \log p_{\theta}(x_t \mid x_{1:t-1})$$

Training is implemented with HuggingFace Transformers on PyTorch. The train/validation split is 99:1, and optimization runs for 5 epochs over the 62M-fragment dataset. The initial learning rate is $5 \times 10^{-5}$, with linear decay and 10% warmup, and the training description notes no restarts. Non-canonical SMILES are retained during training as a form of robustness-oriented augmentation, while validity is enforced only at evaluation through RDKit sanitization and canonicalization.

A concise summary of the core model configuration is useful because the design deliberately balances moderate model scale against fragment-domain specialization:

Component	Specification
Architecture	GPT-2–based decoder-only transformer
Layers / heads	6 layers / 12 attention heads
Hidden size	768
Context window	128 tokens
Tokenization	Character-level SMILES, vocabulary size 42
Parameters	42.7M

The implementation is also described as practically accessible. Generation throughput is reported as greater than 1,000 molecules per second on a single RTX 4090. A plausible implication is that the model is intended not only as a research artifact but also as a high-throughput generator that can be inserted into routine screening and library-construction pipelines.

4. Validity, distributional fidelity, and structural alignment

The primary validity assessment is RDKit-based: generated SMILES are parsed and sanitized, and only strings returning valid molecule objects are retained. The reported totals are 62,015,589 generated SMILES, 61,951,924 valid molecules, and a validity rate of 99.90% (Ho et al., 23 Sep 2025). For a character-level SMILES generator, the reported validity is a central empirical result because syntax and valence errors are a common failure mode of sequence-only molecular generators.

Distributional fidelity is evaluated through descriptor comparisons using Cohen’s $d$ effect sizes. For the “Complete” generated-versus-training comparison, the reported values include QED $d = 0.001$, LogP $d = 0.009$, H-bond donors $L \le -5.00$0, Topological Polar Surface Area $L \le -5.00$1, molecular weight $L \le -5.00$2, heavy atom count $L \le -5.00$3, Synthetic Accessibility score $L \le -5.00$4, H-bond acceptors $L \le -5.00$5, rotatable bonds $L \le -5.00$6, ring count $L \le -5.00$7, and Natural Product score $L \le -5.00$8. The summary given is that all $L \le -5.00$9 for the Complete comparison, which is interpreted in the paper as negligible distributional shift.

Structural alignment is evaluated with three fingerprint methods: Morgan fingerprints (ECFP4, radius 2, 2048 bits), MACCS keys (166 bits), and topological fingerprints (2048 bits). The analysis compares Novel–Novel, Rediscovered–Rediscovered, and Novel–Rediscovered distance distributions. For Morgan fingerprints, the reported means are NN $L \ge 2.40$0, RR $L \ge 2.40$1, and NR $L \ge 2.40$2, with ratios NR/NN $L \ge 2.40$3 and NR/RR $L \ge 2.40$4. Similar patterns are reported for MACCS and topological fingerprints. The interpretation supplied is that ratios close to 1.0 indicate comparable intergroup and intragroup distances, and the associated t-SNE plots show strong overlap between novel and rediscovered fragments rather than disjoint regions of anomalous chemistry.

5. Coverage, novelty, and relevance to fragment-based drug discovery

After RDKit validation and canonicalization, the reported unique canonical generated set contains 42,597,827 SMILES. Of these, 33,210,363 are canonical SMILES shared with the ZINC reference set, yielding a reported ZINC coverage of 53.55%. The corresponding novelty count is 9,387,465 unique canonical fragments not present in the ground-truth ZINC fragment set, equal to 22.04% of unique canonical generated SMILES (Ho et al., 23 Sep 2025).

These two quantities—rediscovery and novelty—are central to how the model is framed. Rediscovery indicates that FragAtlas-62M reproduces a large fraction of known fragment space, while novelty indicates that it does not merely memorize the corpus. The paper’s claim of “practical relevance” for the novel subset is supported by descriptor alignment and fingerprint overlap. For the “Novel Only” subset, the reported effect sizes remain small, including QED $L \ge 2.40$5, LogP $L \ge 2.40$6, molecular weight $L \ge 2.40$7, Synthetic Accessibility score $L \ge 2.40$8, and ring count $L \ge 2.40$9. The associated interpretation is that the novel fragments remain close to the same chemically realistic manifold as known ZINC fragments.

Within FBDD workflows, the model is described as supporting several concrete uses. It can augment fragment libraries with novel but realistic fragments, provide seeds for fragment growing or linking, and serve scaffold-hopping or hit-discovery pipelines when combined with docking, biophysical screening, or structure-based methods. The practical workflow described in the details is sequential: download model and tokenizer, generate fragments up to the 128-token context limit, validate with RDKit, canonicalize and deduplicate, classify fragments as novel or rediscovered by set difference against ZINC, compute descriptors such as LogP, molecular weight, TPSA, HBD/HBA, SA, QED, and NP, and then integrate the selected subset into docking, fragment linking, or target-specific fine-tuning pipelines. Because inference speed is reported as greater than 1,000 molecules per second on RTX 4090, the system is presented as compatible with large-batch fragment proposal.

6. Limitations, release, and nomenclatural ambiguity

The limitations stated for FragAtlas-62M are specific and consequential. The model does not explicitly model stereochemistry, does not incorporate geometric or 3D structural information, and does not encode a fragment assembly grammar for linking or growing fragments into full molecules. The training corpus is ZINC-based, so the resulting generator inherits biases associated with ZINC curation and commercial accessibility. The representation is also limited by character-level SMILES and GPT-2 sequence modeling, even though the reported validity is high (Ho et al., 23 Sep 2025).

These limitations delimit the model’s role. FragAtlas-62M is not presented as a complete end-to-end FBDD pipeline, nor as a structure-based generator with binding-site awareness. Rather, it is positioned as a foundation model to be extended with conditional control, 3D information, fragment-assembly methods, or task-specific fine-tuning. This suggests a modular research program in which sequence-only fragment generation is one component of a broader medicinal chemistry and structure-based design stack.

The release includes training code, preprocessed data, documentation, and model weights. In practical terms, that makes the system a reusable baseline for fragment generation, fragment-library design, and adaptation to property- or target-specific settings.

A separate point of clarification concerns the name itself. In the paper "Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data," the term “FragAtlas-62M” is not used by the authors; in that context, it would only be an external label for a 62M-image pseudo-labeled corpus used for monocular depth estimation rather than a chemical LLM (Yang et al., 2024). In the chemical modeling context, by contrast, FragAtlas-62M denotes the GPT-2–based fragment generator trained on 62,015,589 ZINC-22 fragments (Ho et al., 23 Sep 2025).