--- title: 'ChemBART: Unified Organic Synthesis LLM' url: https://www.emergentmind.com/topics/chembart type: topic --- # ChemBART: Unified Organic Synthesis LLM ChemBART is a large language model tailored for organic chemistry applications, based on the BART-large Transformer architecture and pre-trained with reaction-level masked sequence reconstruction on SMILES notations. It enables a unified “one model, one pre-training, multiple tasks” paradigm, supporting precursor/reagent generation, reaction condition regression, molecular property classification, and reinforcement-learning-driven synthesis planning. ChemBART distinguishes itself from previous approaches through its ability to integrate multiple downstream chemical tasks and efficiently facilitate end-to-end computer-aided synthesis planning, with experimental validation highlighting its practical impact [2601.02915]. ## 1. Model Architecture and Tokenization ChemBART uses the BART-large backbone, comprising 12-layer encoders and decoders, a hidden dimension $d=1024$, 16 attention heads per layer, and a total of approximately 0.4 billion parameters. Reaction expressions are represented as “chemical sentences” in the following canonicalized SMILES format: $$\text{reactant} > \text{reagent} > \text{product}$$ Tokenization is performed at the atom and mapping-index level, matching one token to each atom, mapping, or SMILES punctuation. The vocabulary contains ∼201 tokens, including atom symbols (e.g., C, O, Cl, Br), aromatic indicators, mapping/ring indices ($0-83$), and punctuation (e.g., [,],@,+,—, =, #), with special tokens ``, ``, ``, ``. Each token $x_i$ is mapped to a learned embedding $e_i \in \mathbb{R}^{1024}$, summed with a positional embedding. The self-attention mechanism follows standard Transformer mapping: $$ Q = X W_Q, \quad K = X W_K, \quad V = X W_V $$ Attention for one head: $$ A = \text{softmax}(QK^\top/\sqrt{d_k}), \quad X' = AV $$ Outputs from the 16 heads per layer are concatenated and linearly projected to $d=1024$. ## 2. Pre-training Objective and Optimization ChemBART is pre-trained with a masked sequence prediction objective at the reaction level. For each reaction SMILES input, either the reactant, reagent, or product section is randomly masked (tokens replaced by ``). The decoder autoregressively generates the entire “reactant > reagent > product” string, minimizing standard cross-entropy loss: $$ L(\theta) = - \sum_{t=1}^T \log P_\theta(x_t | x_{ >` to elicit reactant or reagent reconstruction, decoded via beam search (beam size=10). - **Metrics:** Top-$k$ accuracy (% correct among top-$k$ generations), syntax error rate. - **Results (USPTO-full):** - Precursor prediction: Top-1 = 59.2%, Top-5 = 78.2%, Top-1 syntax errors = 2.43% - Reagent prediction: Top-1 = 54.5%, Top-5 = 74.4%, syntax errors ≈1.5% These results match or exceed previous template-free models (PMSR, Molecular Transformer, Chemformer). ### 3.2 Reaction Temperature and Yield Regression Inputs are complete reaction SMILES with `` and `` task tokens. Decoder outputs at these tokens are processed by linear heads, trained using mean-squared error. - **Metrics:** RMSE, MAE, $R^2$. - **ORD dataset (∼700K):** - Temperature: $R^2=0.55$, MAE$=\pm 10^\circ$C (range $-150...250^\circ$C). - Yield: $R^2=0.16$, MAE$=\pm 23\%$ (range 0–110%). - **Suzuki–Miyaura regression (5K):** - Ten-fold $R^2=0.804\pm0.017$, MAE=$7.9\pm0.4\%$, comparable to rxnfp. ### 3.3 Molecular Property Classification Datasets include BBBP, HIV, BACE, Tox21, ClinTox. A token is prepended or appended, and the corresponding encoder embedding is processed by a sigmoid-activated linear head; optionally, LoRA low-rank adaptation is applied. - **Metric:** ROC-AUC. - **Performance (ChemBART-M, Table 2):** - BBBP: 0.910; HIV: 0.809; BACE: 0.881; Tox21: 0.844; ClinTox: 0.866 - LoRA improves some tasks, e.g., ClinTox up to 0.920. ChemBART matches or outperforms previous SMILES-based LLMs and most graph-based approaches. ### 3.4 Reinforcement-Learning Policy and Value Optimization A set of ∼12K retrosynthetic nodes, generated using ReSynZ's MCTS pipeline, is labeled with: - **Value** $v$: discounted completion probability. - **Policy** $p$: normalized child visit counts. ChemBART’s two regression heads are fine-tuned to predict $v$ (value head: SMILES $\to$ scalar) and $p$ (policy head: reaction SMILES $\to$ score, softmaxed across siblings). - **Test RMSE (ChemBART-F/M):** Value = 0.11, Policy = 0.16; comparable to template-based networks and superior to random-initialized baselines. ## 4. Multi-Step Synthesis Planning with MCTS ChemBART is integrated into Monte Carlo Tree Search (MCTS) for end-to-end retrosynthetic route design. ### Node Expansion and Scoring At each tree node: 1. Beam-search generates $K$ candidate single-step retrosyntheses with probabilities $g_i$. 2. Each candidate reaction $r_i$ is checked for validity and scored using the policy head ($p_{2,i}$). 3. Candidates are normalized: $$ p_i^* = \frac{g_i \cdot p_{2,i}}{\sum_j g_j \cdot p_{2,j}} $$ 4. Node value: $$ v(n) = \text{ValueHead}(\text{SMILES}(n)) $$ MCTS selection employs a UCT-like rule (parameterization not explicitly provided) and backpropagates estimated values. Final root policies: $$ \pi_{\text{root}}(i) = \frac{N_i^\tau}{\sum_j N_j^\tau} $$ where $N_i=$ visit count, $\tau=$ temperature. ### Planning Performance - **Retro*-190 (190 targets):** ChemBART-F/M achieve 64.9% / 70.1% full synthesis success rates, approaching template-based planners. - **JMC2025 (53 recent pharmaceutical targets):** - Beam-search: 88.7% success in $\leq6$ steps (mean route length 4.3) - Top-k sampling ($k=10$): 84.9% success - Top-p sampling ($p=0.9$): 83.0% success ## 5. Experimental and Wet-Lab Validation ChemBART-generated multistep routes have been empirically validated. For the PD-L1/VISTA dual inhibitor P1: - Literature route: 6 steps, overall yield 6.5% - ChemBART proposal: 4 steps, overall isolated yield 35% (5× improvement, +28.5% absolute), all conditions and intermediates confirmed with standard lab techniques (full NMR/HRMS provided). The new pathway featured a convergent Suzuki coupling, efficient reductions, and optimized SNAr coupling conditions. ## 6. Interpretation, Limitations, and Future Prospects ChemBART’s reaction-level masked pre-training automatically instills fundamental chemical syntax, valency, and mechanistic knowledge. A single parameter set supports generative, regression, classification, and reinforcement-learning-based policy/value computations, reducing computational and maintenance overhead. ChemBART demonstrates interpretable chemistry by recovering periodic and electronegativity trends in token-embedding spaces and highlighting reactive motifs in attention maps (e.g., C→Br in Grignard reactions). **Limitations:** Pure sequence-based SMILES input constrains the physical-chemical scope (e.g., QM9-type quantum descriptors). Potential for “hallucination” (invalid/novel outputs) increases under sampling-based decoding. **Future directions include:** - Integration of 3D structure through graph or coordinate-based models (e.g., Uni-Mol, KFLM2) - Hybrid sequence-to-graph/contact-map attention modules - Further RL-based improvement via policy-gradient or offline MCTS retraining - Conditional precursor generation modulated by reaction class or functional groups In conclusion, reaction-centric pre-training on SMILES data enables ChemBART to function as a versatile foundation model for organic synthesis, providing unified, experimentally-validated solutions for retrosynthesis, property prediction, and AI-driven planning [2601.02915]. Source: https://www.emergentmind.com/topics/chembart