SMILES-GRPO for Molecular Property Optimization

• SMILES-GRPO is a reinforcement learning framework that optimizes molecular properties using grouped reward normalization and SMILES sequence adaptation.
• It leverages group-centric baselines and an autoregressive Transformer with dynamic grammar masking to generate structurally valid molecules.
• Empirical evaluations show significant improvements in variance reduction, optimization efficiency, and generalization to unseen molecular scaffolds.

SMILES-GRPO (Generalized Reward for Property Optimization) is a reinforcement learning (RL) framework for the optimization of molecular properties, adapting the Group Relative Policy Optimization (GRPO) technique to policies operating on SMILES string representations. This approach supports amortized, scaffold-conditional molecular generation, delivering significant improvements in variance reduction and optimization efficiency over prior instance-based optimizers, particularly when generalizing to unseen molecular scaffolds and under limited oracle evaluation budgets (Javaid et al., 12 Feb 2026).

1. Core Principle: Reward Normalization via Group-Centric Baselines

SMILES-GRPO reframes the molecular optimization Markov Decision Process (MDP) such that reward feedback is provided only at the trajectory’s terminal state, i.e., the final SMILES string. Each optimization batch groups completions originating from a common starting scaffold, with all reward normalization and policy updates performed on a per-group basis.

• For each scaffold $S_i$ in a batch of size $B$, $G$ distinct completions ${O_{i,1},...,O_{i,G}}$ are sampled using stochastic decoding techniques. The per-trajectory reward $r_{i,j} = R(O_{i,j})$ reflects external property evaluation at termination.
• The group mean baseline $\mu_i = \frac{1}{G}\sum_{j=1}^{G} r_{i,j}$ is used to compute a centered advantage for each trajectory: $A_{i,j} = r_{i,j} - \mu_i$.
• No learned value network is required: variance is reduced via direct, group-wise centering. This approach is especially effective when scaffold difficulty and potential reward distribution are highly heterogeneous, as is typical in molecular optimization scenarios.

2. Policy Optimization Objective

The optimization objective for SMILES-GRPO is an actor-only, relative-reward RL objective that, for each group, accumulates policy gradients weighted by the within-group advantage:

$$\mathcal{J}(\theta) = \mathbb{E}_{S_i, O_{i,j}}\left[\sum_{t=0}^{T_{i,j}} \log \pi_\theta(a_t^{(i,j)} \mid s_t^{(i,j)})\;A_{i,j} \right]$$

The corresponding stochastic gradient estimator used in practice is:

$$\nabla_\theta \mathcal{J}(\theta) \approx \frac{1}{B\,G} \sum_{i=1}^{B} \sum_{j=1}^{G} A_{i,j} \sum_{t=0}^{T_{i,j}} \nabla_\theta \log \pi_\theta(a_t^{(i,j)} \mid s_t^{(i,j)})$$

An optional entropy regularization term may be introduced for enhanced exploration, controlled by a coefficient $\beta$.

3. SMILES-GRPO Algorithm and Workflow

The implementation of SMILES-GRPO proceeds with the following key steps:

1. Batch Sampling: Sample $B$0 scaffolds (canonical SMILES) from the dataset.
2. Trajectory Generation: For each scaffold $B$1, generate $B$2 distinct, syntactically valid SMILES completions via beam search or top-$B$3 sampling, starting from the fixed prefix $B$4.
3. Reward Computation: Evaluate each completion $B$5 using a molecular property oracle for scalar reward $B$6.
4. Baseline and Advantages: Compute the group mean $B$7 and set $B$8.
5. Gradient Update: Aggregate gradients for all trajectories using the advantage-weighted log-policy and update $B$9 using normalized step size.

Key features include explicit group-based variance reduction, no use of PPO clipping or a value network, and per-batch parameter updates.

4. Model Architecture and SMILES Adaptation

While the original GRPO was introduced in the context of graph-based Transformers (GraphXForm), its adaptation to SMILES sequences involves the following components:

• Autoregressive Transformer Model: Standard Transformer decoder with token-level (atom, bond, ring, parenthesis) embeddings, plus positional encodings. Input is the partially completed SMILES string, optionally prepended by a canonical scaffold string.
• Grammar Masking: At each decoding step, a dynamic mask precludes syntactically invalid or chemically impossible next-token options (e.g., unbalanced parentheses, unmatched rings, or repeated bond symbols).
• Diverse Decoding: Non-redundant completion generation via diverse beam search or top-$G$0 sampling with masking ensures $G$1 structurally distinct outputs per scaffold.
• Pretraining: Teacher-forcing on large corpora of molecules (with randomized SMILES) is used before RL fine-tuning.
• Group Definition: Canonical SMILES ensures each unique starting scaffold is recognized and grouped accurately, preventing reward leakage or improper normalization.

Advantages of this architecture include ease of implementation and compatibility with standard LLM toolkits. Challenges include the design of effective grammar masks and ensuring correct group assignment for chemically equivalent but syntactically distinct SMILES.

5. Experimental Evaluation and Comparative Performance

SMILES-GRPO inherits its evaluation protocol from the original GRXForm studies:

• Kinase Scaffold Decoration Task: On ZINC-250k Murcko scaffolds, with multi-objective reward $G$2 incorporating GSK3β, JNK3 predicted activity, QED, and synthetic accessibility (SA).
  • GRXForm (graph-based GRPO) yields objective score $G$3 and $G$4 multi-objective success, outperforming LibINVENT, DrugEx v3, Mol GA, and GenMol (all $G$5 score; $G$6 multi-criteria success).
  • Ablations validate that group-relative baselining provides large gains: REINFORCE with a global baseline achieves only $G$7 score and $G$8 success.
• Prodrug Transfer (Few-Shot): 4 training exemplars, 5 out-of-distribution test drugs; GRPO outperforms REINFORCE ($G$9 vs ${O_{i,1},...,O_{i,G}}$0 mean score).
• PMO De-Novo Benchmark: 22 design tasks, 10,000 oracle call budget; GRXForm achieves area-under-top-10-curve (AUC-Top-10) ${O_{i,1},...,O_{i,G}}$1, second to GenMol (${O_{i,1},...,O_{i,G}}$2).
• Oracle Cost Efficiency: GRPO-trained policies realize amortized inference, surpassing Mol GA and GenMol after several scaffold tasks due to fixed pretraining cost (${O_{i,1},...,O_{i,G}}$350,000 oracle calls).

These results confirm that group-wise normalization is critical for successful generalization and robust policy learning on scaffold-constrained molecular optimization tasks.

6. Practical Guidelines and Implementation Strategies

• Canonicalization: All scaffolds/components must be converted to canonical SMILES for unambiguous group assignment. Further robustness can be achieved by clustering similar scaffolds (e.g., by Morgan fingerprint or Murcko core).
• SMILES Masking: A dynamic grammar-based mask is essential for maintaining validity. Optionally, molecular graph parsing can be used for additional chemical validity assurance during decoding.
• Variance Management: For groups exhibiting negligible within-group reward variance, gradient updates may be skipped or a low-variance floor applied to prevent stagnation.
• Decoding Parameters: Typical settings are group size ${O_{i,1},...,O_{i,G}}$4, batch size ${O_{i,1},...,O_{i,G}}$5, total sample size per update ${O_{i,1},...,O_{i,G}}$6–${O_{i,1},...,O_{i,G}}$7, learning rate ${O_{i,1},...,O_{i,G}}$8 (Adam), training epochs ${O_{i,1},...,O_{i,G}}$9–$r_{i,j} = R(O_{i,j})$0.
• Entropy Bonus: Set to zero for deterministically exploiting property space; optionally increase to $r_{i,j} = R(O_{i,j})$1 for greater molecular diversity.
• Convergence Monitoring: Training continues until the GRPO objective plateaus on a held-out validation scaffold pool.

This suggests that, for practitioners seeking to deploy amortized molecular optimization in production or large-scale virtual screening, SMILES-GRPO provides both the theoretical foundation and practical protocol for efficient property optimization with transfer to novel scaffolds.

7. Significance and Broader Implications

Group Relative Policy Optimization, as instantiated in SMILES-GRPO, establishes a generic, model-agnostic algorithm for mitigating variance in RL-based molecular optimization. By reframing each generation instance as a grouped “question”—optimize from this scaffold—SMILES-GRPO leverages relative, rather than absolute, reward signals. This ensures robust, scalable, and efficient training across highly variable chemical spaces.

The adaptation of GRPO to SMILES autoregressive policies demonstrates that the underlying high-variance mitigation and amortization strategies are decoupled from the molecular representation, requiring only that canonical grouping and appropriate grammar-constrained masking are enforced at the token level (Javaid et al., 12 Feb 2026). This framework is poised to support further advances in de novo drug design, scaffold decoration, and few-shot molecular property transfer.