PepThink-R1: Interpretable Peptide Optimization

• PepThink-R1 is a generative peptide design framework integrating LLM-based chain-of-thought reasoning with reinforcement learning for enhanced pharmacological optimization.
• It employs explicit monomer-level reasoning and stepwise modifications, with rigorous RDKit and QSAR checks to ensure structural validity.
• The dual-stage workflow (CoT-SFT and GRPO) balances improvements in key properties, structural similarity, and diversity, achieving high-quality optimized peptide outputs.

PepThink-R1 is a generative framework for interpretable optimization of head-to-tail cyclic peptides, integrating LLMs with chain-of-thought (CoT) supervised fine-tuning (SFT) and reinforcement learning (RL). Designed for the generation of peptides with enhanced pharmacological properties while maintaining explicit, stepwise traceability of sequence modifications, PepThink-R1 is the first LLM-based peptide design system to achieve interpretable, monomer-level reasoning jointly with multi-property RL-driven control (Wang et al., 20 Aug 2025).

1. System Architecture and Workflow

PepThink-R1 is built upon LlaSMol, an LLM derived from Mistral-7B v0.1, enhanced with LoRA adapters to enable domain-specific instructional tuning. The framework’s workflow consists of two major stages:

• Chain-of-Thought Supervised Fine-Tuning (CoT-SFT): Given a cyclic peptide input in SMILES notation and specified property-improvement goals, the model generates a detailed monomer-level reasoning trace (CoT) followed by a single, explicit SMILES edit reflecting one monomer substitution.
• Reinforcement Learning via Group-Relative Policy Optimization (GRPO): Starting from the CoT-SFT policy, RL is used to optimize for a composite reward function. This function balances (i) improvements in predicted pharmacological properties (LogD, rat mean residence time [MRT], simulated intestinal fluid [SIF] stability), (ii) Tanimoto structural similarity to the parent peptide, and (iii) diversity via duplication penalties. KL-regularization maintains distributional proximity to the SFT policy, counteracting mode collapse and reward hacking.

The practical result is a system that, in deployment, automatically selects a mutational site and substitution, provides a corresponding rationale, and outputs an optimized cyclic-peptide SMILES.

2. Monomer-Level Reasoning and Validity Constraints

PepThink-R1’s reasoning process operates at the monomer (single-residue) level. At generation time, input peptides are tokenized so each monomer is an explicit vocabulary element (vocabulary: 10,385 monomers, including 385 natural and 10,000 synthetic non-natural amino acids [NNAAs]). For a property-improvement objective, the model:

• Identifies candidate positions for substitution.
• Articulates a rationale referencing the specific change (e.g., “At position 2, replace the glycine monomer with the NNAA [NH2+] C@@HC(=O)[O-] to (i) raise lipophilicity (LogD), (ii) extend mean residence time (MRT_Rat), and (iii) enhance SIF stability”).
• Yields a mutated peptide with only the specified monomer edit.

Structural validity is enforced by filtering outputs through RDKit parsing and QSAR prediction coverage checks, and by syntactic constraints ensuring only a single-site edit per generation. Invalid or out-of-vocabulary substitutions are filtered out during both SFT and RL.

3. Mathematical Formulation and Reward Structure

The supervised SFT loss is the standard token-level cross-entropy:

$$L_{\textrm{SFT}}(\theta) = -\sum_{(x, y) \in D} \sum_{t=1}^{T} \log \pi_\theta(y_t \mid x, y_1,\ldots, y_{t-1})$$

The RL reward for output $y$ in state $x$ is defined as:

$$R(y \mid x) = \textrm{dup\_fac} \cdot [w_{\textrm{prop}} \cdot \textrm{prop\_smooth} + w_{\textrm{sim}} \cdot \textrm{sim\_fac}]$$

where:

• $w_{\textrm{prop}} = 0.8$, $w_{\textrm{sim}} = 0.2$ (chosen via grid search to maximize high-quality success rate [HQSR]),
• $$\textrm{prop\_smooth} = \frac{1}{3} \sum_{i=1}^3 \sigma((x_i - t_i)/k_i)$$ for property endpoints (LogD, MRT, SIF),
• $\textrm{sim\_fac} = \sigma(\alpha \cdot (s - s_0))$ (Tanimoto similarity $s$, with $s_0=0.6, \alpha=10$),
• $y$0 (duplication penalty with $y$1 prior occurrences, $y$2).

Reinforcement learning proceeds under the GRPO objective, which incorporates KL-regularization with coefficient $y$3:

$y$4

Ablation studies confirm that removal of KL-regularization or property weighting degrades output validity and diversity, and that CoT is essential to preserving rational, interpretable optimization trajectories.

4. Training Regimes, Data, and Hyperparameters

Training data comprises approximately 380,000 cyclic peptide pairs generated by performing 100 random single-HELM monomer edits on 3,778 seed peptides. Each pair is annotated with QSAR-predicted LogD, rat-MRT, and SIF stability, and stratified by the number of improved endpoints (single, dual, triple).

Fine-tuning uses 26,000 samples (triple-improvement plus up to 4,000 dual/single examples per group), with 10% validation. RL is seeded from 600 sampled SFT-train peptides. The held-out test set contains 1,880 pairs with <3 property improvements.

Hyperparameters include:

• SFT: epoch=3, lr=$y$5, batch=4, grad-accum=8, LoRA rank=8, $y$6, dropout=0.05.
• RL: batch=128 seeds × 8 rollouts, temp=1.0, lr=$y$7, $y$8, AdamW optimizer, total RL steps~10,000.

Qualitative validity and property improvements are tracked throughout, and RL policies are checkpointed against quantitative metrics detailed below.

5. Quantitative Evaluation and Benchmarks

PepThink-R1’s performance is summarized as follows (see (Wang et al., 20 Aug 2025) Table 1):

Model	Validity	Novelty	Uniqueness	HQSR	UHQS/seed	HQSR-S
Random mutation	0.876	0.998	0.991	0.003	0.61/100	0.208
GPT-5	0.821	0.996	0.959	0.107	0.85/10	0.386
CoT-SFT	0.956	0.999	0.954	0.196	8.82/50	0.879
SFT-RL	0.987	0.998	0.238	0.833	9.84/50	0.956
PepThink-R1	0.900	1.000