Chem3DLLM: Protein-Conditioned 3D LLM

• Chem3DLLM is a unified model that integrates reversible 3D molecular text encoding, protein embedding, and reinforcement learning to generate chemically valid conformations.
• The system overcomes tokenization challenges by converting 3D structures into discrete text sequences, achieving a greater than 3x compression ratio with high stability.
• It demonstrates state-of-the-art performance in structure-based drug design with a Vina score of -7.21 and sub-angstrom RMSD protein-ligand fits.

Chem3DLLM is a unified protein-conditioned multimodal LLM for chemistry that targets a specific limitation of conventional autoregressive molecular modeling: the inability of standard LLMs to natively generate 3D molecular conformations while jointly handling proteins, ligands, and text. The model is built to address three stated obstacles: 3D molecular structures are incompatible with LLMs’ discrete token space, integrating heterogeneous inputs such as proteins, ligands, and text is difficult within a unified model, and LLMs lack essential scientific priors for enforcing physical and chemical constraints during generation. Its central design combines a reversible text encoding for 3D molecular structures, a lightweight protein embedding projector, and reinforcement learning with stability-based rewards. On structure-based drug design, it reports state-of-the-art performance with a Vina score of $-7.21$ (Jiang et al., 14 Aug 2025).

1. Conceptual scope

Chem3DLLM is formulated around the premise that, in the real world, a molecule is a 3D geometric structure, and that 3D molecules are the most informative molecular modality relative to 1D SMILES sequences and 2D molecular graphs (Jiang et al., 14 Aug 2025). Rather than treating geometry as an auxiliary signal attached to a text-only generator, the model makes 3D structure itself a first-class generative target.

The system is autoregressive. Given a textual prompt and/or a protein pocket $\mathcal{P}$, it generates a token sequence $\mathcal{M}$ that encodes the 3D molecule:

$$P(\mathcal{M}\mid \mathbf{T}, \mathcal{P}) \;=\;\prod_{i=1}^{|\mathcal{M}|} P\bigl(m_i\mid m_{<i},\,[\mathbf{T};\,\mathbf{h}_{\rm aligned}];\theta\bigr)\,.$$

This design places protein-conditioned ligand generation, molecular conformation generation, and text-conditioned molecular generation inside a single LLM architecture rather than separating sequence generation, conformer prediction, and docking into independent modules.

A common misconception is that Chem3DLLM is simply a SMILES-based generator with a downstream conformer post-processor. The model description does not support that reading. Its defining claim is a reversible encoder/decoder that converts 3D molecular graphs into discrete token sequences and back, so that 3D structure can be generated directly in the model’s native autoregressive space (Jiang et al., 14 Aug 2025).

2. Core architecture

Chem3DLLM is built on Qwen2-7B and introduces two additional modules: RCMT, a reversible encoder/decoder for 3D molecular graphs, and a protein-structure projector that maps a protein binding pocket into the LLM hidden space (Jiang et al., 14 Aug 2025). The result is a multimodal stack in which protein information is aligned to the LLM while molecular geometry is serialized into a reversible textual form.

The protein side begins with a pretrained ESM-based protein LLM $\mathcal{E}_{\rm prot}$, which embeds pocket residues to $\mathbf{h}_{\rm pocket}\in\mathbb{R}^{n\times d_p}$. A 2-layer MLP with hidden size $3584$ and GELU activation projects these representations into the LLM hidden dimension:

$$\mathbf{h}_{\rm aligned} = \mathrm{GELU}\bigl(\mathbf{h}_{\rm pocket}W_1 + b_1\bigr)W_2 + b_2 \quad\in\mathbb{R}^{n\times d_h}.$$

These aligned vectors are then prepended or interleaved as “virtual tokens,” allowing long-range attention to jointly process $[\mathbf{T};\mathbf{h}_{\rm aligned}]$ in a unified embedding space.

The training protocol distinguishes supervised and reinforcement stages. In supervised fine-tuning, the ESM-based protein encoder is frozen, and only the projection MLP and the LLM weights $\theta$ are updated by maximizing next-token log-likelihood. In the second stage, reinforcement learning with scientific feedback refines $\mathcal{P}$0 via PPO (Jiang et al., 14 Aug 2025). This division of labor preserves a fixed protein representation backbone while adapting the generative policy to chemical and physical constraints.

3. RCMT and reversible 3D molecular text

RCMT is the model’s mechanism for making 3D molecular structure compatible with an autoregressive LLM. A molecule is represented as $\mathcal{P}$1 with $\mathcal{P}$2 atoms, and the goal is a lossless mapping $\mathcal{P}$3 into a compact text sequence together with an inverse $\mathcal{P}$4 (Jiang et al., 14 Aug 2025).

The encoding first quantizes each atom’s 3D coordinate $\mathcal{P}$5 at step size $\mathcal{P}$6:

$\mathcal{P}$7

For atom $\mathcal{P}$8, a token $\mathcal{P}$9 is formed by concatenating atom identity and quantized coordinate. Bond connectivity is represented through a bond-order adjacency matrix $\mathcal{M}$0, where $\mathcal{M}$1 denote single, double, triple, and aromatic bonds, and $\mathcal{M}$2 denotes the absence of a bond. Because $\mathcal{M}$3 is sparse, run-length encoding is applied to the list of nonzero entries:

$\mathcal{M}$4

Decoding is explicit: each $\mathcal{M}$5 is parsed to recover atom type and quantized coordinate, and the edge set $\mathcal{M}$6 is reconstructed from the triplets $\mathcal{M}$7 (Jiang et al., 14 Aug 2025). The representation is therefore not a heuristic textual description but a reversible molecular serialization.

This encoding is also a compression scheme. The abstract reports a 3x size reduction while preserving complete structural information, and the detailed description gives an empirical compression ratio $\mathcal{M}$8 on QM9 (Jiang et al., 14 Aug 2025). This suggests that Chem3DLLM’s token budget is not only structurally faithful but also economically matched to the context constraints of an LLM.

4. Protein conditioning and reinforcement learning with scientific feedback

Chem3DLLM’s multimodality is not limited to concatenating protein-derived embeddings with molecular text. It also introduces a reward design intended to encode scientific priors during generation. The reward for a generated molecule $\mathcal{M}$9 is

$$P(\mathcal{M}\mid \mathbf{T}, \mathcal{P}) \;=\;\prod_{i=1}^{|\mathcal{M}|} P\bigl(m_i\mid m_{<i},\,[\mathbf{T};\,\mathbf{h}_{\rm aligned}];\theta\bigr)\,.$$0

where $$P(\mathcal{M}\mid \mathbf{T}, \mathcal{P}) \;=\;\prod_{i=1}^{|\mathcal{M}|} P\bigl(m_i\mid m_{<i},\,[\mathbf{T};\,\mathbf{h}_{\rm aligned}];\theta\bigr)\,.$$1 penalizes high total energy, $$P(\mathcal{M}\mid \mathbf{T}, \mathcal{P}) \;=\;\prod_{i=1}^{|\mathcal{M}|} P\bigl(m_i\mid m_{<i},\,[\mathbf{T};\,\mathbf{h}_{\rm aligned}];\theta\bigr)\,.$$2 rewards per-atom stability, $$P(\mathcal{M}\mid \mathbf{T}, \mathcal{P}) \;=\;\prod_{i=1}^{|\mathcal{M}|} P\bigl(m_i\mid m_{<i},\,[\mathbf{T};\,\mathbf{h}_{\rm aligned}];\theta\bigr)\,.$$3 encourages diversity, and $$P(\mathcal{M}\mid \mathbf{T}, \mathcal{P}) \;=\;\prod_{i=1}^{|\mathcal{M}|} P\bigl(m_i\mid m_{<i},\,[\mathbf{T};\,\mathbf{h}_{\rm aligned}];\theta\bigr)\,.$$4 checks chemical validity (Jiang et al., 14 Aug 2025). The specification $$P(\mathcal{M}\mid \mathbf{T}, \mathcal{P}) \;=\;\prod_{i=1}^{|\mathcal{M}|} P\bigl(m_i\mid m_{<i},\,[\mathbf{T};\,\mathbf{h}_{\rm aligned}];\theta\bigr)\,.$$5 makes stability the prioritized objective.

The optimization uses PPO with the clipped surrogate loss

$$P(\mathcal{M}\mid \mathbf{T}, \mathcal{P}) \;=\;\prod_{i=1}^{|\mathcal{M}|} P\bigl(m_i\mid m_{<i},\,[\mathbf{T};\,\mathbf{h}_{\rm aligned}];\theta\bigr)\,.$$6

where $$P(\mathcal{M}\mid \mathbf{T}, \mathcal{P}) \;=\;\prod_{i=1}^{|\mathcal{M}|} P\bigl(m_i\mid m_{<i},\,[\mathbf{T};\,\mathbf{h}_{\rm aligned}];\theta\bigr)\,.$$7 is the policy ratio and $$P(\mathcal{M}\mid \mathbf{T}, \mathcal{P}) \;=\;\prod_{i=1}^{|\mathcal{M}|} P\bigl(m_i\mid m_{<i},\,[\mathbf{T};\,\mathbf{h}_{\rm aligned}];\theta\bigr)\,.$$8 is the advantage estimate (Jiang et al., 14 Aug 2025). In the reported training setup, supervised fine-tuning initializes the policy, and reinforcement learning subsequently biases generation toward stable and valid 3D structures.

This aspect of the model addresses another misconception: Chem3DLLM is not presented as a purely next-token statistical learner. The reward function explicitly incorporates stability, diversity, and validity, and the paper frames this as a response to the lack of essential scientific priors in standard LLMs (Jiang et al., 14 Aug 2025).

5. Datasets and training procedure

Chem3DLLM is trained on QM9 and CrossDocked. QM9, comprising $$P(\mathcal{M}\mid \mathbf{T}, \mathcal{P}) \;=\;\prod_{i=1}^{|\mathcal{M}|} P\bigl(m_i\mid m_{<i},\,[\mathbf{T};\,\mathbf{h}_{\rm aligned}];\theta\bigr)\,.$$9K small molecules, is used for molecular conformation generation, while CrossDocked, with $\mathcal{E}_{\rm prot}$0K protein-ligand pairs, is used for structure-based drug design (Jiang et al., 14 Aug 2025). This division reflects the model’s dual emphasis on intrinsic molecular geometry and protein-conditioned ligand generation.

The training procedure has two stages. The first stage is supervised fine-tuning on CrossDocked for $\mathcal{E}_{\rm prot}$1 epochs with cross-entropy loss,

$\mathcal{E}_{\rm prot}$2

The second stage is RL fine-tuning on QM9 for $\mathcal{E}_{\rm prot}$3 PPO steps using the stability reward (Jiang et al., 14 Aug 2025). The conceptual total objective is

$\mathcal{E}_{\rm prot}$4

The reported optimizer and hyperparameters are Adam with learning rate $\mathcal{E}_{\rm prot}$5, batch size $\mathcal{E}_{\rm prot}$6, PPO $\mathcal{E}_{\rm prot}$7, discount $\mathcal{E}_{\rm prot}$8, and a protein projector with $\mathcal{E}_{\rm prot}$9 MLP layers and hidden size $\mathbf{h}_{\rm pocket}\in\mathbb{R}^{n\times d_p}$0. Training uses the Qwen2-7B base model on $\mathbf{h}_{\rm pocket}\in\mathbb{R}^{n\times d_p}$1A800 GPUs (Jiang et al., 14 Aug 2025). These details situate Chem3DLLM as a full-scale multimodal LLM rather than a parameter-efficient adaptation of a smaller chemistry backbone.

6. Experimental performance

On molecular conformation generation over QM9, Chem3DLLM is evaluated with atom-level stability, molecule-level stability, validity, and uniqueness. It reports $\mathbf{h}_{\rm pocket}\in\mathbb{R}^{n\times d_p}$2 atom stability, $\mathbf{h}_{\rm pocket}\in\mathbb{R}^{n\times d_p}$3 molecular stability, and $\mathbf{h}_{\rm pocket}\in\mathbb{R}^{n\times d_p}$4 validity and uniqueness, surpassing diffusion and equivariant baselines including GeoLDM, EDM-Bridge, and RDKit (Jiang et al., 14 Aug 2025). These metrics are notable because they combine geometric plausibility with exact generative validity.

On structure-based drug design over CrossDocked, docking evaluations use AutoDock Vina, with average, median, and minimum score after pose optimization as reported metrics. In single-task SBDD, Chem3DLLM obtains $\mathbf{h}_{\rm pocket}\in\mathbb{R}^{n\times d_p}$5 kcal/mol and $\mathbf{h}_{\rm pocket}\in\mathbb{R}^{n\times d_p}$6, beating MolCRAFT at $\mathbf{h}_{\rm pocket}\in\mathbb{R}^{n\times d_p}$7 as well as other baselines including AR, Pocket2Mol, and DecompDiff (Jiang et al., 14 Aug 2025). In the joint multi-task setting combining molecular conformation generation and SBDD, performance further improves to $\mathbf{h}_{\rm pocket}\in\mathbb{R}^{n\times d_p}$8 and $\mathbf{h}_{\rm pocket}\in\mathbb{R}^{n\times d_p}$9.

The qualitative analysis reports that visualizations confirm tight protein-ligand fits with Vina $3584$0 and sub-angstrom RMSD for reconstructed conformations (Jiang et al., 14 Aug 2025). The stated interpretation is that the auxiliary conformation task injects geometric bias and yields a synergistic boost in the joint setting. A plausible implication is that the model benefits from sharing geometric competence across unconditional conformer generation and protein-conditioned design rather than learning docking-oriented generation in isolation.

7. Position within 3D molecular LLM research, limitations, and future directions

Chem3DLLM belongs to a broader line of 3D molecular language modeling, but its emphasis is distinct. “Towards 3D Molecule-Text Interpretation in LLMs” introduces 3D-MoLM, which integrates a pretrained 3D-GNN, a Q-Former, and Llama2-7B for 3D molecule-text interpretation, retrieval, captioning, and open-text molecular QA, especially for 3D-dependent properties (Li et al., 2024). That system is interpretation-centric rather than protein-conditioned 3D generation-centric.

“NExT-Mol: 3D Diffusion Meets 1D Language Modeling for 3D Molecule Generation” combines a $3584$1 M-parameter SELFIES LM (“MoLlama”) with a 3D diffusion transformer (“DMT”), yielding a $3584$2 relative improvement in 3D-FCD on GEOM-DRUGS and a $3584$3 average relative gain for conditional 3D generation on QM9-2014 (Liu et al., 18 Feb 2025). Its design factorizes 1D molecular generation and 3D conformer prediction, whereas Chem3DLLM unifies 3D molecular tokenization and protein conditioning inside a single autoregressive LLM architecture.

“SmileyLlama” modifies Llama-3.1-8B-Instruct into a chemical LLM for SMILES generation via supervised fine-tuning and direct preference optimization, but the published paper does not actually couple its SMILES LLM with a physics-based or docking-based RL framework, nor does it generate 3D conformers or perform explicit docking (Cavanagh et al., 2024). This contrast clarifies that Chem3DLLM’s novelty lies not in property-steered string generation alone, but in multimodal 3D generation with protein context and stability-driven RL.

“MolSpectLLM” extends the multimodal foundation-model direction by converting spectra and 3D structure into unified textual descriptions, achieving $3584$4 sequence accuracy and $3584$5 token accuracy on Spectra-to-SMILES and generating 3D molecular structures directly from SMILES or spectral inputs (Shen et al., 26 Sep 2025). Relative to that spectroscopy-centric system, Chem3DLLM is centered on protein-conditioned 3D generation and structure-based drug design.

The limitations reported for Chem3DLLM are specific. The current model size of $3584$6B constrains scaling to ultra-large libraries and very large proteins, and RL fine-tuning is only $3584$7 steps; longer RL or multi-task pretraining may improve robustness (Jiang et al., 14 Aug 2025). The stated future directions are to scale to larger LLM backbones and larger chemical spaces beyond $3584$8 heavy atoms, integrate experimental assay feedback in the loop for end-to-end lead optimization, and explore more expressive protein tokenizations such as 3D surface embeddings together with joint pretraining on protein–ligand–text corpora (Jiang et al., 14 Aug 2025). These directions indicate that Chem3DLLM is best understood as an early unified 3D multimodal chemistry LLM whose central contribution is architectural: it brings reversible 3D molecular serialization, protein-pocket alignment, and scientific-feedback RL into one generative framework.