PolyLLMem: Multimodal Polymer Property Prediction

• PolyLLMem is a multimodal architecture that integrates domain-aware text embeddings from Llama 3 with 3D molecular features from Uni-Mol for polymer property prediction.
• The design leverages low-rank adaptation (LoRA) and a gated fusion module to refine and dynamically combine distinct embedding streams, yielding competitive performance with limited training data.
• By employing polymer SMILES for LLM tokenization and detailed molecular representations, PolyLLMem bridges text and structure modalities to accelerate polymer discovery.

PolyLLMem is a multimodal machine learning architecture designed for polymer property prediction, integrating domain-aware textual knowledge from LLMs with detailed molecular structural representations. It specifically addresses the challenge of limited availability of polymer-specific training data by leveraging text embeddings generated by Llama 3 alongside 3D structural embeddings produced by Uni-Mol. PolyLLMem employs low-rank adaptation (LoRA) layers to align and refine both streams for property prediction, achieving competitive or superior performance relative to existing graph-based and transformer-based models, especially under data-scarce conditions (Zhang et al., 29 Mar 2025).

1. Model Architecture

PolyLLMem combines two principal embedding streams: text-based polymer descriptions and 3D molecular structure encodings. The system processes polymers represented in polymer SMILES (PSMILES) format through Llama 3, extracting a 4096-dimensional text embedding via mean pooling over token-level features. In parallel, Uni-Mol generates a 1536-dimensional embedding encoding geometric and conformational features of the polymer.

Both embeddings are mapped into a common latent space using linear projections with GELU activations and batch normalization. LoRA layers introduce task-specific refinements to these embeddings. A gated fusion module then dynamically combines the refined textual and structural representations. The fused vector is further processed by a refinement block and input into a regression network—typically a multilayer perceptron (MLP)—to predict the target property.

The notational schema is as follows:

• Let $E_{\text{text}} \in \mathbb{R}^{4096}$ and $E_{\text{struct}} \in \mathbb{R}^{1536}$ denote the branch outputs.
• Projected and LoRA-refined embeddings:

$$z_{\text{text}} = \mathrm{GELU}(W_{\text{text}} E_{\text{text}} + b_{\text{text}}), \quad z_{\text{struct}} = \mathrm{GELU}(W_{\text{struct}} E_{\text{struct}} + b_{\text{struct}})$$

$$z_{\text{text}}' = z_{\text{text}} + \mathrm{LoRA}_{\text{text}}(z_{\text{text}}), \quad z_{\text{struct}}' = z_{\text{struct}} + \mathrm{LoRA}_{\text{struct}}(z_{\text{struct}})$$

• Gated fusion:

$$z_{\text{fused}} = \mathrm{Gate}(z_{\text{text}}', z_{\text{struct}}')$$

• Prediction:

$\hat{y} = f(z_{\text{fused}})$

This architecture enables flexible, data-efficient, and chemically informed integration of multimodal features for regression tasks.

2. Low-Rank Adaptation (LoRA) for Domain Adaptation

LoRA layers are incorporated prior to fusion to enable effective adaptation of pretrained embeddings for the polymer property domain, without the need for full fine-tuning of either the LLM or the structural encoder. The low-rank update mechanism injects chemical relevance by:

• Refining the syntactic and contextual information in Llama 3’s PSMILES embeddings to be more chemically meaningful.
• Adjusting Uni-Mol's geometric embedding stream to reflect subtle structural features specific to polymer chemistry.
• Preventing overfitting and enhancing robustness when training data is limited by constraining parameter updates to low-rank subspaces.

This design enables computationally efficient domain transfer, striking a balance between preservation of global chemical knowledge and injection of polymer-specific details.

3. Polymer SMILES Representation and Token Semantics

PolyLLMem employs the polymer SMILES (PSMILES) notation, a standardized extension of conventional SMILES for representing macromolecules, including repeating units, end groups, and connectivity. This representation is particularly advantageous because:

• It encapsulates complex polymer chemical structure and connectivity in a linearizable string, compatible with LLM tokenization schemes.
• Llama 3 extracts chemically meaningful token-level embeddings. Empirical analysis in the architecture shows that important motifs (e.g., aromatic rings, fluorinated groups, branch points) correspond to distinct embedding clusters.
• Quantitative cosine similarity analysis of LLM token embeddings reveals that the model captures subtle context-dependent chemical features, ensuring discriminability between similar but functionally different motifs.

This establishes PSMILES as an effective bridge between textual and structural modalities in polymer informatics.

4. Predictive Performance and Comparative Evaluation

PolyLLMem is benchmarked against graph-based models such as single-task polyGNN, and transformer-based approaches including PolymerBERT and TransPolymer, across 22 polymer properties:

• In glass transition temperature ($T_g$) prediction, PolyLLMem produces an averaged $R^2$ of $0.89 \pm 0.01$, matching or exceeding established SOTA models.
• For properties such as density ($\rho$), melting temperature ($T_m$), thermal decomposition temperature ($T_d$), bandgap ($E_{gc}, E_{gb}$), and gas permeability, PolyLLMem consistently yields $R^2$ scores $\geq 0.8$ using a training set of only $\sim$29,000 samples.
• Performance tables (for example, Table S5 in the Supplementary Information) underline that PolyLLMem frequently equals or outperforms models trained on two to three orders of magnitude more data.

This confirms the effectiveness of multimodal fusion and domain adaptation in breaking the prevailing dependence on extensive pretraining in polymer property modeling.

5. Addressing Data Scarcity Through Multimodal Fusion

PolyLLMem demonstrates that integrating LLM-based textual knowledge with 3D molecular geometry reduces the burden of large domain-specific datasets for accurate property prediction:

• The Llama 3 branch supplies broad, context-rich chemical knowledge learned from general materials science textual data.
• The Uni-Mol branch introduces modality-specific 3D structural details unobtainable from text alone.
• By projecting, refining, and fusing these sources, the model synergistically benefits from complementary modalities, efficiently leveraging cross-domain knowledge transfer.
• The combined architecture permits property prediction with substantially fewer labeled polymer samples than required by standalone transformer or graph-based models.
• This capability directly accelerates the discovery and optimization of advanced polymeric materials, particularly valuable when experimental or simulation data acquisition is costly or infeasible.

6. Future Research Directions

PolyLLMem's architecture highlights flexibility and extensibility; prospective research avenues outlined in the source include:

• Architectural optimization specifically for challenging mechanical properties (e.g., tensile strength, elongation at break) where current results show potential for improvement.
• Incorporation of advanced token-level modeling using multi-head attention to capture richer intra-token chemical dependencies.
• Broadening the multimodal paradigm to incorporate additional data modalities (e.g., spectroscopy, microscopy) and implementing data augmentation to further alleviate data scarcity.
• Investigating multitask and transfer learning paradigms as in PolymerBERT for broader chemical property coverage.
• Systematic expansion to new and more diverse polymer datasets, as well as the paper of rare or edge-case polymers.

A plausible implication is that further refinement and expansion of the PolyLLMem approach will generalize multimodal LLM-based workflows to other domains with analogous data limitations.

7. Significance and Impact

PolyLLMem establishes a template for integrating state-of-the-art LLM and molecular structure encoding in polymer informatics. Its domain-aware fusion mechanism, underpinned by LoRA adaptation and judicious use of PSMILES representations, enables accurate and data-efficient prediction of a diverse set of polymer properties. This approach demonstrates that high predictive fidelity in polymer property estimation does not inherently require massive task-specific data collection, but can be achieved through principled multimodal integration and adaptation of pretrained domain knowledge sources. This suggests broad applicability of PolyLLMem-style multimodal architectures to other data-scarce areas within computational materials science and chemistry.