OOD-GraphLLM: Graph Large Language Model for Out-of-Distribution Generalized Drug Synergy Prediction

Abstract: Drug synergy prediction (DSP) aims to identify efficacious drug combinations under various cellular contexts with different targets. However, the continual emergence of novel compounds results in variations in molecular scaffolds and sizes, causing drug synergy data to exhibit out-of-distribution (O.O.D.) shifts with respect to topological structure. Existing works rely on in-distribution (I.D.) assumption, failing to handle the O.O.D. shifts. To solve this problem, we study out-of-distribution generalized drug synergy prediction through a graph LLM for the first time. Nevertheless, O.O.D. generalized DSP is highly non-trivial, posing several challenges: i) how to discover structurally relevant and irrelevant molecular representations with respect to cell targets; ii) how to find the optimal graph neural architectures that accurately calculate molecular representations; and iii) how to jointly leverage molecular structural and semantic information in LLMs. To address these challenges, we propose OOD-GraphLLM, a novel graphLLM framework which is able to accurately predict drug synergy under O.O.D. settings via jointly optimizing molecular graph representation and biomedical semantic language representations in a unified manner. Furthermore, we finetune DrugSyn-LLM, a biomedical LLM, and employ a retrieval-augmented biomedical instruction tuning strategy to align molecular topological information and molecular semantic information with language-based reasoning for O.O.D. generalized DSP. Both the source code (https://github.com/EkkoXiao/Bio-GraphLLM) and released model (https://mn.cs.tsinghua.edu.cn/bio-graphllm/) are publicly available, where users are allowed to download model resources and interactively use the system through a web interface.

• The paper introduces OOD-GraphLLM, which unifies contextualized molecular graph encoding, architecture search, and language reasoning to tackle out-of-distribution drug synergy prediction.
• It employs disentangled representation learning and pairwise attentive neural architecture search to enhance transferability and overcome performance degradation on unseen chemical spaces.
• Experimental evaluations reveal over 5% gains in classification and significant reductions in regression errors, underscoring robust performance in OOD settings.

OOD-GraphLLM: A Graph LLM for Out-of-Distribution Generalized Drug Synergy Prediction

Introduction

Out-of-distribution (O.O.D.) shifts in molecular scaffolds and compound sizes present a significant challenge in drug synergy prediction (DSP), as real-world drug discovery frequently introduces novel chemotypes that diverge from training distributions. Traditional DSP approaches, both descriptor-based and GNN-based, generally assume that the training and testing molecular topologies are drawn from similar distributions, thus exhibiting substantial performance degradation under O.O.D. settings. The OOD-GraphLLM framework directly addresses this limitation by unifying contextualized molecular graph encoding, architectural search, and language-based reasoning to robustly predict drug synergies in O.O.D. contexts. The method leverages a multi-component pipeline, embedding molecular graphs, cellular context, and biomedical priors, and optimizes a LLM-centric end-to-end architecture suited for the complex structure-activity landscapes of unseen drugs.

Methodology

Overall Framework

OOD-GraphLLM comprises four jointly optimized constituents:

1. Target-Adaptive Disentangled Molecular Graph Encoding: Distinguishes target-relevant and target-irrelevant molecular representations via disentanglement constraints and target-protein conditioned attention. This enables transferability of chemical–biological interactions for unseen drugs and targets.
2. Pairwise Attentive Graph Architecture Search: Employs an operator-level neural architecture search paradigm to dynamically route information through an operator latent space, conditionally optimizing GNN architecture for each drug pair. Operator representations are regularized by layerwise separation, ensuring functionally diverse message passing.
3. Multi-Level Contextualized Cellular Feature Alignment: Integrates cell line-specific information both structurally (feature concatenation at the graph level) and semantically (alignment to the LLM embedding input space via textual and omic features), maximizing contextual fidelity of synergy predictions.
4. Finetuned DrugSyn-LLM with Retrieval-Augmented Biomedical Instruction Tuning: Implements a two-stage LLM adaptation, first grounding the base LLM in retrieved biomedical evidence through instruction tuning, then transitioning to explicit synergy task supervision with multi-modal (graph, textual, omic) prompt–response training.

   Figure 1: The OOD-GraphLLM framework jointly optimizes contextual graph encoding, architecture search, contextual cell line alignment, and LLM-based prediction for O.O.D. generalized DSP.

This end-to-end design ensures that structural, contextual, and semantic representations are tightly coupled, supporting robust generalization under chemical topology shifts and diverse biological backgrounds.

Technical Innovations

• Disentangled Representation Learning: The targeted decomposition of molecular encodings into components relevant and irrelevant to specific targets, enforced with an explicit decorrelation loss, is critical for preserving biologically meaningful features under O.O.D. shifts.
• Pairwise Attentive NAS: Dynamic routing in the operator latent space prevents overfitting to biases in the training graph architectures and supports adaptive representation across novel ligand pairs.
• Contextualized Feature Alignment: The inclusion of both gene expression and cell description data ensures the transferability of synergy predictions across heterogeneous cellular environments.
• Retrieval-Augmented Tuning: Biomedical evidence retrieval and integration during instruction tuning injects domain-specific priors, allowing the LLM to perform explicit chain-of-thought reasoning across molecular, target, and cellular modalities.

Experimental Evaluation

Data Splits and Chemical Diversity

Extensive O.O.D. splits were constructed on DrugComb data, stratified by molecular scaffold and molecular size, with up to four synergy metrics (Bliss, HSA, Loewe, ZIP). Distribution analyses based on t-SNE reveal clear separation between training and test chemical spaces in O.O.D. setups, as opposed to the standard random splits with large overlaps.

Figure 2: Visualization of chemical space shows effective separation between I.D. and O.O.D. splits, highlighting the challenge for robust generalization.

Baseline Comparisons

OOD-GraphLLM was benchmarked against DNN-based, GNN-based, and LLM-based state-of-the-art systems, including DeepSynergy, DeepDDS, AttenSyn, TreeCombo, CancerGPT, and BAITSAO. Metrics span both classification (ACC, AUC) and regression (MAE, RMSE) settings across multiple datasets and O.O.D. splits.

Key findings:

• OOD-GraphLLM achieves superior performance across all tasks and splits, often by >5% in classification and >10% in regression margin over the strongest baselines.
• Notable classification gains: On Loewe (Scaffold O.O.D.), OOD-GraphLLM yields 93.55% ACC and 86.19% AUC, outperforming the prior best by a non-trivial margin.
• Regression robustness: Relative improvements in MAE and RMSE exceed 20% in challenging datasets, reflecting effective modeling of nuanced synergy trends in unseen chemical regions.

  Figure 3: Comparative performance visualization between prior approaches and OOD-GraphLLM in O.O.D. settings.

Ablation and Sensitivity

Comprehensive ablation demonstrates that the exclusion of any single module (context features, architectural search, disentanglement, retrieval-augmented instruction) significantly reduces model accuracy and increases error, underlining the architectural interdependence of design choices.

Figure 4: Ablation studies quantify the necessity of each architectural component for O.O.D. generalization.

Hyperparameter analyses further show that both decorrelation and operator separation losses ($\alpha$, $\beta$) are essential for optimal performance, while the system remains robust across various prompt strategies and architecture search hyperparameters.

Figure 5: Classification and regression performance remain robust across key hyperparameter variations and input prompt formulations.

Case and Interpretability Studies

Case studies on 5-Fluorouracil and Vorinostat in the NCI-H226 cell line (Bliss metric) showcase the system's ability to align biochemical mechanism (e.g., SMN or HDAC4 modulation) across structurally divergent molecules, a hallmark of robust O.O.D. generalization.

Figure 6: Case study elucidates OOD-GraphLLM's mechanistic reasoning for a highly O.O.D. scaffold pair.

Further, interpretability analyses indicate:

• Operator preferences (e.g., GINmol for small, hetero-atom-rich scaffolds; GATmol/GRAPHmol for aromatic rings) correlate with chemical substructure.
• Attention heatmaps on SMILES sequences localize to pharmacologically salient fragments, supporting semantic alignment between learned representations and medicinal chemistry intuition.

  Figure 7: Operation preference visualizations highlight structure-dependent routing in the graph architecture search space.

  

  Figure 8: Substructure-level SMILES attention heatmaps map internal model reasoning to established pharmacophores.

Implications and Future Directions

The demonstrated OOD-GraphLLM framework establishes a new technical paradigm for robust DSP under realistic drug discovery constraints, advancing both methodological and practical frontiers:

• Scalable O.O.D. Adaptation: Integration of GNN and LLM architectures via context-aware, dynamically routed pipelines will likely become standard practice for biopharmaceutical AI workflows where out-of-domain chemistry is routine.
• Explainability: The fine-grained interpretability at target and chemical substructure levels supports regulatory and mechanistic transparency requirements for clinical pipeline deployment.
• Extensibility: The modular nature of OOD-GraphLLM offers avenues for further augmentation, e.g., adaptive prompt engineering, unsupervised context retrieval, or multi-scale omics integration.

These results invite future exploration in:

• Transferable architectures across larger combinatorial chemical and biological hyperspaces.
• Automated multi-task learning using cross-modal feedback, supporting joint optimization of efficacy, safety, and ADME properties.
• Expanded studies in higher-order drug combinations and temporal adaptation under clinical trial settings.

Conclusion

OOD-GraphLLM presents a rigorous, LLM-centric framework integrating graph reasoning, NAS, and context alignment for O.O.D. DSP, achieving consistent superiority over state-of-the-art models on both classification and regression tasks in chemically distant testing spaces. The joint disentanglement of biological targets, adaptive GNN routing, and retrieval-grounded language supervision is critical for robust generalization, and the architectural insights herein have broad implications for AI-driven biomedical modeling and translational drug discovery.