MolCLIP: Drug MoA Recognition Framework

• MolCLIP is a multi-modal framework that fuses time-lapsed mitochondrial images with drug SMILES to enable robust mechanism of action recognition using integrated visual and chemical signals.
• Its design leverages a dual encoder setup with I3D ResNet-50 for video and MolFormer for SMILES, coupled with a CLIP-inspired alignment branch and metric-learning strategies.
• Empirical results demonstrate significant improvements in drug identification and MoA classification accuracy compared to conventional methods, validating its category-aware supervision.

Searching arXiv for recent MolCLIP-related papers and closely related molecule–text CLIP frameworks. MolCLIP is a multi-modal, CLIP-inspired framework for drug mechanism of action (MoA) recognition from time-lapsed mitochondrial image sequences. It was introduced as “the first visual LLM to combine microscopic cell video- and molecule-modalities,” pairing microscopic cell videos with drug SMILES so that cellular dynamics and molecular structure can be learned jointly for MoA-related inference (Pang et al., 10 Jul 2025). In contrast to molecule–text systems, MolCLIP uses the molecular modality as an auxiliary signal for organizing video representations, with the stated objective of guiding video features toward a chemically meaningful latent space and improving both drug identification and MoA recognition on MitoDataset (Pang et al., 10 Jul 2025).

1. Problem formulation and motivation

MolCLIP is motivated by two observations about MoA recognition. First, prior methods based on high-content or fluorescence images emphasize spatial morphology but overlook the temporal dynamics of live cells, even though drug–cell interaction is described as a time-sustained process. Second, prior work did not “deeply utilize drug information,” despite the fact that drug molecules can trigger cellular dynamic variations related to specific MoA (Pang et al., 10 Jul 2025).

The model therefore fuses two modalities: time-lapsed cell sequences, representing how cells respond over time, and drug SMILES, representing what the molecule is. The paper frames this combination as a better fit to MoA than static image-only pipelines because time-lapse videos can reveal dynamic phenotypic changes not visible in a single frame, while the molecular modality carries complementary mechanistic information (Pang et al., 10 Jul 2025).

A common misconception is to treat MolCLIP as a generic molecule–text CLIP model. The paper does not define it that way. Its paired modalities are microscopic cell video and molecule modality, and its central application is MoA recognition rather than text retrieval, captioning, or property prediction (Pang et al., 10 Jul 2025).

2. Architectural organization

MolCLIP operates on batches of paired samples

$\{S_i, V_i, L_{drug}, L_{MoA}\}_{i=1}^{B},$

where $S_i$ is the SMILES of the $i$-th drug, $V_i$ is the corresponding time-lapsed sequence, $L_{drug}$ is the drug type label, and $L_{MoA}$ is the MoA label (Pang et al., 10 Jul 2025).

The architecture has two encoders. The SMILES encoder $E_s(\cdot)$ is MolFormer. SMILES are tokenized using a vocabulary of 2,362 tokens, passed through a Transformer-based encoder, and mapped to features in $\mathbb{R}^{1 \times 2048}$. The time-lapsed sequence encoder $E_v(\cdot)$ is I3D ResNet-50. Its input is a sequence $V \in \mathbb{R}^{T \times H \times W \times 3}$, where each sequence contains $S_i$0 RGB frames and the default spatial size is $S_i$1; its output feature dimension is likewise $S_i$2 (Pang et al., 10 Jul 2025).

At a high level, the model has a CLIP-like alignment branch and a metric-learning branch. The paper calls the system “molecular-auxiliary” because the drug modality serves as an auxiliary supervisory signal for the cell-video representation rather than as a conventional text branch. A linear layer acts as the classification head on the video side (Pang et al., 10 Jul 2025).

This design implies a specific asymmetry between modalities. The molecular encoder supplies chemically structured supervision, while the visual branch is the principal target of representation shaping. A plausible implication is that MolCLIP should be understood less as a symmetric dual-encoder retrieval system than as a drug-guided phenotype learning framework.

3. Category-aware CLIP supervision

The paper argues that standard CLIP supervision is insufficient for MoA because the task is not naturally one-to-one. One drug may correspond to multiple videos, and different drugs may share the same MoA. Under that setting, an ordinary matched-pair supervision matrix creates what the paper calls semantic confusion (Pang et al., 10 Jul 2025).

To address this, MolCLIP replaces ordinary CLIP supervision with a Self- and Class-awareness symmetry Matrix (SCM). The self-awareness component is

$S_i$3

and the class-awareness component is

$S_i$4

The similarity matrix between drug and video features is

$S_i$5

The CLIP-branch loss is then

$S_i$6

(Pang et al., 10 Jul 2025)

The role of this formulation is to align not only each video with its paired drug but also samples sharing the same MoA category. In the paper’s own interpretation, the drug branch provides class-aware structure that helps video embeddings move toward a more chemically meaningful latent manifold. This is the principal mechanism by which MolCLIP departs from vanilla CLIP for MoA recognition (Pang et al., 10 Jul 2025).

4. Metric learning and feature aggregation

MolCLIP explicitly states that alignment alone is insufficient. The CLIP-style branch introduces drug information, but it does not fully constrain the internal geometry of the cell-video feature space. For that reason, the model adds a metric-learning branch to improve aggregation and clustering of video features (Pang et al., 10 Jul 2025).

Three losses are used in this branch: hard triplet loss, center loss, and classification cross-entropy loss. The hard triplet loss is

$S_i$7

where $S_i$8 is the anchor, $S_i$9 is the hardest positive sample, $i$0 is the hardest negative sample, and the default margin $i$1 is 0.3. The center loss is

$i$2

where $i$3 denotes the feature center of the ground-truth class (Pang et al., 10 Jul 2025).

The paper interprets these terms as complementary: triplet loss controls relative distances, whereas center loss enforces absolute compactness around class centers. It further reports that performance improves as the center-loss coefficient increases from 0.01 to 0.1, with 0.1 identified as the best overall setting in the reported study (Pang et al., 10 Jul 2025).

Training follows a two-stage procedure. MolCLIP is first pre-trained using drug labels with no parameters frozen. It is then fine-tuned using MoA labels, with the SMILES encoder frozen and only the video side participating in MoA fine-tuning (Pang et al., 10 Jul 2025). This indicates that cross-modal training is used to inject molecular structure into the phenotype space before specialization to the final MoA task.

5. Dataset, implementation, and evaluation protocol

MolCLIP is evaluated on MitoDataset, released by Yu et al. The dataset contains 35,631 time-lapse sequences, 1,068 FDA-approved drugs plus DMSO treatments, and a total of 570,096 single-cell images. Each sequence contains 16 frames. MoA labels are sourced from Drug Repurposing Hub, ChEMBL, and DrugBank; 38 drug MoAs are retained, with an 80% training and 20% test split. In the test set, one sample from each drug MoA is randomly selected as the query set and the rest become the gallery set (Pang et al., 10 Jul 2025).

Because MitoDataset does not contain SMILES, the authors collected SMILES from the same sources and normalized them using RDKit. Implementation details reported in the paper include an Nvidia 3090 GPU, an improved I3D ResNet-50 as the video encoder, MolFormer XL as the molecular encoder, 500 epochs for training and pretraining, batch size 64, SGD optimization, and learning rate 0.001 (Pang et al., 10 Jul 2025).

The evaluation uses a CMC curve, Accuracy, and mean Average Precision (mAP). The paper presents both drug-recognition-oriented results and MoA identification results, which is important for interpreting the reported gains: some experiments quantify the effect of molecularly informed pretraining, while the main comparison evaluates final MoA recognition against conventional machine learning methods, sequence-only deep models, vanilla CLIP, and MitoReID (Pang et al., 10 Jul 2025).

6. Empirical performance and comparative interpretation

The paper reports several layers of empirical evidence. In a pretraining strategy comparison, “sequence only” yields Rank-1 33.64, mAP 29.11, and Acc 75.22, whereas “sequence + drug” yields Rank-1 40.20, mAP 37.20, and Acc 83.67, corresponding to improvements of +6.56% Rank-1, +8.09% mAP, and +8.45% Accuracy. A third strategy, described as stronger initialization / metric learning effect, reaches Rank-1 89.50, mAP 80.30, and Acc 84.00 (Pang et al., 10 Jul 2025).

The center-loss study reports that the best center-loss coefficient is 0.1, with drug recognition accuracy 91.84%, which is 7.84% higher than strategy 3 in Table 1. The paper notes that this slightly lowers Rank-1 relative to strategy 3 by 0.7%, while improving accuracy and avoiding extra pretraining cost (Pang et al., 10 Jul 2025).

For MoA identification, the main comparison includes Random Forest, SVM, MLP, LSTM, MitoReID, CLIP, and MolCLIP. Reported results are: Random Forest 15.72% Accuracy; SVM 10.76%; MLP 12.65%; LSTM 12.18%; MitoReID 82.46% Accuracy, Rank-1 76.32, mAP 65.92; CLIP 69.91% Accuracy, Rank-1 0.00, Rank-51 2.80, Rank-101 3.70, mAP 1.10; and MolCLIP 93.86% Accuracy, Rank-1 91.20, Rank-51 94.70, Rank-101 94.70, mAP 86.40 (Pang et al., 10 Jul 2025).

The abstract summarizes these outcomes as improvements of 51.2% and 20.5% in mAP for drug identification and MoA recognition, respectively. The paper’s interpretive claim is that ordinary CLIP is not suited to MoA without category-aware supervision, whereas MolCLIP’s SCM and metric-learning components supply the class structure and feature compactness required by the task (Pang et al., 10 Jul 2025).

7. Position within CLIP-derived molecular multimodal research

MolCLIP belongs to a broader family of CLIP-derived multimodal models, but its modality pairing and downstream objective are distinctive. A closely related contrastive molecular framework is GeomCLIP, which aligns ground-state 3D molecular geometry with biomedical text using a dual-encoder design and adds a denoising objective to preserve unimodal geometric learning. GeomCLIP is therefore a molecule–text or geometry–text system for representation learning, retrieval, captioning, and molecular property prediction, rather than a drug–video system for MoA recognition (Xiao et al., 2024).

This distinction matters because “MolCLIP” can be misread as a generic label for molecule–text contrastive pretraining. In the literature provided here, MolCLIP refers specifically to the drug-video framework based on time-lapsed mitochondrial images, whereas GeomCLIP is the 3D geometry-first variant in the molecule–text setting (Pang et al., 10 Jul 2025). By contrast, other CLIP-derived models such as CLIPMH and CLIPErase address multimedia hashing and CLIP unlearning, respectively, and are not MolCLIP architectures (Zhu et al., 2024); (Yang et al., 2024).

Several limitations are not presented in a dedicated section of the MolCLIP paper but can be inferred from the reported setup. The model is evaluated only on MitoDataset; it depends on paired SMILES and time-lapse sequences; and the paper does not provide a broader robustness analysis across different MoA taxonomies (Pang et al., 10 Jul 2025). These constraints do not alter the reported empirical results, but they delimit the scope of the current evidence.

MolCLIP is therefore best characterized as a category-aware, molecule-auxiliary CLIP framework in which molecular structure is used to regularize and inform video-based phenotype learning. Its defining contribution is not generic cross-modal alignment alone, but the combination of MoA-aware supervision, metric learning, and temporal cell imaging within a single framework for mitochondrial drug-response analysis (Pang et al., 10 Jul 2025).