Hierarchical Molecular Language Models

• Hierarchical Molecular Language Models (HMLMs) are frameworks that explicitly combine atom-, motif-, and graph-level representations to enhance predictive accuracy in molecular tasks.
• They employ hierarchical tokenization and structured attention to align molecular graphs with textual encodings, reducing hallucinations and improving multimodal integration.
• Empirical results indicate that tailored pooling strategies and multi-scale embeddings significantly boost reaction prediction accuracy and molecular property performance.

Hierarchical Molecular LLMs (HMLMs) generalize traditional molecular representation learning by explicitly incorporating multi-level hierarchical structure—atom, motif (subgraph/functional group), and global graph—into LLM architectures. HMLMs provide an architectural and pretraining framework for aligning molecular graphs with LLMs through structured attention, hierarchical tokenization, and multimodal fusion. Hierarchical representations are now established as critical for biochemical property prediction, reaction modeling, generative molecular design, and biological sequence reasoning, and are central to robust molecule-language alignment, reducing hallucination, and enabling accurate systems biology modeling (Hu et al., 7 Nov 2024, Chen et al., 20 Jun 2024, Hays et al., 30 Nov 2025).

1. Hierarchical Feature Encoding in Molecular Graphs

The basis of HMLMs is the encoding of molecular structure at multiple granularity levels:

• Atom Level: Each atom $v$ in a molecular graph $G=(V,E)$ is mapped to an embedding $h_v \in \mathbb{R}^d$ using a graph neural network (GNN).
• Motif Level: Subgraphs corresponding to functional groups or structural motifs $M$ are extracted (e.g., by BRICS fragmentation). Each is represented as a motif node $V_m$ with corresponding embeddings $m_j \in \mathbb{R}^d$.
• Graph Level: A virtual node $V_g$—summarizing the entire molecule—receives information from all motifs, yielding a global embedding $g \in \mathbb{R}^d$.

The augmented hierarchical graph is

$$\mathcal{G} = (\mathcal{V}, \mathcal{E}),\quad \mathcal{V} = V \cup V_m \cup \{V_g\},\quad \mathcal{E} = E \cup E_m \cup E_g$$

After GNN message passing and readout, the resulting pool of feature tokens is

$$h_{\mathcal{G}} = \bigl\{ n_i \bigr\}_{i=1}^a \cup \{ m_j \}_{j=1}^b \cup \{g\}$$

where $a=|V|$, $b=|V_m|$, and all are mapped into the LLM embedding space via a projector $f_p$. Pooling strategies include:

• No reduction: All tokens are retained.
• Hierarchical reduction: Average pooling per feature level.
• All reduction: Global average pooling yields a single token.

The choice of pooling directly impacts the granularity of information delivered to the LLM, affecting downstream task performance (Hu et al., 7 Nov 2024).

2. Integration with LLMs

HMLMs fuse hierarchical graph representations with molecular textual encodings (SELFIES, SMILES, or t-SMILES variants) in unified architectures:

• Graph Encoder: A multi-level GNN produces hierarchical graph tokens.
• Projection: Each token is mapped to the LLM's embedding space.
• Fusion: Graph tokens are inserted at predetermined positions in the LLM’s token sequence, typically between the molecular string representation and the task instruction.
• Model Training:

1. Alignment pre-training: The projector is trained with a contrastive loss to align graph and domain text (e.g., SciBERT) embeddings.
2. LoRA fine-tuning: Adapters and projector parameters are further tuned for specific downstream tasks using supervised loss.

This multimodal LLM paradigm, exemplified by llama-based architectures, enables comprehensive molecular understanding and supports both discriminative and generative tasks (Hu et al., 7 Nov 2024, Chen et al., 20 Jun 2024).

3. Hierarchical Tokenization and Molecule-Language Alignment

Advances in hierarchical tokenization have demonstrated that properly discretizing molecules into atom, motif, and graph-level token sets is crucial for effective molecule-language alignment:

• Tokenization: Hierarchical tokenizers (e.g., HIGHT) employ motif discovery (BRICS) followed by vector quantization to generate discrete atom, motif, and graph “super-tokens”.
• Positional Coding: Laplacian eigenvector-based encodings are appended to each token to preserve graph topology and level-specific information.
• LLM Integration: Each level’s token set is projected by separate adapters and prepended to the LLM input, sharing the standard transformer's self-attention without custom graph layers.
• Data Augmentation: Motif-awareness is enforced by instruction-tuning datasets enriched with explicit motif cues ("This molecule has $n$ of $<$motif$>$ groups") (Chen et al., 20 Jun 2024).

Empirical evidence demonstrates that hierarchical tokenization reduces motif hallucination by 40%, improves performance on molecular classification, regression, captioning, and reaction tasks, and is robust to 1D string distractors (Chen et al., 20 Jun 2024).

4. Hierarchical Encodings in Biological Sequences and Systems

HMLM frameworks extend beyond small molecules to nucleic acid and protein sequence modeling:

• Sequence Hierarchy: Biological sequences (mRNA/protein) possess inherent codon/amino acid-level structure. Hierarchical encodings (HELM, HyperHELM) fuse codon and nucleotide embeddings, or embed the codon→amino acid ontology as tree or hyperbolic geometries (Yazdani-Jahromi et al., 16 Oct 2024, Spengler et al., 29 Sep 2025).
• Loss Functions: Hierarchical Cross-Entropy (HXE) and distance-based losses ensure errors are penalized proportionally to biological synonymity or tree distance.
• Systems Modeling: At the cellular scale, HMLMs treat intracellular signaling networks as a molecular language in which molecules are tokens, interactions are syntactic edges, and pathway/function modules form higher-order discourse. Hierarchical attention patterns integrate information across molecular, pathway, and cellular levels, yielding superior accuracy for temporal predictions in signaling networks and scalable, interpretable models for cellular phenotypes (Hays et al., 30 Nov 2025).

5. Task-Specific Effects and Empirical Findings

HMLMs show distinct performance patterns dependent on the choice of hierarchy level and pooling:

• Reaction Prediction: Global graph-level embeddings typically suffice for exact-match outcomes.
• Similarity/Property Tasks: Motif-level tokens drive gains in molecular fingerprint similarity, reagent prediction, and captioning.
• Fine-Grained Manipulation: Node-level (atom) features are necessary to resolve atom-wise changes (Hu et al., 7 Nov 2024).
• Pooling Regimes: Even all-reduction pooling (single token) can match multi-token performance in many tasks, but fine-grained tasks or molecular captioning benefit from retaining full hierarchical detail.

Empirical metrics across benchmarks are summarized as follows:

Task	Best Feature Level	Best Pooling	Key Metrics
Forward Reaction	Graph-level	No reduction	Exact match = 0.565
Reagent Prediction	Motif/Graph-level	All reduction	Fp. sim., BLEU, Exact
Captioning	Motif > Node/Graph	No reduction	BLEU2/4, ROUGE, METEOR
Systems Biology	Multi-scale (Molecule-Cell)	Hierarchical attention (dynamic)	MSE, Pearson $r$

High validity ($>99\%$), novelty ($\approx88$–$100\%$), and robust structural diversity are characteristic for HMLMs using hierarchical encoding, outperforming comparable flat or pure-graph approaches (Hu et al., 7 Nov 2024, Wu et al., 3 Feb 2024, Yu et al., 3 Oct 2024).

6. Limitations, Extensions, and Future Directions

Current limitations include:

• Static Level Weighting: Most HMLMs employ a fixed projector and pooling across tasks. Dynamic, per-task or per-sample attention over levels is advocated but has not yet been standard (Hu et al., 7 Nov 2024).
• Context Limits: For graph-to-tree text encodings (e.g., G2T-LLM), the LLM's context window constrains processable molecule size (Yu et al., 3 Oct 2024).
• Loss of 3D Structural Information: Predominant models operate on 2D graphs or codon trees. Incorporating explicit 3D geometry, higher-order interactions, or multi-omic representations remains an open challenge (Yu et al., 3 Oct 2024, Hays et al., 30 Nov 2025).
• Computational Load: End-to-end HMLMs with LLM inference can require significant compute, though graph-tokenized and tree-based strategies yield gains in runtime and parsing complexity (Wu et al., 3 Feb 2024).

Suggested future directions:

• Dynamic Feature Processing: Implement attention-weighted fusion of node, motif, and graph-level embeddings per instance or task.
• Integration with New Modalities: Multi-modal HMLMs combining structural, text, and sequence information, and adapting to mixed-geometry (Euclidean and hyperbolic) latent spaces (Spengler et al., 29 Sep 2025).
• Scaling to Systems Biology: Foundation-scale pretraining on phosphoproteomics, transcriptomics, and spatial omics to support predictive medicine (Hays et al., 30 Nov 2025).
• Hierarchical Attention Innovations: Incorporate tree positional encodings and block-sparse self-attention for efficient processing of highly hierarchical structures.

7. Best Practices and Implementation Guidelines

Consensus best practices for HMLM development include:

1. Explicit Multi-Level Encoding: Combine atom, motif, and graph (or sequence/codon) features through learned GNN or sequence models.
2. Contrastive Alignment: Align structural graph embeddings with text (e.g., SciBERT) prior to fusion with LLMs (Hu et al., 7 Nov 2024).
3. Flexible Pooling: Begin with all-reduction pooling for efficiency; use no reduction for tasks requiring fine granularity.
4. Motif-Aware Augmentation: Incorporate motif-aware instruction tuning to ground functional group attributes and reduce hallucination (Chen et al., 20 Jun 2024).
5. Joint Optimization: Train using a weighted combination of reconstruction, contrastive, and supervised task losses.
6. Hierarchical Evaluation: Regular monitoring of validity, uniqueness, novelty, and performance on motif-centric outputs, with parsing complexity and generalization robustness reporting(Wu et al., 3 Feb 2024, Hu et al., 7 Nov 2024).

Comprehensive adoption of these strategies allows HMLMs to deliver high validity molecular generation, accelerated convergence, improved molecular property and reaction prediction, as well as patient-specific modeling in cellular signaling and systems biology settings.

---

Key References:

• "Exploring Hierarchical Molecular Graph Representation in Multimodal LLMs" (Hu et al., 7 Nov 2024)
• "HIGHT: Hierarchical Graph Tokenization for Molecule-Language Alignment" (Chen et al., 20 Jun 2024)
• "Hierarchical Molecular LLMs (HMLMs)" (Hays et al., 30 Nov 2025)
• "G2T-LLM: Graph-to-Tree Text Encoding for Molecule Generation..." (Yu et al., 3 Oct 2024)
• "Hierarchical Structure Enhances the Convergence and Generalizability..." (Wu et al., 3 Feb 2024)
• "HELM: Hierarchical Encoding for mRNA Language Modeling" (Yazdani-Jahromi et al., 16 Oct 2024)
• "HyperHELM: Hyperbolic Hierarchy Encoding for mRNA Language Modeling" (Spengler et al., 29 Sep 2025)