Multimodal Hierarchical Graphs

• Multimodal Hierarchical Graphs are structured representations that organize modality-specific and cross-modal relationships across multiple hierarchical levels.
• They integrate diverse modalities such as text, images, and audio using techniques like attention, pooling, and message-passing to improve model performance.
• Recent advances demonstrate that hierarchical graph fusion boosts alignment, generalization, and efficiency in multimodal machine learning tasks.

Multimodal hierarchical graphs are structured representations that capture both modality-specific and cross-modal relationships among entities, organized across multiple hierarchical levels. Such structures are foundational to contemporary multimodal machine learning, enabling models to encode, reason over, and generate heterogeneous data spanning text, images, audio, video, and structured knowledge—with explicit relational and granular dependencies. Research in this area synthesizes principles from attributed graph modeling, hierarchical and multiplex graph theory, and multimodal fusion, providing the backbone for the latest multimodal LLMs, retrieval-augmented reasoning systems, and graph-based fusion schemes.

1. Formalism and Taxonomy of Multimodal Hierarchical Graphs

A multimodal hierarchical graph (MMHG) combines semantic information from multiple modalities with a hierarchy reflecting structural dependencies or levels of abstraction. A typical instantiation is the multimodal attributed graph (MMAG) as in

$G = (V, E, \{A^{(m)}\}_{m=1}^M)$

where $V$ is the node set, $E$ the edge set, and $A^{(m)}$ the attributes of modality $m$ (e.g., text, image, audio) per node (Fang et al., 17 Feb 2025). Hierarchies arise from:

• Intra-modal structure: e.g., syntactic parse trees (text), region-ROI graphs (vision), motif-to-atom hierarchies (molecular graphs (Hu et al., 2024)), scene-part hierarchies (3D scene graphs (Werby et al., 1 Oct 2025)).
• Inter-modal structure: e.g., object–sentence graphs for summaries (Zhang et al., 2021), cross-modal relation graphs that reconstruct one modality’s features using neighborhoods defined by another (Rehman et al., 22 Aug 2025).
• Multiplex/hierarchical levels: e.g., stacking modality-specific graphs and then fusing them into a joint or consensus graph (Dimiccoli et al., 2021, Abdous et al., 2023), or encoding document structure via parent–child trees in text/image layouts (Shin et al., 28 May 2026).

In many frameworks, each node carries a tuple of feature vectors—one per modality—which may themselves be hierarchically pooled or injected into downstream models with varying levels of reduction.

2. Hierarchical Modeling Paradigms: Levels, Aggregation, and Fusion

Hierarchical modeling in MMHGs occurs both within and across modalities. Principled aggregation schemes have been developed to exploit this:

• Hierarchical Q-Former architecture: For multimodal LLMs, GraphGPT-o introduces a two-stage hierarchy: intra-node fusion (text/image) followed by subgraph-level fusion (neighboring nodes), both via transformer attention blocks (Fang et al., 17 Feb 2025).
• Multi-level graph tokens: Molecular MMHGs extract embeddings at atom, motif, and global levels, then aggregate into single or multiple summary tokens. Experimental results demonstrate that different downstream tasks in chemistry are sensitive to the granularity at which this pooling is performed (Hu et al., 2024).
• Hierarchical aggregation in multiplex graphs: The HMGE framework learns at each layer non-linear combinations of multiple dimensions (relations or modalities), recursively refining node embeddings and adjacency matrices up the hierarchy. Each layer reduces the number of “dimensions” (modalities/relations) via learned attention, culminating in a global consensus graph (Abdous et al., 2023).
• Hierarchical attention/fusion: In vision–language summarization, HCSCL hierarchically fuses information across word–object and sentence–scene levels via cross-modality transformer blocks, capturing both fine- and coarse-grained dependencies (Zhang et al., 2021). Similarly, MM-ORIENT separates hierarchical intra-modal attention (HIMA) from cross-modal graph fusion to improve robustness (Rehman et al., 22 Aug 2025).

In all cases, each level of the hierarchy incorporates local information at lower tiers and aggregates it to build richer, higher-level representations, often using attention, pooling, or message-passing mechanisms.

3. Graph Construction and Encoding Techniques

Graph construction for multimodal hierarchical graphs is highly task- and modality-dependent:

• Neighbor selection: Personalized PageRank, k-hop neighborhoods, and similarity thresholds are used to construct local subgraphs for each node, with K typically tuned between 3–5 for optimal balance between context coverage and noise (Fang et al., 17 Feb 2025, Ning et al., 19 Oct 2025).
• Edge typing and relation labeling: Edges may be labeled by modality, context, or semantic relation (e.g., speaker/time in conversation graphs (Tang et al., 2021), or type-of-visual-element in 3D scenes (Werby et al., 1 Oct 2025)).
• Cross-modal relation graphs: MM-ORIENT constructs four sparse graphs for all pairs of modalities, with node neighborhoods defined by feature similarity in the other modality, followed by one-stage GraphSAGE aggregation (Rehman et al., 22 Aug 2025).
• Graph linearization vs. deep encoding: Flat linearization approaches transform graphs into token sequences, while hierarchical attention-based encoders (Q-Former, MM-QFormer) fuse structural and semantic information in deep transformer blocks, yielding significantly improved generation and alignment metrics (Fang et al., 17 Feb 2025, Ning et al., 19 Oct 2025).

The table summarizes major encoding paradigms:

Framework	Graph Construction	Hierarchy Levels	Encoding Technique
GraphGPT-o (Fang et al., 17 Feb 2025)	PPR neighbors, undirected MMAG	Node, subgraph	Hierarchical Q-Former
MM-ORIENT (Rehman et al., 22 Aug 2025)	Cross-modal similarity graphs	Feature, batch	SAGE, hierarchical attn
HFGCN (Tang et al., 2021)	Multi-level relational graphs	Intra-utterance, convo	Relational GCN/GCN
HMGE (Abdous et al., 2023)	Dimension-wise adjacency	Multiplex → consensus	Per-dim GCN + attn
KeySG (Werby et al., 1 Oct 2025)	3D containment hierarchy	Floor, room, object, func	Keyframe VLM fusion

4. Fusion and Alignment of Multimodal and Structural Information

Aligning modalities and structure is a central challenge. Recent methods achieve this using:

• Multi-stage attention: Hierarchical Q-Formers (Fang et al., 17 Feb 2025) and MM-QFormer (Ning et al., 19 Oct 2025) first fuse intra-node modalities, then aggregate across subgraphs, providing deep alignment beyond flat concatenation or serialization.
• Mutual information objectives: HMGE maximizes the mutual information between local node patches and global embeddings, enhancing the capture of both local and systemic patterns in high-dimensional multiplex or multimodal graphs (Abdous et al., 2023).
• Graph-structured multimodal transformers: GsiT formalizes MulTs as hierarchical modal-wise heterogeneous graphs, where block-adjacency matrices and interlaced attention masks enable explicit all-modal message-passing with parameter efficiency (Jin et al., 2 May 2025).
• Coarse-to-fine retrieval: HiKEY leverages a document tree where retrieval, reranking, and token allocation are guided by the document’s explicit section and multimodal hierarchy, enabling efficient evidence integration within bounded context windows (Shin et al., 28 May 2026).
• Task-aware and dynamic fusion: In molecular LLMs, preference for graph/motif/atom-level tokens varies by prediction task, suggesting a need for adaptive, task-specific pooling or projection mechanisms (Hu et al., 2024).

A common observation is that directly treating “graph” as a modality is suboptimal; instead, structure should scaffold attention or fusion across the semantic space.

5. Empirical Results, Applications, and Benchmarking

Multimodal hierarchical graph techniques now underpin a range of state-of-the-art models in classification, generation, and retrieval:

• Generative and comprehension tasks: GraphGPT-o’s hierarchical aligner surpasses flat and prior multimodal LLM designs (+1–2 CLIP-I2, better KL-DV, lower perplexity) on domain-diverse graph datasets (Fang et al., 17 Feb 2025).
• Node and subgraph-level adaptation: K=3–5 neighbor selection optimally balances local context and robustness (Fang et al., 17 Feb 2025). Performance drops observed when either node-level or graph-level Q-Former layers are removed.
• Downstream QA/IR: HiKEY achieves up to +12.9% retrieval recall and +6.8% QA accuracy by leveraging both hierarchy and multimodal document blocks (Shin et al., 28 May 2026).
• Efficiency and scaling: GsiT delivers 3× parameter reduction and +4.5%–11.2% absolute accuracy gains vs. baseline MulTs with custom block-sparse computation kernels (Jin et al., 2 May 2025).
• Multitask learning robustness: MM-ORIENT demonstrates that late fusion of hierarchical intra-modal attention with cross-modal graphs yields better generalization and less modality noise propagation (Rehman et al., 22 Aug 2025).
• Interpretability: Hierarchical projections (e.g., valence-arousal in AER (Tang et al., 2021), 2D/3D scene graphs in KeySG (Werby et al., 1 Oct 2025)) provide interpretable reasoning spaces and improve semantic recall.

Experiments consistently support the claim that hierarchical and graph-aware fusion brings measurable improvements in alignment, generalization, and efficiency across tasks and domains.

6. Limitations and Emerging Research Directions

Despite empirical successes, several limitations and open challenges remain:

• Pretraining for multi-level semantics: Most MLLMs show limited use of fine-grained graph tokens; shallow pooling often matches more complex hierarchies, indicating insufficient pretraining alignment between modalities and structure (Hu et al., 2024).
• Scalability: Large graphs or long document trees may exceed transformer context limits; hierarchical RAG, graph compression, and keyframe selection partially mitigate but do not eliminate these issues (Werby et al., 1 Oct 2025, Shin et al., 28 May 2026).
• Dynamic and heterogeneous relations: Extensions to dynamic graphs, time-aware diffusion, and typed/heterogeneous edge relations remain underexplored (Ning et al., 19 Oct 2025).
• Fusion mechanism design: There is a research need for adaptive, task- or instance-aware fusion mechanisms (dynamic projectors, gating, or attention pools) in place of static projections (Hu et al., 2024).
• Unified theoretical perspectives: The recognition that attention is graph message-passing, and vice versa, opens new avenues for spectral, regularized, or higher-order (hypergraph) transformer–graph architectures (Jin et al., 2 May 2025).

A plausible implication is that future advances will focus on joint pretraining objectives, hierarchical alignment across broader sets of modalities (e.g., audio/video/3D), real-time scalable graph handling, and deeper theoretical integration between graph-theoretic foundations and large model architectures.

---

For further details, see "GRAPHGPT-O: Synergistic Multimodal Comprehension and Generation on Graphs" (Fang et al., 17 Feb 2025), "Graph4MM: Weaving Multimodal Learning with Structural Information" (Ning et al., 19 Oct 2025), "Exploring Hierarchical Molecular Graph Representation in Multimodal LLMs" (Hu et al., 2024), "Hierarchical Aggregations for High-Dimensional Multiplex Graph Embedding" (Abdous et al., 2023), "Multimodal Transformers are Hierarchical Modal-wise Heterogeneous Graphs" (Jin et al., 2 May 2025), and associated works cited throughout.