Multi-Modal Graph Neural Networks

• Multi-Modal Graph Neural Networks (MM-GNN) are models that integrate multiple modality encoders with graph structures to learn complex relationships in heterogeneous data.
• They employ modality-specific encoding, cross-attention, and tailored fusion mechanisms to effectively combine visual, textual, and other sensor data.
• MM-GNNs have achieved notable success in vehicular communications, medical diagnosis, and recommendation systems by leveraging both relational and multimodal signals.

Multi-Modal Graph Neural Network (MM-GNN) denotes a family of graph neural network models that incorporate and learn from multiple modalities associated with graph-structured data, most prominently visual and textual information, but also audio, semantics, mobility, phenotypic attributes, and sensor-derived signals. In these models, graph structure supplies relational context, while modality-specific encoders supply heterogeneous content representations; the central objective is to learn jointly from both rather than to treat multimodal inputs as isolated vectors or simple paired samples (Zhu et al., 2024, He et al., 2 Feb 2025). The literature spans task-specific systems for vehicular communications, question answering, recommendation, urban forecasting, and medical diagnosis, as well as broader frameworks for multimodal graph pre-training and multimodal graph benchmarks (Lu et al., 20 Feb 2026, Yoon et al., 2023).

1. Conceptual scope and terminology

In MM-GRAPH, MM-GNNs are defined as graph neural network models designed to incorporate and learn from multiple modalities, in particular visual and textual information associated with nodes in graphs (Zhu et al., 2024). UniGraph2 sharpens this graph-centric formulation by defining multimodal graphs as graphs where each node is associated with features from different modalities, while the edges capture the relationships between these entities (He et al., 2 Feb 2025). This definition accommodates both homogeneous settings, where many nodes share the same modal inventory, and heterogeneous settings, where modalities may be absent or asymmetrically distributed across the graph.

The role of the graph is itself a point of methodological differentiation. In MMGL, graphs are used to represent complex, many-to-many multimodal relationships that go beyond one-to-one mappings such as image-caption pairs (Yoon et al., 2023). Graph4MM argues that graphs should be used as structure to guide input assembly and attention, rather than treated as a standalone modality, because previous works fail to distinguish multi-hop neighbors and fragment the overall understanding when the graph is isolated from the multimodal backbone (Ning et al., 19 Oct 2025). A plausible implication is that the term MM-GNN refers not merely to “a GNN with multimodal node features,” but to a broader design space in which modality interaction and structural reasoning are co-designed.

A recurring source of confusion is acronym overload. “MM-GNN” also names the “Mix-Moment Graph Neural Network,” which models neighborhood feature distributions through multi-order moments and an Element-wise Attention-based Moment Adaptor module; despite the acronym, that model is not a multimodal architecture in the sense used by multimodal graph learning papers (Bi et al., 2022). In encyclopedia usage, the generic term therefore denotes the multimodal family, while paper-specific acronyms must be interpreted in context.

2. Graph construction and multimodal representation

MM-GNNs differ first in how they encode multimodal data into graph form. In the urban M3G framework, neighborhoods are nodes, street view images and POI features characterize nodes, and human mobility and spatial distance characterize inter-neighborhood relations as directed or undirected edges within a multi-modal multi-graph (Huang et al., 2021). In MMGA, the graph modality is a social network, while image and text are attached to users and aligned with the graph through a multi-step graph alignment mechanism (Yang et al., 2022). These formulations exemplify two common strategies: modalities as node attributes, and modalities as edge-defined relational signals.

Application-specific MM-GNNs often use highly structured graph abstractions. MM-ResGNN formulates the propagation environment as an environment sensing path loss graph (ESPL-Graph), where nodes represent the transmitter and the nearest $K = 50$ receivers for each snapshot, and edges jointly describe Tx–Rx transmission links and Rx–Rx spatial correlation links (Lu et al., 20 Feb 2026). Each transmission edge is annotated with a 14-dimensional physics-aware feature vector including distance, blockage ratios, dynamic obstacle presence, and angular directions. In this setting, RGB images, semantic information, and ray-tracing path loss data are precisely aligned in space and time in the Vehicular Multi-Modal Path Loss dataset, so graph construction is explicitly tied to both electromagnetic and physical-space representations (Lu et al., 20 Feb 2026).

Other MM-GNNs construct multiple subgraphs rather than a single heterogeneous graph. For scene-text VQA, MM-GNN represents an image as three sub-graphs—visual, semantic, and numeric—and then introduces three aggregators to guide message passing from one graph to another (Gao et al., 2020). VQA-GNN instead combines a scene graph, a concept graph, a QA-context node, and a QA-concept node into a multimodal semantic graph for bidirectional fusion between structured and unstructured multimodal knowledge (Wang et al., 2022). In neuroimaging, MM-GTUNets uses a population graph where each subject is a node with imaging and non-imaging features, and the adjacency matrix encodes subject affinity learned through a reward system (Cai et al., 2024). These examples show that multimodality can be instantiated as multiplex graphs, heterogeneous supergraphs, or population graphs, depending on whether the central relation is physical propagation, semantic reasoning, or cross-subject similarity.

3. Architectural patterns for multimodal interaction

A dominant architectural pattern is the use of modality-specific encoders followed by graph-based interaction. MMGA uses ViT as image encoder, BERT as text encoder, and a multimodal-mixture GNN as graph encoder, after which the three embeddings are concatenated for downstream tasks (Yang et al., 2022). MM-ResGNN follows a dual-branch residual correction paradigm in which a physics-embedded graph learning branch operates on the ESPL-Graph and a visual semantic branch uses a pre-trained ResNet-18 whose shallow layers are frozen and higher layers fine-tuned for domain adaptation (Lu et al., 20 Feb 2026). In socially aware music recommendation, lyrics, audio, and visuals are encoded using BERT, YAMNet, and ResNet, then processed through separate modality-specific GNNs under a fusion-free deep mutual learning strategy (Ziaoddini, 13 Sep 2025).

Fusion mechanisms vary substantially. MM-ResGNN integrates topology-aware graph representations and fine-grained visual semantics through a learned gating coefficient,

$$w = \sigma\left(\mathbf{W}_g \left[\mathbf{f}_{geo} \parallel \mathbf{f}_{vis}\right] + \mathbf{b}_g\right), \qquad \mathbf{f}_{final} = w \mathbf{f}_{geo} + (1-w) \mathbf{f}_{vis},$$

so that the contribution of graph-based and vision-based features is regulated adaptively by context (Lu et al., 20 Feb 2026). MGNM divides processing into local and global interaction: local interaction uses GNN propagation over collaborative and modality graphs, while global interaction uses modality-guided global feature purifiers to denoise modality features unrelated to user preference (Mo et al., 2024). In the socially aware music model, modality alignment is not performed by feature concatenation at the raw-feature level; instead, Deep Mutual Learning minimizes pairwise KL divergence between modality-specific prediction distributions, while maintaining robustness against missing modalities (Ziaoddini, 13 Sep 2025).

Another major pattern is to inject graph structure directly into attention. The Multimodal Graph Transformer modifies self-attention by composing adjacency-derived masks from a text graph, dense region graph, and semantic graph, then adding them to the attention logits together with a trainable bias (He et al., 2023). Graph4MM extends this logic to multi-hop structure with Hop-Diffused Attention,

$$\mathcal{A} = \sum_{i=0}^{K} \theta_i \mathbf{A}^i, \qquad \theta_i = \alpha (1-\alpha)^i,$$

and then couples it with MM-QFormer for cross-modal fusion (Ning et al., 19 Oct 2025). In a different direction, M-GWCN performs intra-modality feature extraction with multiscale graph wavelet transforms and cross-modality correlation through learned permutation matrices, explicitly without requiring the same number of data across modalities or prior correspondences between modalities (Behmanesh et al., 2021). These systems indicate that MM-GNN architecture is not synonymous with message passing alone; it includes graph transformers, graph wavelet networks, graph-aware cross-attention, and modality-specific branch designs.

4. Learning objectives, pretraining, and optimization regimes

MM-GNN training objectives are as heterogeneous as the graphs themselves. MM-ResGNN is organized around deterministic–stochastic decoupling: a geometry-driven physical baseline models deterministic path loss, while the network predicts a normalized residual,

$$y_{\text{target}} = \frac{(PL_{\text{raw}} - PL_{\text{base}}) - \mu_{\text{res}}}{\sigma_{\text{res}}},$$

and reconstructs the final estimate by adding the predicted residual back to the fixed baseline (Lu et al., 20 Feb 2026). This design separates geometry-driven attenuation trends from residual variations and is paired with evaluation by NMSE, MAE, and MAPE.

Self-supervision and contrastive learning are also prominent. MMGA uses graph contrastive loss to supervise image and text encoders from graph structure, while the multimodal-mixture GNN uses image and text embeddings to guide graph propagation; it also combines language modeling loss, image-text alignment loss, node feature modeling loss, and graph structure modeling loss (Yang et al., 2022). M3G uses triplet-based contrastive losses for both intra-neighborhood modalities and inter-neighborhood edge modalities, with embeddings unified in the same $d$-dimensional space (Huang et al., 2021). UniGraph2 pre-trains with multimodal masking, Mixture of Experts feature alignment, feature reconstruction, and shortest path distance reconstruction across a collection of multimodal graphs from various domains (He et al., 2 Feb 2025).

Recommendation-oriented MM-GNNs emphasize ranking, denoising, and decorrelation. MGNM combines Bayesian Personalized Ranking loss with cross-modal contrastive losses and a Dynamic De-redundancy loss that penalizes correlation in learned features through the feature coefficient matrix (Mo et al., 2024). The same model uses a two-fold modality-guided purification mechanism to eliminate modality features irrelevant to user preferences and capture complex relationships within each modality. In generative multimodal graph learning, MMGL studies self-attention and cross-attention strategies for infusing multimodal neighbor information into pretrained LLMs, together with graph positional encodings such as sequential encoding, Laplacian Position Encoding, and GNN encoding, and evaluates Prefix Tuning, LoRA, and Flamingo-style tuning as parameter-efficient finetuning strategies (Yoon et al., 2023).

Recent benchmarking work organizes this broader landscape into paradigms. Graph-MLLM distinguishes MLLM-as-Encoder, MLLM-as-Aligner, and MLLM-as-Predictor, thereby placing GNN-centric, alignment-centric, and predictor-centric MM-GNN pipelines within a common experimental frame (Liu et al., 12 Jun 2025). This suggests that “MM-GNN” is increasingly a systems-level category rather than a single architectural template.

5. Application domains and representative results

MM-GNNs have been deployed in domains where modality interaction and relational context are both indispensable. In vehicular communications, MM-ResGNN is proposed for millimeter-wave path loss prediction in vehicular communications for the first time and reports a normalized mean squared error of 0.0098, a mean absolute error of 5.7991 dB, and a mean absolute percentage error of 5.0498%; it also shows cross-scenario adaptation through few-shot fine-tuning, including MAE 6.43 dB on suburban forking road with 5% target-domain data and MAE 3.34 dB on urban crossroad with 10% target-domain data (Lu et al., 20 Feb 2026). In brain disorder prediction, MM-GTUNets reports, on ABIDE and ADHD-200 respectively, ACC 82.9 and AUC 88.2, and ACC 82.7 and AUC 90.7, using Modality Reward Representation Learning, VAE-based non-imaging alignment, and Adaptive Cross-Modal Graph Learning with a GTUNet encoder (Cai et al., 2024).

The following examples illustrate the range of reported outcomes across domains:

Domain	Representative model	Reported result
mmWave vehicular path loss	MM-ResGNN	NMSE 0.0098; MAE 5.7991 dB; MAPE 5.0498% (Lu et al., 20 Feb 2026)
Brain disorders prediction	MM-GTUNets	ABIDE: ACC 82.9, AUC 88.2; ADHD-200: ACC 82.7, AUC 90.7 (Cai et al., 2024)
Multimodal recommendation	MGNM	Up to 9.54% on Baby, 8.32% on Video, 9.75% on Beauty over the second-best (Mo et al., 2024)
Scene-text VQA	MM-GNN	TextVQA: 31.44 validation, 31.10 test; ensemble 32.92 validation, 32.46 test (Gao et al., 2020)
Alzheimer’s disease diagnosis	IFDCGCN	Up to 96.68% accuracy (Zhanga et al., 2023)

Recommendation and question answering expose different strengths of MM-GNN design. MGNM reports state-of-the-art performance on all datasets and metrics across Amazon Baby, Video Games, and Beauty, with the reported relative improvements attributed to dynamic de-redundancy, modality-guided purification, and self-supervised consistency (Mo et al., 2024). In scene-text reasoning, MM-GNN improves TextVQA accuracy over LoRRA and BERT + MFH baselines by refining visual, semantic, and numeric node features through inter-modality aggregators (Gao et al., 2020). VQA-GNN reports gains of 3.2% on VCR (Q-AR) and 4.6% on GQA through bidirectional fusion between structured and unstructured multimodal knowledge, while showing that a single GNN over the whole graph underperforms the final design with two modality-specialized GNNs (Wang et al., 2022).

Medical and urban applications make a similar point from a different direction. In Alzheimer’s diagnosis, the proposed multi-modal GNN framework combines sMRI, PET, and phenotypic information through node-vector fusion, adjacency-matrix fusion, and late fusion, with IFDCGCN attaining up to 96.68% accuracy for AD vs. NC (Zhanga et al., 2023). In urban spatiotemporal forecasting, the problem is explicitly formulated as a multi-modal machine learning problem on multi-graph convolution networks, and grouped GCN plus multi-linear relationship GCN reduces Beijing RMSE from 11.82 for MGCN to 9.31 and Shanghai RMSE from 8.64 to 7.88 (Geng et al., 2019). Although these domains differ sharply, both use multiple relations or modalities to mitigate incomplete single-view representations.

6. Benchmarks, limitations, and ongoing directions

The benchmark literature tempers the view that any dedicated multimodal architecture will automatically dominate generic graph baselines. MM-GRAPH introduces the first comprehensive multimodal graph benchmark with seven diverse datasets of varying scales and finds that dedicated MM-GNNs such as MMGCN and MGAT do not systematically outperform GraphSAGE, GCN, BUDDY, or even MLP baselines (Zhu et al., 2024). The same study reports that feature alignment is crucial, that using both visual and textual features increases performance by more than 6% across tasks compared to using text alone in the MLP ablation, and that current full-batch MM-GNNs run out of memory on the largest graphs. This suggests that architecture-specific inductive bias is only one part of the MM-GNN problem; encoder choice, alignment space, and scalability can dominate final performance.

Graph-MLLM extends this benchmarking perspective to multimodal LLMs and identifies three paradigms—MLLM-as-Encoder, MLLM-as-Aligner, and MLLM-as-Predictor—across six datasets (Liu et al., 12 Jun 2025). It observes that jointly considering visual and textual attributes benefits graph learning, that converting visual attributes into textual descriptions further improves performance compared to directly using visual inputs, and that fine-tuning MLLMs on specific multimodal graphs can achieve state-of-the-art results in most scenarios even without explicit graph structure information. A plausible implication is that the boundary between MM-GNNs and multimodal graph-augmented language systems is becoming increasingly porous.

Several open technical directions recur across the surveyed works. MM-GRAPH recommends moving beyond late-stage fusion, prioritizing feature alignment, developing mini-batch or sampling-based designs, and expanding beyond text and vision (Zhu et al., 2024). Graph4MM argues for multi-hop structural integration and principled cross-modal fusion rather than treating graph structure as an isolated branch (Ning et al., 19 Oct 2025). M-GWCN shows that multimodal graph learning need not assume prior correspondences or equal sample counts across modalities (Behmanesh et al., 2021), while the socially aware music model shows that a fusion-free design with deep mutual learning can maintain robustness against missing modalities (Ziaoddini, 13 Sep 2025). Taken together, these results indicate that the central research problem has shifted from simple multimodal attachment to graph nodes toward the joint design of alignment, structure infusion, scalability, and robustness in increasingly heterogeneous multimodal graph regimes.