Cross-Modal Relation Graphs

• Cross-Modal Relation Graphs are multimodal representations that model intra- and inter-modal relationships using nodes from diverse data sources like text, images, and videos.
• They employ strategies such as scene graph fusion, similarity-driven induction, and context-conditioned topology to dynamically construct and refine graph structures.
• Applications span retrieval, captioning, summarization, and reasoning while addressing challenges like noise control, modality imbalance, and temporal variations.

Cross-modal relation graphs are graph-based multimodal representations in which nodes are drawn from different modalities, or from modality-specific units, and edges encode intra-modal and inter-modal relationships. In the cited literature, they appear as visual and textual scene graphs, sequential cross-modal semantic graphs, multimodal knowledge graphs, relation subgraphs, and multimodal attributed graphs, and they are used for retrieval, captioning, summarization, reasoning, grounding, and dynamic scene understanding (Wang et al., 2019, Kim et al., 26 Mar 2025, Lee et al., 2024). A recurring objective is to preserve fine-grained relational semantics while controlling noise, modality imbalance, and temporal variation (Wu et al., 2023, Zhang et al., 12 Jun 2026).

1. Historical emergence and scope

Early work in cross-modal retrieval established a core design pattern: represent one modality with an explicit graph and learn a shared space with another modality. “Modeling Text with Graph Convolutional Network for Cross-Modal Information Retrieval” modeled text documents as graphs whose vertices correspond to words and whose edges are defined by $k$-NN relations in word2vec space, then coupled a GCN-based text path with an image path and optimized a pairwise similarity loss (Yu et al., 2018). “Semantic Modeling of Textual Relationships in Cross-Modal Retrieval” extended this idea by integrating distributed semantic relations, word co-occurrence, and knowledge-base relations into a featured graph, and reported improvements of 3.4% and 6.3% on two benchmark datasets (Yu et al., 2018).

The next stage shifted from textual word graphs to scene-level relational structures. “Cross-modal Scene Graph Matching for Relationship-aware Image-Text Retrieval” represented images with a visual scene graph and sentences with a textual scene graph, then matched them at both object and relationship levels; on Flickr30k, it reported 71.8 R@1 for caption retrieval and 53.5 R@1 for image retrieval, while on MSCOCO it reported 50.0 R@1 for caption retrieval (Wang et al., 2019). “A Universal Model for Cross Modality Mapping by Relational Reasoning” generalized the paradigm by explicitly separating Intra Graphs and an Inter Graph, then iteratively updating node and edge features to estimate cross-modal mappings across image classification, social recommendation, and sound recognition (Li et al., 2021).

Subsequent work broadened the notion of cross-modal relation graphs beyond retrieval. “Cross-Modal Graph with Meta Concepts for Video Captioning” dynamically built meta concept graphs from weakly learned cross-modal meta concepts and combined them with frame-level and video-level scene graphs (Wang et al., 2021). “Target-oriented Sentiment Classification with Sequential Cross-modal Semantic Graph” serialized scene graph triples, captions, and tweet text into a sequential cross-modal semantic graph governed by a multi-modal adjacency matrix (Huang et al., 2022). More recent systems further extended the paradigm to multimodal relation extraction, dynamic scene graphs, multimodal knowledge graphs, dynamic text-attributed graphs, and multimodal attributed graphs (Wu et al., 2023, Wang et al., 2023, Lee et al., 2024, Chang et al., 7 May 2026, Zhang et al., 12 Jun 2026).

2. Graph primitives and construction strategies

Across the cited literature, cross-modal relation graphs differ primarily in node granularity, edge semantics, and whether connectivity is fixed, learned, or task-adaptive. Nodes may correspond to words, scene-graph objects and predicates, image regions, utterances, text segments, visual segments, knowledge-graph entities, or graph nodes with text and image attributes (Yu et al., 2018, Wang et al., 2019, Liu et al., 22 Mar 2026, Kim et al., 26 Mar 2025, Zhang et al., 12 Jun 2026). Edges may encode semantic similarity, word co-occurrence, knowledge links, speaker-dependent dependencies, scene-graph relations, cross-modal similarity, or reliable topology filtered by multimodal consistency (Yu et al., 2018, Liu et al., 22 Mar 2026, Wu et al., 2023, Zhang et al., 12 Jun 2026).

One recurrent construction strategy is explicit scene-graph fusion. In multimodal relation extraction, a visual scene graph and a textual scene graph are fused into a unified cross-modal graph $G=(V,E)$ with $V=V^I \cup V^T$ and $E=E^I \cup E^T \cup E^{\times}$, where cross-modal edges are created when CLIP-based cosine similarity exceeds a threshold (Wu et al., 2023). SeqCSG uses a different but related design: scene-graph triples, captions, and tweet text are serialized into a single sequence, while a multi-modal adjacency matrix $M$ determines which token pairs can interact during self-attention (Huang et al., 2022).

A second strategy is similarity-driven graph induction. CSS-GR constructs a graph over textual and visual segments and learns the adjacency matrix dynamically:

$$A_{ij} = \begin{cases} 1 & \text{if } \cos(f_i, f_j) \geq \tau \ 0 & \text{otherwise} \end{cases}$$

This graph captures both intra-modal and inter-modal relationships, and its connectivity can be updated during training (Kim et al., 26 Mar 2025). MM-ORIENT similarly computes similarity matrices per modality, thresholds them into adjacency matrices, and then builds graphs $\mathcal{G}(m_1,m_2)$ in which the nodes come from one modality while the edges are decided by another modality (Rehman et al., 22 Aug 2025).

A third strategy is context-conditioned topology. AS3D constructs semantic-spatial scene graphs for 3D visual grounding, where object proposals are nodes and edges are established based on semantic relevance to the description rather than merely spatial proximity (Xiao et al., 7 May 2025). CoMAG estimates edge reliability from multimodal semantic consistency, complements raw topology with semantic neighbors, and mixes reliability-filtered edges, recovered semantic neighbors, and self-loops through a task-aware gate (Zhang et al., 12 Jun 2026). This suggests that recent work treats graph construction less as a static preprocessing step and more as a learnable component of multimodal inference.

3. Message passing, fusion, and alignment

Once constructed, cross-modal relation graphs are typically coupled to GCN, GAT, Transformer, or state-space components. CSS-GR uses stackable GNN layers for message passing,

$$h_i^{(l+1)} = \sigma\left(\sum_{j \in \mathcal{N}(i)} \alpha_{ij}^{(l)} W_1 h_j^{(l)} + W_2 h_i^{(l)}\right),$$

and combines them with a discrete-time state-space model that maintains a global hidden state; removing the graph reduces ROUGE-L by 2.2, removing the state-space module reduces ROUGE-L by 1.5, and static rather than dynamic connectivity reduces ROUGE-L by 0.7 (Kim et al., 26 Mar 2025).

Several systems use graphs as explicit alignment substrates. MR-MKG employs a relation graph attention network to encode retrieved subgraphs from multimodal knowledge graphs and a cross-modal alignment module with triplet loss to align image and text entities, while training only approximately 2.25% of the LLM’s parameter size (Lee et al., 2024). MMKG-T5 does not construct a single monolithic graph encoder; instead, it dynamically generates link-aware multimodal contexts using a pre-trained VLM, samples relation context and neighbor context, serializes the result, and frames knowledge graph completion as sequence-to-sequence generation. On FB15k-237-IMG it reports MRR 0.308 and Hits@10 0.488 (Ma et al., 26 Jan 2025).

Other methods use relation graphs for completion and context recovery rather than direct matching. PCCA constructs relation graphs from cross-modal nearest neighbor sets and prototypes to generate missing modal features in incomplete text-based person ReID, then applies a prototype-aware cross-modal alignment loss for fine-grained alignment (Gong et al., 2023). CoMAG, by contrast, maintains modality-specific multi-hop trajectories, aligns modality-hop tokens across modalities, and explicitly decouples shared and private representations, with an orthogonality term controlling redundancy between them (Zhang et al., 12 Jun 2026).

A recurring methodological disagreement concerns whether the graph should be treated as a standalone modality. Graph4MM explicitly argues that treating the graph as a standalone modality “fragments the overall understanding,” and instead uses Hop-Diffused Attention to integrate multi-hop structural information into self-attention plus MM-QFormer for cross-modal fusion (Ning et al., 19 Oct 2025). MM-ORIENT makes an adjacent point from a different angle: multimodal representations can be acquired cross-modally without explicit interaction between different modalities, by reconstructing monomodal features based on neighborhoods determined by another modality (Rehman et al., 22 Aug 2025).

4. Temporal, dynamic, and iterative relation learning

Temporal variation is a particularly demanding setting for cross-modal relation graphs because relations may change while object identity remains stable. TR$^2$ addresses dynamic scene graph generation by explicitly supervising the temporal difference of visual relation features with the difference of prompted text embeddings obtained from a frozen CLIP text encoder:

$$L_{guidance} = \frac{1}{T-1} \sum_{i=2}^T \left[ (e_{r_t} - e_{r_{t-1}}) - (e_{s_t} - e_{s_{t-1}}) \right]^2.$$

It pairs this explicit guidance with an implicit relation feature fusion module consisting of a transformer and a message token that describes the difference between adjacent frames, and reports gains of 2.1% and 2.6% over previous state of the art under two settings on Action Genome (Wang et al., 2023).

Dynamic structure also appears in conversation emotion recognition. The denoising-and-diffusion model for MCER constructs inter-speaker and intra-speaker relation subgraphs over utterance-level textual features, distinguishes self-loop, forward, and backward relation types, and then uses a text-guided cross-modal diffusion mechanism to inject denoised audio and visual evidence into the textual stream (Liu et al., 22 Mar 2026). Here, the graph is not merely a fusion aid; it is the semantic anchor that determines how noisy non-text modalities enter the representation.

Iterative refinement is another temporalized interpretation of cross-modal relation learning. PRISM organizes dynamic text-attributed graphs into semantic and behavioral modalities, initializes the latent state at the semantic prior, and performs $G=(V,E)$0 refinement steps in which cross-modal attention retrieves relevant behavioral evidence and an Euler-style update moves the state toward a behavior-conditioned posterior (Chang et al., 7 May 2026). Multiple refinement steps outperform one-step fusion, while removing the behavioral modality causes a much larger degradation than removing the semantic modality. This suggests that, in dynamic graphs, relation learning is often better modeled as a trajectory than as a single fusion event.

5. Applications and empirical patterns

In retrieval, relation-aware graph models consistently report gains over feature-only baselines. GIN reported 17% improvement on accuracy for the best case in cross-modal information retrieval (Yu et al., 2018). SCKR improved average MAP from 61.0 to 64.4 on Eng-Wiki and from 17.7 to 24.9 on CMPlaces by integrating semantic, co-occurrence, and knowledge relations (Yu et al., 2018). SGM reported that removing relationship information drops R@1 scores by 16–20%, indicating that explicit edge modeling is central rather than auxiliary (Wang et al., 2019).

In sentiment classification and relation extraction, relation graphs support fine-grained association rather than coarse multimodal pooling. SeqCSG reported 79.3 accuracy and 75.0 F1 on Twitter2015, and 74.6 accuracy and 73.2 F1 on Twitter2017 (Huang et al., 2022). In multimodal relation extraction, the CMG-based denoising-and-topic framework outperformed prior systems by +1.75% Accuracy and +2.08% F1, and ablations showed performance drops when either the cross-modal graph or the graph information bottleneck refinement was removed (Wu et al., 2023).

In generation and summarization, graph structure is closely tied to interpretability. CMG for video captioning reported BLEU@4 44.9, METEOR 29.6, ROUGE-L 62.9, and CIDEr 53.0 on MSR-VTT, with CIDEr 55.9 under reinforcement learning; on MSVD it reported BLEU@4 59.5, METEOR 38.8, ROUGE-L 76.2, and CIDEr 107.3 (Wang et al., 2021). CSS-GR reported approximately 2%+ ROUGE-L gains on TVSum and VMSMO and attributed them to the combination of graph reasoning, dynamic connectivity, and state-space global modeling (Kim et al., 26 Mar 2025).

In grounding, localization, and reasoning, cross-modal relation graphs are often used as compact external structure. SceneGraphLoc localizes a query image against 3D scene graphs and achieves performance close to state-of-the-art image-database methods while requiring three orders-of-magnitude less storage and operating orders-of-magnitude faster (Miao et al., 2024). AS3D uses semantic-spatial scene graphs and graph attention for relationship-oriented 3D visual grounding, with especially strong results in settings with multiple similar distractors (Xiao et al., 7 May 2025). MR-MKG improves multimodal reasoning with MMKG-grounded prompts and reports 93.63% average accuracy on ScienceQA for the FLAN-UL2-19B variant, while Graph4MM reports a 6.93% average improvement across generative and discriminative tasks (Lee et al., 2024, Ning et al., 19 Oct 2025).

6. Recurring tensions, misconceptions, and research directions

A recurring tension is the trade-off between relational expressivity and information contamination. Several systems explicitly identify noise as a first-order problem. The multimodal relation extraction framework applies structure refinement under the graph information bottleneck principle to discard less-informative nodes and edges (Wu et al., 2023). The MCER model uses a differential Transformer to enhance temporally consistent information while suppressing time-irrelevant noise in audio and video (Liu et al., 22 Mar 2026). MM-ORIENT argues that direct multimodal interaction can propagate noise across modalities and instead reconstructs monomodal features using neighborhoods decided by another modality (Rehman et al., 22 Aug 2025).

Another tension concerns compression versus modality preservation. One-shot fusion is repeatedly challenged in the cited work. PRISM contrasts iterative posterior refinement with one-shot fusion and reports that multiple refinement steps are more effective (Chang et al., 7 May 2026). CoMAG argues that uniformly fused representations cause task-agnostic propagation and over-compressed fusion, then introduces reliable context learning and modality-preserving hop-token alignment to retain modality-specific cues while producing graph and modality representations from one forward pass (Zhang et al., 12 Jun 2026).

A common misconception is that richer structure always means denser structure. Several results indicate the opposite. RR-Net reports that excessive inter-edges harm performance and that selective high-confidence inter-edges are preferable (Li et al., 2021). CSS-GR reports a smaller but notable drop when graph connectivity is made static rather than dynamic (Kim et al., 26 Mar 2025). Graph4MM argues that the graph should guide interaction rather than be appended as a standalone modality (Ning et al., 19 Oct 2025). This suggests that current cross-modal relation graph research favors selective, reliability-aware, and task-adaptive structure over indiscriminate densification.

A plausible implication is that future systems will continue to move toward dynamic edge construction, modality-preserving alignment, and iterative or temporally-aware refinement. The strongest evidence for that direction comes from work on time-variant relation supervision, task-aware reliable contexts, dynamic graph updates, and semantic–behavioral posterior refinement, all of which treat cross-modal relation graphs not as static data structures but as adaptive mechanisms for multimodal reasoning (Wang et al., 2023, Zhang et al., 12 Jun 2026, Kim et al., 26 Mar 2025, Chang et al., 7 May 2026).