CrossGMMI-DUKGLR: Unified KG & Multimodal Rec Framework

• CrossGMMI-DUKGLR is a unified knowledge graph-based framework integrating multimodal representation learning, graph reasoning, and cross-graph alignment for personalized recommendations.
• It leverages advanced encoders like BERT and CLIP to capture fine-grained textual and visual semantics while addressing cold-start challenges.
• The framework introduces a two-phase process with mutual-information contrastive learning for effective cross-graph alignment and recommendation fine-tuning.

CrossGMMI-DUKGLR, abbreviated from Cross-Graph Cross-Modal Mutual Information-Driven Unified Knowledge Graph Learning and Recommendation Framework, is a knowledge graph-based personalized multimodal recommendation framework proposed to integrate multimodal representation learning, structural knowledge graph reasoning, cross-graph entity alignment, and downstream recommendation within a single pipeline (Fang, 3 Sep 2025). It is designed for recommendation settings in which knowledge graphs (KGs), user-item interactions, and item-side multimodal information, especially text and images, are simultaneously available. The framework is motivated by three identified deficiencies in earlier methods: insufficient modeling of fine-grained modal interaction, weak exploitation of higher-order graph dependencies, and the lack of a unified architecture that jointly handles cross-graph alignment, multimodal fusion, mutual-information-based knowledge sharing, and recommendation optimization (Fang, 3 Sep 2025).

1. Problem formulation and motivation

CrossGMMI-DUKGLR is situated in the personalized recommendation setting in which the goal is to predict a user’s preference for an item for top-$N$ ranking or binary recommendation. The input space includes users, items, user-item interaction data $D=\{(u,v,y)\}$, knowledge graph entities, knowledge graph relations / triples, textual/attribute data, visual/image data, and cross-graph aligned entity pairs used during pre-training (Fang, 3 Sep 2025).

The motivating argument is that conventional recommendation methods based on collaborative filtering or isolated attribute modeling do not adequately capture rich semantic item information, nuanced user interests, multi-hop semantic relations, interpretability requirements, or robustness under sparsity and cold-start conditions (Fang, 3 Sep 2025). Within that diagnosis, knowledge graphs are treated as a source of semantic-level auxiliary information, multi-hop reasoning structure, item-item and item-attribute relations, and more interpretable recommendation signals. Multimodal signals are treated as complementary evidence: text may describe functionality, semantics, plot, or attributes, whereas images may convey style, appearance, or aesthetics (Fang, 3 Sep 2025).

The framework is therefore presented as an answer to two central gaps. First, many multimodal recommendation systems either process modalities independently or fuse them through simple concatenation or weighted combination, which is described as inadequate for fine-grained correlations between text and images. Second, many KG-based recommenders use only one or two graph layers and therefore underexploit multi-hop entity relationships and user interest chains. A further stated gap is the absence of a unified framework that can jointly handle cross-graph entity alignment, intra-graph multimodal fusion, mutual-information-based knowledge sharing, and recommendation fine-tuning (Fang, 3 Sep 2025).

2. Framework composition and pipeline

The acronym encodes the framework’s main design commitments. Cross-Graph refers to modeling multiple knowledge graphs from diverse sources, including preliminary entity alignment, cross-graph subgraph construction, and mutual-information-based agreement between corresponding entity representations. Cross-Modal refers to interaction between textual and visual modalities via pre-trained encoders and cross-attention. Mutual Information-Driven refers to an InfoNCE-based mutual information objective used to align entity representations across graphs. Unified Knowledge Graph Learning refers to the integration of multimodal feature extraction, structural graph encoding, cross-graph alignment, and recommendation into one modular system. Recommendation refers to the downstream scoring of user-item pairs with learned embeddings (Fang, 3 Sep 2025).

The end-to-end workflow is described as a two-phase process. In the pre-training phase, the framework preprocesses and aligns multiple KGs, constructs cross-graph subgraphs, encodes multimodal and structural signals, and applies mutual-information contrastive learning across aligned entities. In the fine-tuning phase, the aligned multimodal and structural representations are concatenated into unified vectors and then optimized on user-item interaction data for recommendation (Fang, 3 Sep 2025).

The preprocessing stage includes unified preprocessing on knowledge graphs from diverse sources, preliminary entity alignment, redundancy noise reduction, and construction of cross-graph subgraphs using a sampling strategy that includes $n$-hop structural neighborhoods and multimodal information. For the recommendation scenario discussed in the paper, the KG contains item-linked entities, relation triples, and multimodal attributes; in the MovieLens setting, movie entities are mapped to KG nodes with unified IDs (Fang, 3 Sep 2025).

The framework’s treatment of users is less fully specified than its treatment of items and entities. The paper states that for user $u$ and item $v$, vectors $h_u$ and $h_v$ are obtained “from the above encoders,” and it further introduces learned user profile embeddings $p_u$ and item profile embeddings $q_i$ for personalized fusion. This suggests that user representation is at least interaction-derived and may share components with the broader encoder stack, but the exact user encoder architecture is not formally detailed (Fang, 3 Sep 2025).

3. Multimodal encoders and cross-modal fusion

The multimodal side of CrossGMMI-DUKGLR is built around modality-specific encoders of the form

$h^{(k)} = f_k(m^{(k)};\theta_k),$

where $D=\{(u,v,y)\}$0 is modality $D=\{(u,v,y)\}$1, $D=\{(u,v,y)\}$2 is the corresponding encoder, $D=\{(u,v,y)\}$3 denotes encoder parameters, and $D=\{(u,v,y)\}$4 is the resulting modality embedding (Fang, 3 Sep 2025). In the concrete recommendation workflow, the explicitly developed modalities are text and images. Textual attributes are encoded with BERT,

$D=\{(u,v,y)\}$5

and visual inputs are encoded with CLIP,

$D=\{(u,v,y)\}$6

The generalized description also mentions ResNet or Vision Transformer for visual encoders and wav2vec 2.0 for