DGMRec: Disentangling & Generating Modality Recommender

• The paper introduces a modularized framework that disentangles modality features into general and specific components using information-theoretic objectives.
• It employs generative modeling to synthesize high-quality features for missing modalities by leveraging available cross-modal data and user preference embeddings.
• It integrates graph-based refinement and contrastive learning to enhance cross-modal retrieval and maintain robust performance under incomplete data scenarios.

Disentangling and Generating Modality Recommender (DGMRec) denotes a class of multimodal recommendation models serving two intertwined objectives: (1) disentangling modality representations into general (shared across modalities) and specific (unique to each modality) components and (2) generating high-quality features for missing modalities by leveraging the rich cross-modal and user preference structure present in recommendation contexts. This comprehensive approach is motivated by the realistic scenario where user–item interactions involve diverse and sometimes incomplete multimodal data (e.g., text, image, audio), and robust, interpretable, and adaptable recommendation systems must intelligently compensate for missing information and articulate the roles played by different modalities.

1. Disentangling Modality Representations

The central architectural principle of DGMRec frameworks is a modularized feature partitioning process grounded in information theory. Let $X_m$ be the raw feature of modality $m$. For each modality, DGMRec produces:

• A general feature, $E_m^g = f^g(h_m(X_m))$, which resides in a common latent space, designed to capture modality-invariant semantics (i.e., information shared across all observed modalities such as item semantics).
• A specific feature, $E_m^s = f_m^s(X_m)$, uniquely parameterized per modality, encapsulating distinct modality properties (e.g., visual style, speech prosody, text phrasing).

The separation is maintained and validated using information-theoretic objectives. Mutual information upper bounds—such as CLUB losses—provide explicit regularization to ensure that $E_m^g$ and $E_m^s$ capture non-overlapping sources of variation.

To further promote invariance across modalities for $E_m^g$, DGMRec implements an InfoNCE loss, aligning general features of the same item across modalities via contrastive learning:

$$L_{\mathrm{InfoNCE}} = \sum_{i=1}^{|I|} -\log \frac{ \exp(\langle \bar{e}_i^{g, m}, \bar{e}_i^{g, m'} \rangle) }{ \sum_{j=1}^{|I|} \exp(\langle \bar{e}_i^{g, m}, \bar{e}_j^{g, m'} \rangle) }.$$

This architectural schema enables nuanced representation of both modality-shared and modality-unique factors, in contrast to conventional approaches that entangle all information into a single latent space (Kim et al., 23 Apr 2025).

2. Generation and Recovery of Missing Modalities

A distinguishing innovation of DGMRec is explicit handling of incomplete modality scenarios through generative modeling. When one or more modalities are missing at inference or training time, DGMRec synthesizes high-quality surrogate features:

• General Feature Generation: Given available modalities $M'$, a generator $G_m^g$ takes the concatenation of the general features from $M'$, i.e.,

$$\hat{E}_m^g = G_m^g\left( \bigoplus_{m' \in M'} \bar{E}_{m'}^g \right),$$

thus producing a modality-invariant estimate by aligning the available shared features.

• Specific Feature Generation: To supplement missing modality-specific details, DGMRec leverages user modality preference embeddings. For an item $i$, modality $m$, user-level preferences $p_{u,m}$ (with $u$ in the neighbor set $N_i$) are averaged:

$$p_{i,m} = \frac{1}{|N_i|} \sum_{u \in N_i} p_{u,m}.$$

This preference vector is then passed to a generator $G_m^s$ to yield the item-specific feature $\hat{E}_m^s$.

The final reconstruction is performed via a modality-specific decoder $f_m^{dec}$:

$$\bar{X}_m = f_m^{dec}(\bar{E}_m^g \oplus \bar{E}_m^s),$$

where bar notation denotes GCN-refined feature vectors. This mechanism avoids simplistic nearest-neighbor imputations and employs the joint structure of available modalities and user interactions to synthesize plausible, informative features for downstream recommendation (Kim et al., 23 Apr 2025).

3. User Modality Preferences and Alignment

DGMRec explicitly incorporates user-specific preferences over modalities. Each user $u$ has a modality preference embedding $P_{u,m}$, which serves dual roles:

• Denoising: The item’s modality representation is refined via element-wise product with the sigmoid-activated preference embedding, i.e., $\tilde{E}_m = E_m \odot \sigma(p_{i,m})$, which emphasizes the informative subspace tailored to the user's historical interactions.
• Feature Generation: As described above, the learned $p_{i,m}$ informs the generator of missing modality-specific features.

To further bind the modality-augmented representations to the collaborative filtering signal, DGMRec employs behavior–modality ($L_{\mathrm{align}}$) and user–item alignment objectives, enforcing that the learned features are not only semantically rich but also conducive to accurate recommendation.

4. Graph-Refined Relational Structure

The DGMRec framework utilizes item–item graphs to capture high-order relationships and semantic affinities across items. For each modality, a similarity matrix $S_m$ is computed using cosine similarity over the raw features:

$$S_{m}(i,j) = \frac{ x_{i,m}^\top x_{j,m} }{ \| x_{i,m} \| \| x_{j,m} \| }.$$

After synthetic modality features are generated for missing cases, a new similarity graph $\hat{S}_m$ is constructed. The item–item graph is adaptively updated via a convex combination:

$S_m \leftarrow \alpha S_m + (1-\alpha) \hat{S}_m,$

with $\alpha$ a learnable or tunable parameter, maintaining the relevance of both observed and generated modality features in the final multi-graph structure for GCN propagation.

5. Training Losses and Optimization

The learning objective for DGMRec integrates the following loss components:

• BPR Loss: $L_{\mathrm{BPR}}$ for optimizing ranking quality.
• Reconstruction Loss: $L_{\mathrm{recon}}$ for raw feature reconstruction.
• Generation Loss: $L_{\mathrm{gen}}$ for matching generated and ground-truth features.
• Disentanglement Loss: $L_{\mathrm{disentangle}}$, as a sum of the CLUB and InfoNCE losses, promoting separation and alignment of general vs. specific features.
• Alignment Loss: $L_{\mathrm{align}}$ to fuse CF and modality signals.

The final loss is a weighted combination:

$$L = L_{\mathrm{BPR}} + L_{\mathrm{recon}} + L_{\mathrm{gen}} + \lambda_1 L_{\mathrm{disentangle}} + \lambda_2 L_{\mathrm{align}}.$$

6. Empirical Performance and Cross-Modal Retrieval

In benchmarking on Amazon and TikTok datasets under a variety of missing modality and cold-start regimes, DGMRec demonstrates robust performance, consistently surpassing both classic multimodal recommendation methods (e.g., LGMRec, DAMRS) and dedicated missing-modality techniques (e.g., MILK, CI2MG). Its performance degrades much more gracefully under increasing missing ratios.

A particular strength is in cross-modal retrieval: owing to the generative design that reconstructs both general and specific missing features, DGMRec supports similarity-based retrieval between items with disjoint observed modalities. Experimental results indicate substantial improvements in Hit@10 and Hit@20 for missing-modality cross-modal retrieval compared to nearest neighbor imputation baselines, establishing its suitability for practical systems where data incompleteness is routine (Kim et al., 23 Apr 2025).

7. Real-World Applications and Open Directions

DGMRec’s architecture supports robust, modality-flexible recommendations in practical environments such as:

• E-commerce, where item images or text may be partially unavailable;
• Social media or UGC-driven platforms, where user or item content is frequently incomplete;
• Cold-start and cross-modal matching, e.g., finding visually similar items given only textual cues.

The explicit disentanglement of modality-specific and modality-shared features, the use of user-driven preference modeling for generating missing features, and adaptive graph construction collectively yield a model highly robust to missing or sparse data scenarios.

Current limitations include the computational cost of joint training, sensitivity to parameter selection (e.g., $\alpha$, representation dimensionality), and reliance on sufficient interaction data to learn robust user discrimination over modalities. Further investigation into lightweight or online-generation variants, tighter integration with causal inference frameworks, and extensions to additional modalities (e.g., audio, video) remains a fertile area for future work.

---

In summary, DGMRec formalizes a principled architecture for disentangling and generating modality representations in multimodal recommendation systems, with an emphasis on missing modality contexts. Its methodological innovations span information-theoretic disentanglement, user preference-driven generation, and graph-based refinement, establishing new standards of adaptability, robustness, and interpretability in modern multimodal recommender systems (Kim et al., 23 Apr 2025).