Do We Really Need to Drop Items with Missing Modalities in Multimodal Recommendation? (2408.11767v1)

Abstract: Generally, items with missing modalities are dropped in multimodal recommendation. However, with this work, we question this procedure, highlighting that it would further damage the pipeline of any multimodal recommender system. First, we show that the lack of (some) modalities is, in fact, a widely-diffused phenomenon in multimodal recommendation. Second, we propose a pipeline that imputes missing multimodal features in recommendation by leveraging traditional imputation strategies in machine learning. Then, given the graph structure of the recommendation data, we also propose three more effective imputation solutions that leverage the item-item co-purchase graph and the multimodal similarities of co-interacted items. Our method can be plugged into any multimodal RSs in the literature working as an untrained pre-processing phase, showing (through extensive experiments) that any data pre-filtering is not only unnecessary but also harmful to the performance.

• The paper presents a preprocessing pipeline that employs both traditional and graph-aware imputation techniques to retain items with missing modalities.
• Graph-aware methods, such as MultiHop and PersPageRank, significantly enhance feature propagation and improve recommendation accuracy.
• Experimental results on Amazon Reviews data demonstrate that imputing missing modalities can match or exceed the performance of dropped item setups.

Overview of "Do We Really Need to Drop Items with Missing Modalities in Multimodal Recommendation?"

The paper entitled "Do We Really Need to Drop Items with Missing Modalities in Multimodal Recommendation?" by Daniele Malitesta et al. addresses a critical question in the field of multimodal recommender systems (RSs) regarding the handling of items with missing modalities. Traditionally, RSs drop items with missing information to streamline the recommendation process. However, the authors challenge this conventional wisdom by proposing a preprocessing pipeline to impute missing multimodal features, leveraging both traditional and novel graph-based imputation strategies.

Motivation and Context

Multimodal recommender systems enhance user-item interaction data with additional information, such as visual, textual, and audio features. This added information is crucial in scenarios with sparse user-item interactions, making RSs more robust and accurate. However, real-world datasets often exhibit missing modalities, which are typically handled by excluding such items from the dataset. This practice leads to a reduction in the dataset size and potential loss of valuable information, which could adversely affect the recommendation performance.

Methodology

Problem Formalization

The authors formalize the issue of missing modalities within the RS framework. They model the recommendation data as a user-item interaction matrix $\mathbf{R}$, and the multimodal features of items as tensor $\mathbf{F}$. Missing modalities imply the unavailability of specific feature vectors for certain items.

Imputation Strategies

The proposed imputation methodology is categorized into traditional and graph-aware strategies:

1. Traditional Imputation Strategies:
   • Zeros: Replaces missing features with zeros.
   • Random: Replaces missing features with random values.
   • GlobalMean: Replaces missing features with the mean of the available features.
2. Graph-Aware Imputation Strategies:
   • NeighMean: Uses the mean features of neighboring items in the item-item co-purchase graph.
   • MultiHop: Propagates features through multiple hops in the graph, smoothing the influence of neighboring items.
   • PersPageRank: Incorporates personalized PageRank for diffusion, enabling robust feature propagation at multiple hops.

These graph-aware strategies exploit item-item co-purchase relationships to infer missing modalities more accurately than traditional methods.

Experiments and Results

The paper employs the Amazon Reviews data, specifically focusing on three categories: Office Products, Digital Music, and Beauty. The datasets are split into two versions: dropped (removing items with missing modalities) and imputed (applying the proposed imputation strategies).

Performance Analysis

Experiments compare the performance of pure collaborative filtering (CF) models with their multimodal versions, both in dropped and imputed settings. Key models used include BPRMF, NGCF, VBPR, NGCF-M, LightGCN, SGL, FREEDOM, and BM3.

1. Performance Improvement:
   • Imputing missing modalities consistently retains or improves the performance gap between CF and multimodal RSs. The imputed setting often shows a higher performance improvement than the dropped one.
   • The novel graph-aware imputation methods—particularly MultiHop and PersPageRank—demonstrate superior efficacy over traditional strategies, reflecting the benefits of leveraging graph structures.
2. Sensitivity Analysis:
   • Top-$k$ sparsification and the number of propagation hops significantly impact the performance, with higher sparsification and multi-hop propagation generally boosting results.

Implications and Future Work

Practical Implications

The proposed preprocessing pipeline negates the need to drop items with missing modalities, facilitating the retention of valuable user-item interactions and enhancing recommendation performance. This approach is versatile and can be integrated into any existing multimodal RS without the need for extensive retraining, providing a practical and efficient solution to the problem of missing data.

Theoretical Contributions

The paper advances the state-of-the-art in multimodal recommendation by formalizing and addressing the issue of missing modalities with a novel graph-aware perspective. The findings advocate for a paradigm shift in handling incomplete multimodal data, emphasizing the importance of leveraging item-item interaction graphs.

Future Directions

Future research could explore:

• Integration of imputation methods directly into end-to-end training pipelines for real-time recommendations.
• Expansion to more diverse datasets and modalities, such as sensor data in IoT applications.
• Development of more refined graph-based imputation techniques, such as attention mechanisms to weigh the influence of different neighbors dynamically.

Conclusion

The research by Malitesta et al. offers a comprehensive and nuanced perspective on handling missing modalities in multimodal recommendation. By proposing both traditional and innovative graph-based imputation strategies, the authors provide clear evidence that imputing rather than dropping items with missing features leads to substantial improvements in recommendation performance. This work serves as a significant contribution to the field, advocating for more sophisticated handling of incomplete multimodal data in future recommender systems.