Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Unleashing the Power of Imbalanced Modality Information for Multi-modal Knowledge Graph Completion (2402.15444v1)

Published 22 Feb 2024 in cs.AI, cs.CL, cs.LG, and cs.MM

Abstract: Multi-modal knowledge graph completion (MMKGC) aims to predict the missing triples in the multi-modal knowledge graphs by incorporating structural, visual, and textual information of entities into the discriminant models. The information from different modalities will work together to measure the triple plausibility. Existing MMKGC methods overlook the imbalance problem of modality information among entities, resulting in inadequate modal fusion and inefficient utilization of the raw modality information. To address the mentioned problems, we propose Adaptive Multi-modal Fusion and Modality Adversarial Training (AdaMF-MAT) to unleash the power of imbalanced modality information for MMKGC. AdaMF-MAT achieves multi-modal fusion with adaptive modality weights and further generates adversarial samples by modality-adversarial training to enhance the imbalanced modality information. Our approach is a co-design of the MMKGC model and training strategy which can outperform 19 recent MMKGC methods and achieve new state-of-the-art results on three public MMKGC benchmarks. Our code and data have been released at https://github.com/zjukg/AdaMF-MAT.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. Maksym Andriushchenko and Nicolas Flammarion. 2020. Understanding and improving fast adversarial training. In Proc. of NeurIPS.
  2. Beit: BERT pre-training of image transformers. In Proc. of ICLR.
  3. Translating embeddings for modeling multi-relational data. In Proc. of NeurIPS.
  4. Liwei Cai and William Yang Wang. 2018. KBGAN: adversarial learning for knowledge graph embeddings. In Proc. of NAACL.
  5. Cross-modal knowledge graph contrastive learning for machine learning method recommendation. In Proc. of ACM MM.
  6. Otkge: Multi-modal knowledge graph embeddings via optimal transport. Proc. of NeurIPS.
  7. Pairre: Knowledge graph embeddings via paired relation vectors. In Proc. of ACL.
  8. Knowledge graphs meet multi-modal learning: A comprehensive survey. CoRR, abs/2402.05391.
  9. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL.
  10. Generative adversarial nets. In Proc. of NeurIPS.
  11. Explaining and harnessing adversarial examples. In Proc. of ICLR.
  12. Openke: An open toolkit for knowledge embedding. In Proc. of EMNLP.
  13. Knowledge graph embedding via dynamic mapping matrix. In ACL (1), pages 687–696. The Association for Computer Linguistics.
  14. Adversarial machine learning at scale. In Proc. of ICLR.
  15. Dbpedia - A large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web.
  16. Knowledge graph contrastive learning based on relation-symmetrical structure. IEEE Transactions on Knowledge and Data Engineering, pages 1–12.
  17. Learn from relational correlations and periodic events for temporal knowledge graph reasoning. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1559–1568.
  18. Reasoning over different types of knowledge graphs: Static, temporal and multi-modal. arXiv preprint arXiv:2212.05767.
  19. MMKG: multi-modal knowledge graphs. In The Semantic Web - 16th International Conference, ESWC 2019, Portorovz, Slovenia, June 2-6, 2019, Proceedings.
  20. MMKRL: A robust embedding approach for multi-modal knowledge graph representation learning. Appl. Intell.
  21. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml.
  22. BAGAN: data augmentation with balancing GAN. CoRR.
  23. Adversarial training methods for semi-supervised text classification. In Proc. of ICLR.
  24. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proc. of EMNLP.
  25. A multimodal translation-based approach for knowledge graph representation learning. In Proc. of AACL.
  26. Adversarial training for free! In Proc. of NeurIPS.
  27. Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In Proc. of ICLR.
  28. Multi-modal knowledge graphs for recommender systems. In Proc. of CIKM.
  29. Rotate: Knowledge graph embedding by relational rotation in complex space. In Proc. of ICLR.
  30. Orthogonal relation transforms with graph context modeling for knowledge graph embedding. In Proc. of ACL.
  31. Positive-unlabeled learning with adversarial data augmentation for knowledge graph completion. In Proc. of IJCAI.
  32. Knowledge graph completion via complex tensor factorization. J. Mach. Learn. Res.
  33. J. D. Tygar. 2011. Adversarial machine learning. IEEE Internet Comput.
  34. Is visual context really helpful for knowledge graph? A representation learning perspective. In MM ’21: ACM Multimedia Conference, Virtual Event, China, October 20 - 24, 2021.
  35. Incorporating GAN for negative sampling in knowledge representation learning. In Proc. of AAAI.
  36. Knowledge graph embedding: A survey of approaches and applications. IEEE Trans. Knowl. Data Eng.
  37. Multimodal data enhanced representation learning for knowledge graphs. In Proc. of IJCNN.
  38. Image-embodied knowledge representation learning. In Proc. of IJCAI.
  39. Relation-enhanced negative sampling for multimodal knowledge graph completion. In Proc. of ACM MM.
  40. Embedding entities and relations for learning and inference in knowledge bases. In Proc. of ICLR.
  41. QA-GNN: reasoning with language models and knowledge graphs for question answering. In Proc. of NAACL.
  42. Modality-aware negative sampling for multi-modal knowledge graph embedding. CoRR.
  43. Knowledgeable preference alignment for llms in domain-specific question answering. CoRR, abs/2311.06503.
  44. Making large language models perform better in knowledge graph completion. CoRR, abs/2310.06671.
  45. Yichi Zhang and Wen Zhang. 2022. Knowledge graph completion with pre-trained multimodal transformer and twins negative sampling. CoRR.
  46. Mose: Modality split and ensemble for multimodal knowledge graph completion. In Proc. of EMNLP.
  47. MMKG: multi-modal knowledge graphs. In ESWC, volume 11503 of Lecture Notes in Computer Science, pages 459–474. Springer.
  48. Yago: a core of semantic knowledge. In WWW, pages 697–706. ACM.
  49. Denny Vrandecic and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Commun. ACM, 57(10):78–85.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now