Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Multi-modal Misinformation Detection: Approaches, Challenges and Opportunities (2203.13883v7)

Published 25 Mar 2022 in cs.LG, cs.AI, cs.CV, cs.CY, cs.MM, and cs.SI

Abstract: As social media platforms are evolving from text-based forums into multi-modal environments, the nature of misinformation in social media is also transforming accordingly. Taking advantage of the fact that visual modalities such as images and videos are more favorable and attractive to the users and textual contents are sometimes skimmed carelessly, misinformation spreaders have recently targeted contextual connections between the modalities e.g., text and image. Hence many researchers have developed automatic techniques for detecting possible cross-modal discordance in web-based content. We analyze, categorize and identify existing approaches in addition to challenges and shortcomings they face in order to unearth new research opportunities in the field of multi-modal misinformation detection.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (114)
  1. Identifying Misinformation from Website Screenshots. Proceedings of the International AAAI Conference on Web and Social Media 15, 1 (May 2021), 2–13. https://ojs.aaai.org/index.php/ICWSM/article/view/18036
  2. HiJoD: Semi-Supervised Multi-aspect Detection of Misinformation using Hierarchical Joint Decomposition. In ECML/PKDD.
  3. KNH: Multi-View Modeling with K-Nearest Hyperplanes Graph for Misinformation Detection. CoRR abs/2102.07857 (2021). arXiv:2102.07857 https://arxiv.org/abs/2102.07857
  4. VQA: Visual Question Answering. Int. J. Comput. Vision 123, 1 (may 2017), 4–31. https://doi.org/10.1007/s11263-016-0966-6
  5. A Survey on Multimodal Disinformation Detection.
  6. Flamingo: a Visual Language Model for Few-Shot Learning. ArXiv abs/2204.14198 (2022). https://api.semanticscholar.org/CorpusID:248476411
  7. Santiago Alonso-Bartolome and Isabel Segura-Bedmar. 2021. Multimodal Fake News Detection. CoRR abs/2112.04831 (2021). arXiv:2112.04831 https://arxiv.org/abs/2112.04831
  8. Santiago Alonso-Bartolome and Isabel Segura-Bedmar. 2021. Multimodal Fake News Detection. arXiv:2112.04831 [cs.CL]
  9. Multimodal fusion for multimedia analysis: a survey. Multimedia Systems 16 (2010), 345–379.
  10. Semi-Supervised Learning and Graph Neural Networks for Fake News Detection (ASONAM ’19). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3341161.3342958
  11. Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformation. arXiv:2112.08594 [cs.CV]
  12. Detection and visualization of misleading content on Twitter. International Journal of Multimedia Information Retrieval 7, 1 (2018), 71–86. https://doi.org/10.1007/s13735-017-0143-x
  13. Early, intermediate and late fusion strategies for robust deep learning-based multimodal action recognition. Machine Vision and Applications 32 (11 2021). https://doi.org/10.1007/s00138-021-01249-8
  14. Language Models are Few-Shot Learners. ArXiv abs/2005.14165 (2020). https://api.semanticscholar.org/CorpusID:218971783
  15. Threats to Online Advertising and Countermeasures: A Technical Survey. Digital Threats: Research and Practice 1, 2, Article 11 (may 2020), 27 pages. https://doi.org/10.1145/3374136
  16. Pro-Cap: Leveraging a Frozen Vision-Language Model for Hateful Meme Detection. ArXiv abs/2308.08088 (2023). https://api.semanticscholar.org/CorpusID:260925510
  17. Prompting for Multimodal Hateful Meme Classification. In Conference on Empirical Methods in Natural Language Processing. https://api.semanticscholar.org/CorpusID:256461095
  18. MMCoVaR: Multimodal COVID-19 Vaccine Focused Data Repository for Fake News Detection and a Baseline Architecture for Classification (ASONAM ’21). Association for Computing Machinery, New York, NY, USA, 31–38. https://doi.org/10.1145/3487351.3488346
  19. Microsoft COCO Captions: Data Collection and Evaluation Server. CoRR abs/1504.00325 (2015). arXiv:1504.00325 http://arxiv.org/abs/1504.00325
  20. Hyewon Choi and Youngjoong Ko. 2022. Effective fake news video detection using domain knowledge and multimodal data fusion on youtube. Pattern Recognition Letters 154 (2022), 44–52. https://doi.org/10.1016/j.patrec.2022.01.007
  21. Anshika Choudhary and Anuja Arora. 2021. ImageFake: An Ensemble Convolution Models Driven Approach for Image Based Fake News Detection. In 2021 7th International Conference on Signal Processing and Communication (ICSC). 182–187. https://doi.org/10.1109/ICSC53193.2021.9673192
  22. Multimodal Propaganda Detection Via Anti-Persuasion Prompt enhanced contrastive learning. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 1–5. https://doi.org/10.1109/ICASSP49357.2023.10096771
  23. Limeng Cui and Dongwon Lee. 2020. CoAID: COVID-19 Healthcare Misinformation Dataset. arXiv:2006.00885 [cs.SI]
  24. DETERRENT: Knowledge Guided Graph Attention Network for Detecting Healthcare Misinformation (KDD ’20). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3394486.3403092
  25. Can Machines Learn to Detect Fake News? A Survey Focused on Social Media. In HICSS.
  26. Dipto Das. 2019. A MULTIMODAL APPROACH TO SARCASM DETECTION ON SOCIAL MEDIA. Ph. D. Dissertation.
  27. GAME-ON: Graph Attention Network based Multimodal Fusion for Fake News Detection.
  28. Performance Evaluation of Early and Late Fusion Methods for Generic Semantics Indexing. Pattern Anal. Appl. 17, 1 (feb 2014), 37–50. https://doi.org/10.1007/s10044-013-0336-8
  29. Image and Text fusion for UPMC Food-101 using BERT and CNNs. In 2020 35th International Conference on Image and Vision Computing New Zealand (IVCNZ). 1–6. https://doi.org/10.1109/IVCNZ51579.2020.9290622
  30. Few-shot fake news detection via prompt-based tuning. Journal of Intelligent & Fuzzy Systems Preprint (2023), 1–10.
  31. Multimodal Multi-image Fake News Detection. In 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA). 647–654. https://doi.org/10.1109/DSAA49011.2020.00091
  32. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017), 6325–6334.
  33. Semi-supervised Content-Based Detection of Misinformation via Tensor Embeddings. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). 322–325. https://doi.org/10.1109/ASONAM.2018.8508241
  34. VizWiz Grand Challenge: Answering Visual Questions from Blind People. (02 2018).
  35. Captioning Images Taken by People Who Are Blind. 417–434. https://doi.org/10.1007/978-3-030-58520-4˙25
  36. An ensemble machine learning approach through effective feature extraction to classify fake news. Future Generation Computer Systems 117 (2021), 47–58. https://doi.org/10.1016/j.future.2020.11.022
  37. Stefan Helmstetter and Heiko Paulheim. 2018. Weakly Supervised Learning for Fake News Detection on Twitter. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). 274–277. https://doi.org/10.1109/ASONAM.2018.8508520
  38. Benjamin D. Horne and Sibel Adali. 2017. This Just In: Fake News Packs a Lot in Title, Uses Simpler, Repetitive Content in Text Body, More Similar to Satire than Real News. CoRR abs/1703.09398 (2017). arXiv:1703.09398 http://arxiv.org/abs/1703.09398
  39. Drew Hudson and Christopher Manning. 2019. GQA: a new dataset for compositional question answering over real-world images.
  40. Fighting Fake News: Image Splice Detection via Learned Self-Consistency.
  41. Deep learning for misinformation detection on online social networks: a survey and new perspectives. Social Network Analysis and Mining 10 (12 2020). https://doi.org/10.1007/s13278-020-00696-x
  42. AENeT: an attention-enabled neural architecture for fake news detection using contextual features. Neural Computing and Applications 34 (01 2022). https://doi.org/10.1007/s00521-021-06450-4
  43. Fake News Detection Using BERT-VGG19 Multimodal Variational Autoencoder. In 2021 IEEE 8th Uttar Pradesh Section International Conference on Electrical, Electronics and Computer Engineering (UPCON). 1–5. https://doi.org/10.1109/UPCON52273.2021.96676
  44. Fake news detection via knowledgeable prompt learning. Information Processing & Management 59, 5 (2022), 103029.
  45. Similarity-Aware Multimodal Prompt Learning for Fake News Detection. ArXiv abs/2304.04187 (2023). https://api.semanticscholar.org/CorpusID:256654907
  46. Multimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs. In Proceedings of the 25th ACM International Conference on Multimedia (Mountain View, California, USA) (MM ’17). Association for Computing Machinery, New York, NY, USA, 795–816. https://doi.org/10.1145/3123266.3123454
  47. NewsBag: A multimodal benchmark dataset for fake news detection. , 138- 145 pages.
  48. TRANSFAKE: Multi-task Transformer for Multimodal Enhanced Fake News Detection. In 2021 International Joint Conference on Neural Networks (IJCNN). 1–8. https://doi.org/10.1109/IJCNN52387.2021.9533433
  49. S. Wang K. Shu, A. Sliva and H. Liu. 2019. Beyond news contents: the role of social context for fake news detection. In WSDM ’19 Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. 312–320.
  50. MVAE: Multimodal Variational Autoencoder for Fake News Detection. In The World Wide Web Conference (San Francisco, CA, USA) (WWW ’19). Association for Computing Machinery, New York, NY, USA, 2915–2921. https://doi.org/10.1145/3308558.3313552
  51. The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes. CoRR abs/2005.04790 (2020). arXiv:2005.04790 https://arxiv.org/abs/2005.04790
  52. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations. Int. J. Comput. Vision 123, 1 (may 2017), 32–73. https://doi.org/10.1007/s11263-016-0981-7
  53. Srijan Kumar and Neil Shah. 2018. False information on web and social media: A survey. arXiv preprint arXiv:1804.08559 (2018).
  54. Rina Kumari and Asif Ekbal. 2021. AMFB: Attention based multimodal Factorized Bilinear Pooling for multimodal Fake News Detection. Expert Systems with Applications 184 (2021), 115412. https://doi.org/10.1016/j.eswa.2021.115412
  55. Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects. Proc. IEEE 103 (2015), 1449–1477.
  56. The Power of Scale for Parameter-Efficient Prompt Tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 3045–3059. https://doi.org/10.18653/v1/2021.emnlp-main.243
  57. A Multi-Modal Method for Satire Detection using Textual and Visual Cues. ArXiv abs/2010.06671 (2020).
  58. MM-COVID: A Multilingual and Multimodal Data Repository for Combating COVID-19 Disinformation. arXiv:2011.04088 [cs.SI]
  59. Zero-shot rumor detection with propagation structure via prompt learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 5213–5221.
  60. Yi-Ju Lu and Cheng-Te Li. 2020. GCAN: Graph-aware Co-Attention Networks for Explainable Fake News Detection on Social Media. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 505–514. https://doi.org/10.18653/v1/2020.acl-main.48
  61. Héctor P. Martínez and Georgios N. Yannakakis. 2014. Deep Multimodal Fusion: Combining Discrete Events and Continuous Signals. In Proceedings of the 16th International Conference on Multimodal Interaction (Istanbul, Turkey) (ICMI ’14). Association for Computing Machinery, New York, NY, USA, 34–41. https://doi.org/10.1145/2663204.2663236
  62. AIMH at SemEval-2021 Task 6: Multimodal Classification Using an Ensemble of Transformer Models. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021). Association for Computational Linguistics, Online, 1020–1026. https://doi.org/10.18653/v1/2021.semeval-1.140
  63. Fakeddit: A New Multimodal Benchmark Dataset for Fine-grained Fake News Detection. In Proceedings of the 12th Language Resources and Evaluation Conference. European Language Resources Association, Marseille, France, 6149–6157. https://aclanthology.org/2020.lrec-1.755
  64. Dan Saattrup Nielsen and Ryan McConville. 2022. MuMiN: A Large-Scale Multilingual Multimodal Fact-Checked Misinformation Social Network Dataset. arxiv (2022). https://arxiv.org/abs/2202.11684
  65. OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL]
  66. Improving Fake News Detection by Using an Entity-Enhanced Framework to Fuse Diverse Multimodal Clues. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3474085.3481548
  67. Exploiting Multi-domain Visual Information for Fake News Detection. 518–527. https://doi.org/10.1109/ICDM.2019.00062
  68. Hierarchical Multi-Modal Contextual Attention Network for Fake News Detection. Association for Computing Machinery, New York, NY, USA, 153–162. https://doi.org/10.1145/3404835.3462871
  69. Learning Transferable Visual Models From Natural Language Supervision. In ICML.
  70. Chahat Raj and Priyanka Meel. 2022. ARCNN framework for multimodal infodemic detection. Neural Networks 146 (2022), 36–68. https://doi.org/10.1016/j.neunet.2021.11.006
  71. Zero-Shot Text-to-Image Generation. ArXiv abs/2102.12092 (2021). https://api.semanticscholar.org/CorpusID:232035663
  72. Socially Aware Multimodal Deep Neural Networks for Fake News Classification. In 2021 IEEE 4th International Conference on Multimedia Information Processing and Retrieval (MIPR). 253–259. https://doi.org/10.1109/MIPR51284.2021.00048
  73. FaceForensics++: Learning to Detect Manipulated Facial Images. CoRR abs/1901.08971 (2019). arXiv:1901.08971 http://arxiv.org/abs/1901.08971
  74. SCATE: Shared Cross Attention Transformer Encoders for Multimodal Fake News Detection. In Proceedings of the 2021 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (Virtual Event, Netherlands) (ASONAM ’21). Association for Computing Machinery, New York, NY, USA, 399–406. https://doi.org/10.1145/3487351.3490965
  75. A Multimodal Misinformation Detector for COVID-19 Short Videos on TikTok. 899–908. https://doi.org/10.1109/BigData52589.2021.9671928
  76. DEFEND: Explainable Fake News Detection. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, New York, NY, USA, 395–405. https://doi.org/10.1145/3292500.3330935
  77. Detecting Fake News With Weak Social Supervision. IEEE Intelligent Systems PP (05 2020), 1–1. https://doi.org/10.1109/MIS.2020.2997781
  78. Hierarchical Propagation Networks for Fake News Detection: Investigation and Exploitation. arXiv:1903.09196 [cs.SI]
  79. Fake News Detection on Social Media: A Data Mining Perspective. SIGKDD Explor. Newsl. 19 (2017).
  80. Understanding User Profiles on Social Media for Fake News Detection. In 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR). 430–435. https://doi.org/10.1109/MIPR.2018.00092
  81. Leveraging Multi-Source Weak Social Supervision for Early Detection of Fake News.
  82. The Role of User Profiles for Fake News Detection (ASONAM ’19). Association for Computing Machinery, New York, NY, USA, 436–439. https://doi.org/10.1145/3341161.3342927
  83. Propagation2Vec: Embedding partial propagation networks for explainable fake news early detection. Information Processing & Management 58, 5 (2021), 102618. https://doi.org/10.1016/j.ipm.2021.102618
  84. Embracing Domain Differences in Fake News: Cross-domain Fake News Detection using Multi-modal Data. (02 2021).
  85. Towards VQA Models that can Read.
  86. Inter-Modality Discordance for Multimodal Fake News Detection. In ACM Multimedia Asia (Gold Coast, Australia) (MMAsia ’21). Association for Computing Machinery, New York, NY, USA, Article 33, 7 pages. https://doi.org/10.1145/3469877.3490614
  87. SpotFake: A Multi-modal Framework for Fake News Detection. In 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM). 39–47. https://doi.org/10.1109/BigMM.2019.00-44
  88. A multimodal fake news detection model based on crossmodal attention residual and multichannel convolutional neural networks. Inf. Process. Manag. 58 (2021), 102437.
  89. Temporally evolving graph neural network for fake news detection. Information Processing & Management 58, 6 (2021), 102712. https://doi.org/10.1016/j.ipm.2021.102712
  90. Temporally evolving graph neural network for fake news detection. Information Processing & Management 58, 6 (nov 2021), 102712. https://doi.org/10.1016/j.ipm.2021.102712
  91. Link-Context Learning for Multimodal LLMs. ArXiv abs/2308.07891 (2023). https://api.semanticscholar.org/CorpusID:260899869
  92. MetaTroll: Few-shot Detection of State-Sponsored Trolls with Transformer Adapters. In Proceedings of the ACM Web Conference 2023. 1743–1753.
  93. Deepfakes and beyond: A Survey of face manipulation and fake detection. Information Fusion 64 (2020), 131–148. https://doi.org/10.1016/j.inffus.2020.06.014
  94. Inna Vogel and Meghana Meghana. 2020. Detecting Fake News Spreaders on Twitter from a Multilingual Perspective. 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA) (2020), 599–606.
  95. FMFN: Fine-Grained Multimodal Fusion Networks for Fake News Detection. Applied Sciences 12, 3 (2022). https://doi.org/10.3390/app12031093
  96. Recipe recognition with large multimodal food dataset. In 2015 IEEE International Conference on Multimedia Expo Workshops (ICMEW). 1–6. https://doi.org/10.1109/ICMEW.2015.7169757
  97. EANN: Event Adversarial Neural Networks for Multi-Modal Fake News Detection (KDD ’18). Association for Computing Machinery, New York, NY, USA, 9 pages. https://doi.org/10.1145/3219819.3219903
  98. Multimodal Emergent Fake News Detection via Meta Neural Process Networks. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (2021).
  99. Fake News Detection via Knowledge-Driven Multimodal Graph Convolutional Networks. Association for Computing Machinery, New York, NY, USA, 540–547. https://doi.org/10.1145/3372278.3390713
  100. N24News: A New Dataset for Multimodal News Classification. arXiv:2108.13327 [cs.CL]
  101. Detecting Medical Misinformation on Social Media Using Multimodal Deep Learning. IEEE Journal of Biomedical and Health Informatics 25, 6 (2021), 2193–2203. https://doi.org/10.1109/JBHI.2020.3037027
  102. Gleaning wisdom from the past: Early detection of emerging rumors in social media. In Proceedings of the 2017 SIAM International Conference on Data Mining. SIAM, 99–107.
  103. Liang Wu and Huan Liu. 2018. Tracing Fake-News Footprints: Characterizing Social Media Messages by How They Propagate. Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (2018).
  104. Detecting fake news by exploring the consistency of multimodal data. Inf. Process. Manag. 58 (2021), 102610.
  105. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2 (2014), 67–78. https://doi.org/10.1162/tacl˙a˙00166
  106. Florence: A New Foundation Model for Computer Vision. ArXiv abs/2111.11432 (2021). https://api.semanticscholar.org/CorpusID:244477674
  107. MetaAdapt: Domain Adaptive Few-Shot Misinformation Detection via Meta Learning. arXiv preprint arXiv:2305.12692 (2023).
  108. Fake news detection for epidemic emergencies via deep correlations between text and images. Sustainable Cities and Society 66 (12 2020), 102652. https://doi.org/10.1016/j.scs.2020.102652
  109. MDMN: Multi-task and Domain Adaptation based Multi-modal Network for early rumor detection. Expert Systems with Applications 195 (2022), 116517. https://doi.org/10.1016/j.eswa.2022.116517
  110. ReCOVery: A Multimodal Repository for COVID-19 News Credibility Research (CIKM ’20). Association for Computing Machinery, New York, NY, USA, 3205–3212. https://doi.org/10.1145/3340531.3412880
  111. SAFE: Similarity-Aware Multi-Modal Fake News Detection. ArXiv abs/2003.04981 (2020).
  112. Xinyi Zhou and Reza Zafarani. 2019. Network-Based Fake News Detection: A Pattern-Driven Approach. SIGKDD Explor. Newsl. 21, 2 (nov 2019), 48–60. https://doi.org/10.1145/3373464.3373473
  113. Zhi-Hua Zhou. 2017. A Brief Introduction to Weakly Supervised Learning. National Science Review 5 (08 2017). https://doi.org/10.1093/nsr/nwx106
  114. Continually Detection, Rapidly React: Unseen Rumors Detection Based on Continual Prompt-Tuning. In Proceedings of the 29th International Conference on Computational Linguistics. 3029–3041.
Citations (13)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now