2000 character limit reached
TT-BLIP: Enhancing Fake News Detection Using BLIP and Tri-Transformer (2403.12481v1)
Published 19 Mar 2024 in cs.LG and cs.CV
Abstract: Detecting fake news has received a lot of attention. Many previous methods concatenate independently encoded unimodal data, ignoring the benefits of integrated multimodal information. Also, the absence of specialized feature extraction for text and images further limits these methods. This paper introduces an end-to-end model called TT-BLIP that applies the bootstrapping language-image pretraining for unified vision-language understanding and generation (BLIP) for three types of information: BERT and BLIP\textsubscript{Txt} for text, ResNet and BLIP\textsubscript{Img} for images, and bidirectional BLIP encoders for multimodal information. The Multimodal Tri-Transformer fuses tri-modal features using three types of multi-head attention mechanisms, ensuring integrated modalities for enhanced representations and improved multimodal data analysis. The experiments are performed using two fake news datasets, Weibo and Gossipcop. The results indicate TT-BLIP outperforms the state-of-the-art models.
- “Combating multimodal fake news on social media: methods, datasets, and future perspective,” Multimedia systems, vol. 28, no. 6, pp. 2391–2422, 2022.
- “Novel visual and statistical image features for microblogs news verification,” IEEE transactions on multimedia, vol. 19, no. 3, pp. 598–608, 2016.
- “Information credibility on twitter,” in Proceedings of the 20th international conference on World wide web, 2011, pp. 675–684.
- “Automatic deception detection: Methods for finding fake news,” Proceedings of the association for information science and technology, vol. 52, no. 1, pp. 1–4, 2015.
- “Detecting rumors from microblogs with recurrent neural networks,” 2016.
- “Call attention to rumors: Deep attention based recurrent neural networks for early rumor detection,” in Trends and Applications in Knowledge Discovery and Data Mining: PAKDD 2018 Workshops, BDASC, BDM, ML4Cyber, PAISI, DaMEMO, Melbourne, VIC, Australia, June 3, 2018, Revised Selected Papers 22. Springer, 2018, pp. 40–52.
- “Eann: Event adversarial neural networks for multi-modal fake news detection,” in Proceedings of the 24th acm sigkdd international conference on knowledge discovery & data mining, 2018, pp. 849–857.
- “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in International Conference on Machine Learning. PMLR, 2022, pp. 12888–12900.
- “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
- “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
- “Multimodal fusion with recurrent neural networks for rumor detection on microblogs,” in Proceedings of the 25th ACM international conference on Multimedia, 2017, pp. 795–816.
- “Fakenewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media,” Big data, vol. 8, no. 3, pp. 171–188, 2020.
- “Mvae: Multimodal variational autoencoder for fake news detection,” in The world wide web conference, 2019, pp. 2915–2921.
- “Spotfake: A multi-modal framework for fake news detection,” in 2019 IEEE fifth international conference on multimedia big data (BigMM). IEEE, 2019, pp. 39–47.
- “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
- “Spotfake+: A multimodal framework for fake news detection via transfer learning (student abstract),” in Proceedings of the AAAI conference on artificial intelligence, 2020, vol. 34, pp. 13915–13916.
- “Safe: similarity-aware multi-modal fake news detection (2020),” Preprint. arXiv, vol. 200304981, pp. 2, 2020.
- “Cross-modal ambiguity learning for multimodal fake news detection,” in Proceedings of the ACM Web Conference 2022, 2022, pp. 2897–2905.
- “Leveraging intra and inter modality relationship for multimodal fake news detection,” in Companion Proceedings of the Web Conference 2022, 2022, pp. 726–734.
- “Multimodal fusion with co-attention networks for fake news detection,” in Findings of the association for computational linguistics: ACL-IJCNLP 2021, 2021, pp. 2560–2569.
- “Like article, like audience: Enforcing multimodal correlations for disinformation detection,” arXiv preprint arXiv:2108.13892, 2021.
- “Bdann: Bert-based domain adaptation neural network for multi-modal fake news detection,” in 2020 international joint conference on neural networks (IJCNN). IEEE, 2020, pp. 1–8.
- “Multimodal fake news detection via clip-guided learning,” in 2023 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2023, pp. 2825–2830.
- “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PMLR, 2021, pp. 8748–8763.
- “Supervised multimodal bitransformers for classifying images and text,” arXiv preprint arXiv:1909.02950, 2019.
- “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
- “Detecting fake news articles,” in 2019 IEEE International Conference on Big Data (Big Data). IEEE, 2019, pp. 3021–3025.
- “Xlnet: Generalized autoregressive pretraining for language understanding,” Advances in neural information processing systems, vol. 32, 2019.
- “Early versus late fusion in semantic video analysis,” in Proceedings of the 13th annual ACM international conference on Multimedia, 2005, pp. 399–402.
- “Multi-level fusion of audio and visual features for speaker identification,” in Advances in Biometrics: International Conference, ICB 2006, Hong Kong, China, January 5-7, 2006. Proceedings. Springer, 2005, pp. 493–499.
- “Tensor fusion network for multimodal sentiment analysis,” arXiv preprint arXiv:1707.07250, 2017.
- Eunjee Choi (3 papers)
- Jong-Kook Kim (14 papers)