2000 character limit reached
Image Matters: A New Dataset and Empirical Study for Multimodal Hyperbole Detection (2307.00209v3)
Published 1 Jul 2023 in cs.CV, cs.AI, and cs.CL
Abstract: Hyperbole, or exaggeration, is a common linguistic phenomenon. The detection of hyperbole is an important part of understanding human expression. There have been several studies on hyperbole detection, but most of which focus on text modality only. However, with the development of social media, people can create hyperbolic expressions with various modalities, including text, images, videos, etc. In this paper, we focus on multimodal hyperbole detection. We create a multimodal detection dataset from Weibo (a Chinese social media) and carry out some studies on it. We treat the text and image from a piece of weibo as two modalities and explore the role of text and image for hyperbole detection. Different pre-trained multimodal encoders are also evaluated on this downstream task to show their performance. Besides, since this dataset is constructed from five different topics, we also evaluate the cross-domain performance of different models. These studies can serve as a benchmark and point out the direction of further study on multimodal hyperbole detection.
- Harnessing privileged information for hyperbole detection. In Proceedings of the The 19th Annual Workshop of the Australasian Language Technology Association, pages 58–67, Online. Australasian Language Technology Association.
- Multi-modal sarcasm detection in twitter with hierarchical fusion model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, page 2506–2515. Association for Computational Linguistics.
- Towards multimodal sarcasm detection (an _obviously_ perfect paper). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Florence, Italy. Association for Computational Linguistics.
- Revisiting pre-trained models for Chinese natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 657–668, Online. Association for Computational Linguistics.
- Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Human Language Technology: Conference of the North American Chapter of the Association of Computational Linguistics, page 4171–4186.
- Gated attention fusion network for multimodal sentiment classification. Knowledge-Based Systems, 240.
- Gaëlle Ferré. 2014. Multimodal hyperbole. Multimodal Communication, 3(1):25–50.
- Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738.
- Identity mappings in deep residual networks. In European conference on computer vision, pages 630–645. Springer.
- Wenlan: Bridging vision and language by large-scale multi-modal pre-training. CoRR, abs/2103.06561.
- Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
- Identifying exaggerated language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, page 7024–7034. Association for Computational Linguistics.
- Figurative language occurrence and co-occurrence in contemporary literature. In Empirical Approaches to Literature and Aesthetics, pages 83–87, Norwood, NJ. Ablex Publishing Corporation.
- Multi-modal sarcasm detection with interactive in-modal and cross-modal graphs. In Proceedings of the 29th ACM International Conference on Multimedia, MM ’21, page 4707–4715, New York, NY, USA. Association for Computing Machinery.
- Towards multi-modal sarcasm detection via hierarchical congruity modeling with knowledge enhancement. arXiv preprint arXiv:2210.03501.
- Laura Cano Mora. 2009. All or nothing: A semantic analysis of hyperbole. Journal of Language and Applied Language, 4:25–35.
- Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763. PMLR.
- A computational exploration of exaggeration. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, page 3296–3304. Association for Computational Linguistics.
- Transmodality: An end2end fusion method with transformer for multimodal sentiment analysis. In Proceedings of The Web Conference 2020, pages 2514–2520.
- Modeling incongruity between modalities for multimodal sarcasm detection. IEEE MultiMedia, 28(2):86–95.
- Reasoning with multimodal sarcastic tweets via modeling cross-modality contrast and semantic association. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3777–3786, Online. Association for Computational Linguistics.
- Multi-level attention map network for multimodal sentiment analysis. IEEE Transactions on Knowledge and Data Engineering.
- Chinese clip: Contrastive vision-language pretraining in chinese. arXiv preprint arXiv:2211.01335.
- Multimodal sentiment detection based on multi-channel graph neural networks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 328–339. Association for Computational Linguistics.
- Yiming Yang and Jan O Pedersen. 1997. A comparative study on feature selection in text categorization. In Icml, volume 97, page 35. Citeseer.