Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Finding Challenging Metaphors that Confuse Pretrained Language Models (2401.16012v1)

Published 29 Jan 2024 in cs.CL

Abstract: Metaphors are considered to pose challenges for a wide spectrum of NLP tasks. This gives rise to the area of computational metaphor processing. However, it remains unclear what types of metaphors challenge current state-of-the-art models. In this paper, we test various NLP models on the VUA metaphor dataset and quantify to what extent metaphors affect models' performance on various downstream tasks. Analysis reveals that VUA includes a large number of metaphors that pose little difficulty to downstream tasks. We would like to shift the attention of researchers away from these metaphors to instead focus on challenging metaphors. To identify hard metaphors, we propose an automatic pipeline that identifies metaphors that challenge a particular model. Our analysis demonstrates that our detected hard metaphors contrast significantly with VUA and reduce the accuracy of machine translation by 16\%, QA performance by 4\%, NLI by 7\%, and metaphor identification recall by over 14\% for various popular NLP systems.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.12712.
  2. Generating similes effortlessly like a pro: A style transfer approach for simile generation. arXiv preprint arXiv:2009.08942.
  3. Flute: Figurative language understanding through textual explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7139–7159.
  4. Jonathan Charteris-Black. 2000. Metaphor and vocabulary teaching in esp economics. English for specific purposes, 19(2):149–165.
  5. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR.
  6. On sampling strategies for neural network-based collaborative filtering. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 767–776.
  7. MelBERT: Metaphor detection via contextualized late interaction using metaphorical identification theories. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1763–1773.
  8. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936.
  9. Weeding out conventionalized metaphors: A corpus of novel metaphor annotations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1412–1424, Brussels, Belgium. Association for Computational Linguistics.
  10. Bill Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005).
  11. Simcse: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910.
  12. Pragglejaz Group. 2007. MIP: A method for identifying metaphorically used words in discourse. Metaphor and symbol, 22(1):1–39.
  13. Exploiting a semantic annotation tool for metaphor analysis. In Corpus Linguistics Conference (CL2007).
  14. Adam Kilgarriff. 1997. I don’t believe in word senses. Computers and the Humanities, 31(2):91–113.
  15. Beata Beigman Klebanov and Michael Flor. 2013. Argumentation-relevant metaphors in test-taker essays. In Proceedings of the First Workshop on Metaphor in NLP, pages 11–20.
  16. Veronika Koller. 2004. Metaphor and gender in business media discourse: A critical cognitive study. Springer.
  17. Saisuresh Krishnakumaran and Xiaojin Zhu. 2007. Hunting elusive metaphors using lexical resources. In Proceedings of the Workshop on Computational approaches to Figurative Language, pages 13–20.
  18. DGST: a dual-generator network for text style transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7131–7136.
  19. The secret of metaphor on expressing stronger emotion. arXiv preprint arXiv:2301.13042.
  20. Cm-gen: A neural framework for chinese metaphor generation with explicit context modelling. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6468–6479.
  21. Nominal metaphor generation with multitask learning. In Proceedings of the 15th International Conference on Natural Language Generation, pages 225–235.
  22. Metaphor detection via explicit basic meanings modelling. arXiv preprint arXiv:2305.17268.
  23. Metaphor detection via explicit basic meanings modelling. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 91–100, Toronto, Canada. Association for Computational Linguistics.
  24. Framebert: Conceptual metaphor detection with frame embedding learning. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1550–1555.
  25. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
  26. Birte Lönneker. 2004. Lexical databases as resources for linguistic creativity: Focus on metaphor. In Proceedings of the LREC 2004 workshop on language resources for linguistic creativity, pages 9–16. Citeseer.
  27. Word embedding and WordNet based metaphor identification and interpretation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1222–1231.
  28. End-to-end sequential metaphor identification inspired by linguistic theories. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3888–3898.
  29. Using a semantic concordance for sense identification. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994.
  30. Metaphor as a medium for emotion: An empirical study. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics, pages 23–33.
  31. Roberto Navigli. 2018. Natural language understanding: instructions for (present and future) use. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 5697–5702.
  32. An analysis of language models for metaphor recognition. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3722–3736, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  33. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67.
  34. Sunny Rai and Shampa Chakraverty. 2020. A survey on computational metaphor processing. ACM Computing Surveys (CSUR), 53(2):1–37.
  35. A primer in bertology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842–866.
  36. Metaphor identification using verb and noun clustering. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 1002–1010.
  37. Gerard Steen. 2010. A method for linguistic metaphor identification: From MIP to MIPVU, volume 14. John Benjamins Publishing.
  38. IMPLI: Investigating NLI models’ performance on figurative language. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5375–5388, Dublin, Ireland. Association for Computational Linguistics.
  39. Latent semantic similarity based interpretation of chinese metaphors. Engineering Applications of Artificial Intelligence, 48:188–203.
  40. Hanna Skorczynska Sznajder and Jordi Pique-Angordans. 2005. A corpus based description of metaphorical marking patterns in scientific and popular business discourse. English for Specific Purposes, 22:172–182.
  41. Recent advances in neural metaphor processing: A linguistic, cognitive and social perspective. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4673–4686.
  42. Tony Veale. 2011. Creative language retrieval: A robust hybrid of information retrieval and linguistic creativity. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 278–287.
  43. Tony Veale and Yanfen Hao. 2008. A fluid knowledge representation for understanding and generating creative metaphors. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 945–952.
  44. Ufsac: Unification of sense annotated corpora and tools. In Language Resources and Evaluation Conference (LREC).
  45. Metaphor detection with effective context denoising. arXiv preprint arXiv:2302.05611.
  46. Does BERT make any sense? interpretable word sense disambiguation with contextualized embeddings. arXiv preprint arXiv:1909.10430.
  47. Jin Zhou. 2020. Love is as complex as math: Metaphor generation system for social chatbot. In Chinese Lexical Semantics: 20th Workshop, CLSW 2019, Beijing, China, June 28–30, 2019, Revised Selected Papers, volume 11831, page 337. Springer Nature.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Yucheng Li (31 papers)
  2. Frank Guerin (30 papers)
  3. Chenghua Lin (127 papers)
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now