Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

ARN: Analogical Reasoning on Narratives (2310.00996v4)

Published 2 Oct 2023 in cs.CL

Abstract: As a core cognitive skill that enables the transferability of information across domains, analogical reasoning has been extensively studied for both humans and computational models. However, while cognitive theories of analogy often focus on narratives and study the distinction between surface, relational, and system similarities, existing work in natural language processing has a narrower focus as far as relational analogies between word pairs. This gap brings a natural question: can state-of-the-art LLMs detect system analogies between narratives? To gain insight into this question and extend word-based relational analogies to relational system analogies, we devise a comprehensive computational framework that operationalizes dominant theories of analogy, using narrative elements to create surface and system mappings. Leveraging the interplay between these mappings, we create a binary task and benchmark for Analogical Reasoning on Narratives (ARN), covering four categories of far (cross-domain)/near (within-domain) analogies and disanalogies. We show that while all LLMs can largely recognize near analogies, even the largest ones struggle with far analogies in a zero-shot setting, with GPT4.0 scoring below random. Guiding the models through solved examples and chain-of-thought reasoning enhances their analogical reasoning ability. Yet, since even in the few-shot setting, the best model only performs halfway between random and humans, ARN opens exciting directions for computational analogical reasoners.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (70)
  1. Alexandra Alexieva and Penka Hristova. 2017. Processing differences between near and far analogies.
  2. Enhanced story comprehension for large language models through dynamic document-based knowledge graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10436–10444.
  3. Mieke Bal. 2017. Narratology: Introduction to the Theory of Narrative, 4th edition. University of Toronto Press.
  4. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  5. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416.
  6. On reality and the limits of language data. arXiv preprint arXiv:2208.11981.
  7. Scientific and creative analogies in pretrained language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2094–2100, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  8. Kevin N Dunbar and David Klahr. 2012. Scientific thinking and reasoning. In Keith J Holyoak and Robert G Morrison, editors, Oxford handbook of thinking and reasoning, page 701–718.
  9. Michael C Frank. 2023. Baby steps in evaluating the capacities of large language models. Nature Reviews Psychology, 2(8):451–452.
  10. Wordrep: A benchmark for research on learning word representations. arXiv preprint arXiv:1407.1640.
  11. Janet Gardner. 2003. Writing about Literature. Bedford Books.
  12. Analogical reasoning, 2012. Encyclopedia of Human Behavior, 2nd ed., VS Ramachandran, ed., Elsevier, Oxford, UK, pages 130–136.
  13. Dedre Gentner. 1982. Are scientific analogies metaphors. Metaphor: Problems and perspectives, 7(7).
  14. Dedre Gentner. 1983. Structure-mapping: A theoretical framework for analogy. Cognitive science, 7(2):155–170.
  15. Dedre Gentner and Mary Jo Rattermann. 1991. Language and the career of similarity. University of Illinois at Urbana-Champaign, Center for the Study of Reading.
  16. The roles of similarity in transfer: Separating retrievability from inferential soundness. Cognitive psychology, 25(4):524–575.
  17. Dedre Gentner and Cecile Toupin. 1986. Systematicity and surface similarity in the development of analogy. Cognitive Science, 10(3):277–300.
  18. Sayan Ghosh and Shashank Srivastava. 2021. epic: Employing proverbs in context as a benchmark for abstract language understanding. arXiv preprint arXiv:2109.06838.
  19. Mary L Gick and Keith J Holyoak. 1980. Analogical problem solving. Cognitive psychology, 12(3):306–355.
  20. Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t. In Proceedings of the NAACL Student Research Workshop, pages 8–15.
  21. Automatic activation of categorical and abstract analogical relations in analogical reasoning. Memory & cognition, 34:1414–1421.
  22. The micro-category account of analogy. Cognition, 106(2):1004–1016.
  23. Connecting long distance: semantic distance in analogical reasoning modulates frontopolar cortex activity. Cerebral cortex, 20(1):70–76.
  24. Graeme S Halford. 1992. Analogical reasoning and conceptual complexity in cognitive development. Human Development, 35(4):193–217.
  25. Mary Hesse. 1965. Models and analogies in science.
  26. Douglas R Hofstadter. 2001. Analogy as the core of cognition. The analogical mind: Perspectives from cognitive science, pages 499–538.
  27. K. J. Holyoak. 2012. Analogy and relational reasoning. In K. J. Holyoak and R. G. Morrison, editors, Oxford handbook of thinking and reasoning. Oxford University Press, New York.
  28. Keith J Holyoak and Paul Thagard. 1996. Mental leaps: Analogy in creative thought. MIT press.
  29. David Patrick Houghton. 1998. Analogical reasoning and policymaking: Where and when is it used? Policy Sciences, 31(3):151–176.
  30. Verbal analogy problem sets: An inventory of testing materials. Behavior research methods, 52:1803–1816.
  31. Molly E Ireland and James W Pennebaker. 2010. Language style matching in writing: synchrony in essays, correspondence, and poetry. Journal of personality and social psychology, 99(3):549.
  32. Storyanalogy: Deriving story-level analogies from large language models to unlock analogical understanding. arXiv preprint arXiv:2310.12874.
  33. SemEval-2012 task 2: Measuring degrees of relational similarity. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 356–364, Montréal, Canada. Association for Computational Linguistics.
  34. Unifiedqa: Crossing format boundaries with a single qa system. arXiv preprint arXiv:2005.00700.
  35. Vince Kotchian and Curt Simmons. 2012. SSAT &; Isee for dummies. John Wiley &; Sons, Inc.
  36. Martha Lewis and Melanie Mitchell. 2024. Using counterfactual tasks to evaluate the generality of analogical reasoning in large language models. arXiv preprint arXiv:2402.08955.
  37. Probabilistic analogical mapping with semantic relation networks. Psychological review.
  38. Inderjeet Mani. 2013. Computational modeling of narrative. Morgan & Claypool Publishers.
  39. Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276–282.
  40. Wolfgang Mieder. 1993. Proverbs are never out of season : popular wisdom in the modern age. Oxford University Press New York, New York.
  41. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
  42. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26.
  43. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746–751, Atlanta, Georgia. Association for Computational Linguistics.
  44. Melanie Mitchell. 2021. Abstraction and analogy-making in artificial intelligence. Annals of the New York Academy of Sciences, 1505(1):79–101.
  45. Comparing humans, gpt-4, and gpt-4v on abstraction and reasoning tasks. arXiv preprint arXiv:2311.09247.
  46. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839–849.
  47. Understanding narratives through dimensions of analogy.
  48. OpenAI. 2022. Chatgpt. https://openai.com/blog/chatgpt. Accessed: April 30, 2023.
  49. R OpenAI. 2023. Gpt-4 technical report. arXiv, pages 2303–08774.
  50. Darwin’s mistake: Explaining the discontinuity between human and nonhuman minds. Behavioral and brain sciences, 31(2):109–130.
  51. David Premack. 1983. The codes of man and beasts. Behavioral and Brain Sciences, 6(1):125–136.
  52. Language models are unsupervised multitask learners.
  53. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  54. Do large language models solve verbal analogies like children do? arXiv preprint arXiv:2310.20384.
  55. Tiered reasoning for intuitive physics: Toward verifiable commonsense language understanding. arXiv preprint arXiv:2109.04947.
  56. Oren Sultan and Dafna Shahaf. 2022. Life is a circus and we are the clowns: Automatically finding analogies between situations and processes. arXiv preprint arXiv:2210.12197.
  57. Oyvind Tafjord and Peter Clark. 2021. General-purpose question-answering with macaw. arXiv preprint arXiv:2109.02593.
  58. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  59. Combining independent modules to solve multiple-choice synonym and analogy problems. arXiv preprint cs/0309035.
  60. Varsity Tutors. Analogies - ssat elementary level verbal.
  61. BERT is to NLP what AlexNet is to CV: Can pre-trained language models identify analogies? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3609–3624, Online. Association for Computational Linguistics.
  62. Attention is all you need. Advances in neural information processing systems, 30.
  63. Far-out thinking: Generating solutions to distant analogies promotes relational thinking. Psychological science, 25(4):928–933.
  64. A narratology-based framework for storyline extraction. Computational Analysis of Storylines: Making Sense of Events, 125.
  65. Emergent analogical reasoning in large language models.
  66. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682.
  67. Chain-of-thought prompting elicits reasoning in large language models.
  68. Zero-shot information extraction via chatting with chatgpt.
  69. Below the surface: Analogical similarity and retrieval competition in reminding. Cognitive Psychology, 26(1):64–101.
  70. ANALOGICAL - a novel benchmark for long text analogy evaluation in large language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3534–3549, Toronto, Canada. Association for Computational Linguistics.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Zhivar Sourati (12 papers)
  2. Filip Ilievski (53 papers)
  3. Pia Sommerauer (5 papers)
  4. Yifan Jiang (79 papers)
Citations (1)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets