Evaluation of LLMs in Generalizing Analogy Solving Across Domains
The paper "Can LLMs generalize analogy solving like children can?" by Stevenson et al. investigates the comparative abilities of humans and LLMs in solving analogies and transferring this skill to unfamiliar domains. The central question posed by the authors is whether LLMs can emulate the human child's capacity to generalize analogy-solving methods across different contexts, such as the transition from familiar alphabets to unfamiliar symbol sets.
The paper assesses mechanisms of analogical reasoning, a cornerstone of cognitive development and learning in humans. By transferring abstract reasoning from known contexts, humans can adeptly handle novel domain analogies, as seen in problems like mapping "body : feet" to "table : legs." Numerous studies support the idea that humans can make such contextual shifts even at a young age. Thus, the paper hypothesizes whether LLMs, which have shown proficiency in handling analogy problems in certain contexts, possess the same level of adaptive generalization.
The researchers devised a task engaging children, adults, and various LLMs, including models from Anthropic, Google's Gemma, OpenAI's GPT, and Meta's Llama. The participants were tasked with solving letter-string analogies across familiar (Latin alphabet), near-transfer (Greek alphabet), and far-transfer (symbol set) domains. The findings confirm that human participants, including children, readily generalized across domains. In contrast, LLMs displayed robust performance only in the well-known Latin alphabet domain. Their capability to generalize to the Greek alphabet was weaker, with a significant decline in performance observed in symbol-based analogies, marking an apparent limitation in these models' analogical reasoning.
The LLMs' reduced performance in unfamiliar domains pinpoints a lack of flexible abstraction, a quality innate to human intelligence and critical for achieving robust generalization. This inflexibility raises concerns about the current models' assumed capabilities for understanding and applying conceptual rules beyond familiar contexts. The authors further investigated the errors made by LLMs, revealing that these models have a predisposition to literal rule application and exhibit difficulties processing predecessor and two-place successor operations, which likely impairs their analogy-solving ability in novel realms.
The paper's major implication lies in its challenge to the notion of LLMs approximating human-like reasoning, particularly in the domain of transfer learning. This paper contributes to the ongoing discourse on the cognitive capabilities of LLMs and suggests revisiting the underlying architectures and training paradigms if the broader generalization is desired. The findings emphasize the necessity for models capable of abstracting and transferring relational concepts across domains that feature differing surface-level similarities—a haLLMark of genuine analogical reasoning.
Future research may explore how training regimes might be modified to foster such robust abstraction in LLMs. Incorporating multisensory or symbolic interaction data could enhance the models' adaptability and generalization capabilities. Additionally, examining whether alternative model designs, which focus on explicit reasoning mechanisms, can overcome the observed limitations, remains an essential avenue of inquiry. This paper thus provides valuable insights and prompts further investigation into machine learning and artificial general intelligence, encouraging innovations that drive LLMs closer towards human-like intelligence.