Do human-level analogy models use human-like mechanisms?

Ascertain whether foundation models that solve analogical reasoning tasks at human-level performance (such as large language models evaluated on analogy benchmarks) possess reasoning mechanisms similar to those observed in human cognition, rather than alternative computational strategies that diverge from human reasoning.

Background

The paper raises concerns that strong task performance does not necessarily imply mechanistic similarity to humans. For analogical reasoning, recent work shows LLMs achieving human-level results, but it remains contested whether these models rely on cognitive-like processes or on different computational strategies that exploit training data statistics.

This debate is situated within the broader theme of distinguishing predictive success from explanatory insight. The authors point to evidence that performance often degrades outside familiar training regimes, motivating targeted evaluations to determine if the mechanisms align with human reasoning.