CLEAR-3K: Assessing Causal Explanatory Capabilities in Language Models (2506.17180v1)

Abstract: We introduce CLEAR-3K, a dataset of 3,000 assertion-reasoning questions designed to evaluate whether LLMs can determine if one statement causally explains another. Each question present an assertion-reason pair and challenge LLMs to distinguish between semantic relatedness and genuine causal explanatory relationships. Through comprehensive evaluation of 21 state-of-the-art LLMs (ranging from 0.5B to 72B parameters), we identify two fundamental findings. First, LLMs frequently confuse semantic similarity with causality, relying on lexical and semantic overlap instead of inferring actual causal explanatory relationships. Second, as parameter size increases, models tend to shift from being overly skeptical about causal relationships to being excessively permissive in accepting them. Despite this shift, performance measured by the Matthews Correlation Coefficient plateaus at just 0.55, even for the best-performing models.Hence, CLEAR-3K provides a crucial benchmark for developing and evaluating genuine causal reasoning in LLMs, which is an essential capability for applications that require accurate assessment of causal relationships.