Potemkin Understanding in Large Language Models (2506.21521v2)

Abstract: LLMs are regularly evaluated using benchmark datasets. But what justifies making inferences about an LLM's capabilities based on its answers to a curated set of questions? This paper first introduces a formal framework to address this question. The key is to note that the benchmarks used to test LLMs -- such as AP exams -- are also those used to test people. However, this raises an implication: these benchmarks are only valid tests if LLMs misunderstand concepts in ways that mirror human misunderstandings. Otherwise, success on benchmarks only demonstrates potemkin understanding: the illusion of understanding driven by answers irreconcilable with how any human would interpret a concept. We present two procedures for quantifying the existence of potemkins: one using a specially designed benchmark in three domains, the other using a general procedure that provides a lower-bound on their prevalence. We find that potemkins are ubiquitous across models, tasks, and domains. We also find that these failures reflect not just incorrect understanding, but deeper internal incoherence in concept representations.

• The paper introduces 'potemkin understanding' to describe how LLMs can correctly answer keystone questions yet fail to apply concepts coherently.
• It employs dual evaluation methods—benchmark tests and automated self-consistency checks—to reveal high rates of conceptual inconsistency across models.
• The study challenges current benchmark validity and calls for more dynamic, human-aligned evaluation strategies for genuine LLM understanding.

Potemkin Understanding in LLMs: A Critical Examination of Benchmark Validity

The paper "Potemkin Understanding in LLMs" (2506.21521) presents a formal and empirical investigation into the limitations of current benchmark-based evaluations of LLMs. The authors introduce the concept of "potemkin understanding," a failure mode in which LLMs appear to demonstrate conceptual understanding on human-designed benchmarks, yet fail to exhibit coherent or human-like application of those concepts in practice. This work provides both a theoretical framework and empirical evidence that challenge the validity of using standard benchmarks as proxies for genuine conceptual understanding in LLMs.

Theoretical Framework: Keystone Sets and Potemkin Understanding

The authors formalize the notion of conceptual understanding by distinguishing between the structured, limited ways in which humans misunderstand concepts and the potentially unstructured, divergent ways in which LLMs can fail. They introduce the concept of a "keystone set": a minimal set of questions such that, for humans, answering all correctly implies true understanding of a concept. This property underpins the validity of standardized tests for human assessment.

However, the paper demonstrates that this assumption does not hold for LLMs. The set of possible LLM misunderstandings ($\mathcal{F}_l$) is not necessarily constrained in the same way as the set of human misunderstandings ($\mathcal{F}_h$). As a result, LLMs can answer all keystone questions correctly while still lacking a coherent or human-like grasp of the underlying concept. The authors define "potemkin understanding" as the phenomenon where an LLM passes keystone tests but fails on other tasks that require genuine conceptual application, in ways that are irreconcilable with any plausible human misunderstanding.

Empirical Methodology: Benchmark and Automated Procedures

To quantify the prevalence of potemkin understanding, the authors develop two complementary empirical procedures:

1. Benchmark-Based Evaluation: A new dataset spanning 32 concepts across literary techniques, game theory, and psychological biases is constructed. For each concept, models are evaluated on their ability to (a) define the concept (keystone), (b) classify instances, (c) generate constrained examples, and (d) edit examples to fit or not fit the concept. Only cases where the model provides a correct definition are considered for subsequent tasks, isolating the failure to apply concepts after demonstrating apparent understanding.
2. Automated Self-Consistency Evaluation: An automated procedure measures incoherence by prompting a model to generate an instance of a concept and then classify its own output. Disagreement between generation and classification indicates internal inconsistency. A further automated pipeline uses LLM self-judgment to provide a lower bound on the prevalence of potemkins, by checking whether the model's answers to new, related questions are judged as correct by itself.

Key Results

The empirical findings are robust and consistent across models, domains, and tasks:

• High Potemkin Rates:

Across seven state-of-the-art LLMs, the average potemkin rate—defined as the proportion of incorrect answers on application tasks, conditional on a correct definition—ranges from 0.36 to 0.66 depending on the task and model. The overall average is approximately 0.55 for classification, 0.40 for generation, and 0.40 for editing tasks. These rates are substantially above chance, even when models demonstrate near-perfect performance on definition tasks.

• Incoherence:

The automated self-consistency evaluation reveals incoherence scores ranging from 0.02 to 0.64 (where 0 is perfect consistency and 1 is random), indicating that models frequently fail to apply their own conceptual explanations in a consistent manner.

• Lower Bound on Potemkin Prevalence:

The automated procedure yields a lower-bound potemkin rate of 0.62, corroborating the benchmark-based findings and suggesting that the true prevalence may be even higher.

• Minimal Gains from Expanding Keystone Sets:

Simulations show that increasing the number of keystone questions (e.g., requiring correct application in addition to definition) yields only modest improvements in downstream application performance, indicating that the problem is not easily mitigated by more comprehensive testing.

Implications

The findings have significant implications for both the evaluation and development of LLMs:

• Benchmark Validity:

The paper makes the strong claim that standard benchmarks, when used as proxies for conceptual understanding, are invalid for LLMs unless LLM misunderstandings mirror those of humans. The high prevalence of potemkin understanding observed empirically invalidates this assumption for current models.

• Nature of LLM Failures:

The observed failures are not merely errors of factual recall or minor misapplication, but reflect deeper internal incoherence and non-human patterns of misunderstanding. This challenges the assumption that LLMs' high benchmark scores reflect robust, generalizable understanding.

• Evaluation Methodology:

The work motivates the need for new evaluation paradigms that go beyond static, human-designed benchmarks. Dynamic, interaction-based, or adversarial testing, as well as explicit tests for internal consistency and conceptual coherence, are necessary to more accurately assess LLM capabilities.

• Model Development:

The identification of potemkin understanding as a distinct failure mode suggests that future model training and architecture should explicitly target conceptual coherence and human-like generalization, rather than optimizing solely for benchmark performance.

Future Directions

The paper suggests several avenues for further research:

• Broader and More Diverse Benchmarks:

Expanding the range of concepts and keystone types could provide a more comprehensive assessment of potemkin understanding.

• Potemkin Detection and Mitigation:

Developing systematic methods for detecting and reducing potemkin rates during training and evaluation could improve the reliability of LLMs in real-world applications.

• Theoretical Characterization:

Further formalization of the space of LLM misunderstandings and their relationship to model architecture and training data may yield insights into the root causes of potemkin understanding.

Conclusion

"Potemkin Understanding in LLMs" provides a rigorous critique of current LLM evaluation practices, demonstrating that high benchmark performance does not guarantee genuine conceptual understanding. The introduction of the potemkin understanding framework, along with robust empirical evidence, compels a re-examination of how LLMs are assessed and trusted for real-world deployment. Addressing the challenges identified in this work will be essential for the development of LLMs that can be reliably aligned with human conceptual reasoning and application.