Can AI Refute Economic Theory? Evidence from Beyond the Knowledge Cutoff

Abstract: Can AI refute economic theory? I document experiments in which I asked several AI models (Gemini, Refine, Claude, and ChatGPT) to check the correctness of four published papers in economic theory, each containing an error that I helped identify or correct. ChatGPT Pro performed best, occasionally constructing counterexamples and corrected proofs, while other models fared worse. However, no model located a true error without substantial human guidance, and data contamination complicates interpretation. I argue that a competent human paired with a frontier model can outperform current peer review, but AI cannot yet refute economic theory on its own.

• The paper demonstrates that ChatGPT Pro excels in detecting logical gaps and constructing counterexamples to correct errors in established economic theories.
• It systematically evaluates LLM performance, highlighting strengths in mathematical reasoning and economic interpretation while addressing data contamination concerns.
• Implications suggest that integrating AI into peer review can enhance rigor in economic research, though fully autonomous error detection remains out of reach.

AI's Capabilities in Refuting Economic Theory: An Expert Evaluation

Overview and Motivation

The paper "Can AI Refute Economic Theory? Evidence from Beyond the Knowledge Cutoff" (2606.05383) investigates whether current frontier AI models, specifically LLMs such as ChatGPT, Claude, Gemini, and Refine, are capable of autonomously identifying and refuting errors in published economic theory. The author conducts a series of targeted experiments with four peer-reviewed economics papers, each containing a subtle mathematical or conceptual error previously identified and corrected by the author or colleagues. This work is motivated by the growing utilization of LLMs in mathematical domains and the open question of whether automated reasoning can rival or surpass human scrutiny in economics peer review.

Experimental Design and Model Assessment

The experiments systematically upload the full text of each paper to the tested LLMs and initially prompt them to assess the correctness of key results. The author then incrementally challenges the models, directing their attention to specific problematic parts. Crucially, to address data contamination concerns, the analysis for Tirole (1985) is performed with ChatGPT Pro's memory and web-search disabled shortly after the correction was published, minimizing the likelihood of retrieval-based responses.

Performance is evaluated based on:

• Precision and depth in critiquing proofs
• Ability to construct valid counterexamples
• Correctness and rigor of any corrected proofs generated
• Economic interpretative strength beyond formal logic

ChatGPT Pro emerges as the most reliable for mathematical reasoning, capable of identifying relevant flaws and providing constructive counterexamples with minimal prompting. Claude Opus demonstrates strengths in economic interpretation and judgment but is weaker in rigorous mathematical critique. Gemini exhibits significant deficiencies, including initial endorsement of incorrect arguments, plausible yet unfounded rationalizations, and occasional hallucinations. Refine, though promising in iterative chat mode, is resource-constrained and largely outperformed by ChatGPT Pro on both cost and efficacy.

Key Numerical Results and Claims

The strongest quantitative and qualitative outcomes demonstrated by ChatGPT Pro include:

• Immediate recognition of critical logical gaps (e.g., in Tirole 1985 and Kocherlakota 1992)
• Construction of rigorous counterexamples, often matching or exceeding published corrections
• Generation of elegant corrected proofs—sometimes more succinct than those in existing econometric literature
• Robust disambiguation between competing economic definitions (e.g., asset bubbles as per Santos-Woodford versus Miao-Wang terminology)

A bold claim in the paper is that a "competent economist working with a frontier model can already outperform the status quo of refereeing" with respect to technical logical scrutiny. However, no model located a true error autonomously without substantial directed human input.

Implications for Economic Peer Review and Model Limitations

The results highlight structural limitations in the peer review process:

• Peer review admits both Type I and Type II errors, often due to reviewers underchecking intricacies of proofs.
• The LLMs do not reliably uncover deep errors without a human first flagging the suspect region; they excel at validating or refuting targeted logical steps and at constructing counterexamples post hoc.
• Data contamination remains a systemic risk: evaluations may reflect models recalling corrections seen during training, rather than genuine reasoning.

Practically, this suggests that LLMs currently serve as potent tools for post-hoc verification, rapid algebraic checking, and candidate counterexample generation once a human points out the locus of possible error. The autonomous capability to refute complex theory is not yet realized. The paper advocates for integration of frontier LLMs into economic editorial workflows, arguing that prohibition lacks enforceability and that journal confidentiality protocols could be adapted (e.g., requiring arXiv preprints).

Theoretically, widespread adoption of AI-assisted verification could drive higher standards of rigor in economics, akin to formal proof verification in mathematics. However, risks include overreliance on AI-models that may embed systematic reasoning errors or contaminated benchmark data.

Prospects for Future AI Developments in Economics

The author speculates that continued advancement in LLMs, possibly through hybrid integration with formal proof assistants and domain-specific training, could narrow the gap between autonomous reasoning and human-guided error detection. Future models may achieve full independence in proof verification, provided they address both the contamination problem and the nuanced challenge of theory localization in complex macroeconomic arguments. Additionally, systemic changes to peer review—potentially involving real-time AI co-review or automated cross-model scrutiny—may become standard, fundamentally altering economics publication norms.

Conclusion

This study provides robust empirical evidence that LLMs, particularly ChatGPT Pro, are highly effective in targeted error identification, counterexample construction, and proof correction in economic theory, once guided by an informed human. The results decisively demonstrate the limitations of current models in discovering deep theoretical flaws autonomously. Incorporation of LLMs into peer review processes, under controlled protocols, promises substantial improvement over traditional human-only methods in mathematical economics. Nonetheless, realization of fully autonomous AI critique awaits further advances in reasoning, contamination mitigation, and economic domain adaptation.