Gödel Test: Can Large Language Models Solve Easy Conjectures? (2509.18383v1)

Abstract: Recent announcements from frontier AI model labs have highlighted strong results on high-school and undergraduate math competitions. Yet it remains unclear whether LLMs can solve new, simple conjectures in more advanced areas of mathematics. We propose the G\"odel Test: evaluating whether a model can produce correct proofs for very simple, previously unsolved conjectures. To this end, we study the performance of GPT-5 on five conjectures in combinatorial optimization. For each problem, we provided one or two source papers from which the conjecture arose, withheld our own conjecture, and then assessed the model's reasoning in detail. On the three easier problems, GPT-5 produced nearly correct solutions; for Problem 2 it even derived a different approximation guarantee that, upon checking, refuted our conjecture while providing a valid solution. The model failed on Problem 4, which required combining results from two papers. On Problem 5, a harder case without a validated conjecture, GPT-5 proposed the same algorithm we had in mind but failed in the analysis, suggesting the proof is more challenging than expected. Although our sample is small, the results point to meaningful progress on routine reasoning, occasional flashes of originality, and clear limitations when cross-paper synthesis is required. GPT-5 may represent an early step toward frontier models eventually passing the G\"odel Test.

• The paper introduces the Gödel Test as a novel benchmark for evaluating LLMs’ mathematical reasoning on unsolved yet elementary conjectures.
• The approach leverages techniques like the Measured Greedy Frank-Wolfe algorithm and continuous greedy methods to adapt proofs and derive approximation guarantees.
• The study reveals that while LLMs can handle routine reasoning in simple cases, they struggle with integrating and synthesizing proofs across complex scenarios.

Gödel Test: Evaluating LLMs on Novel, Simple Mathematical Conjectures

Motivation and Problem Formulation

The paper introduces the Gödel Test as a benchmark for assessing the mathematical reasoning capabilities of LLMs, specifically GPT-5, on new, simple conjectures in advanced mathematical domains. Unlike prior evaluations focused on high-school or undergraduate competition problems, the Gödel Test targets conjectures that are straightforward for trained mathematicians but not directly available in existing literature. The test is designed to probe whether LLMs can synthesize knowledge and produce correct proofs for previously unsolved, yet elementary, problems—an essential step toward genuine mathematical maturity in AI systems.

The authors select five conjectures from combinatorial optimization, particularly submodular maximization, ensuring that each is accessible to a competent graduate student but novel enough to avoid direct memorization or retrieval. For each problem, GPT-5 is provided with minimal context and one or two relevant source papers, but no hints or guidance regarding solution strategies.

Experimental Design and Conjecture Selection

The five conjectures span a range of submodular maximization settings:

1. Maximizing the sum of monotone and non-monotone DR-submodular functions over a down-closed convex set.
2. Bicriteria maximization of a monotone submodular function under a $p$-system constraint.
3. Maximizing a monotone, $\gamma$-weakly DR-submodular function over a convex set.
4. Maximizing a partially monotone, weakly submodular function under a cardinality constraint.
5. Maximizing a monotone, weakly submodular function under two matroid intersection constraints.

Each conjecture is accompanied by a prompt and relevant literature, but the actual conjecture is withheld to ensure the model must reason and synthesize rather than retrieve.

Model Performance and Analysis

Routine Reasoning and Proof Adaptation

On the three easier problems, GPT-5 demonstrates near-correct reasoning and proof construction. For instance, in Problem 1, the model adapts the Measured Greedy Frank-Wolfe algorithm and provides a split guarantee for the sum of monotone and non-monotone DR-submodular functions, achieving the expected $(1-1/e)$ and $1/e$ approximation factors. The proof is mathematically sound, though GPT-5 exhibits a tendency to closely follow the structure of the reference paper, omitting unchanged steps and generalizing beyond necessity. This behavior mirrors human tendencies to minimize redundant work but can obscure clarity and rigor.

In Problem 2, GPT-5 not only adapts the bicriteria framework for $p$-systems but also refutes the original conjecture by deriving a different approximation guarantee, demonstrating a degree of originality. The model correctly identifies the scaling of the infeasibility ratio with $p$ and provides a tight analysis, albeit with minor technical inaccuracies.

For Problem 3, the model successfully generalizes the continuous greedy/Frank-Wolfe method to $\gamma$-weakly DR-submodular functions, yielding a $(1-e^{-\gamma})$ approximation. The proof is detailed and self-contained, though it unnecessarily restricts the feasible set to down-closed convex sets and occasionally misinterprets terminology from the reference literature.

Limitations in Cross-Paper Synthesis and Complex Analysis

Problems 4 and 5 require the integration of techniques from multiple papers and the synthesis of disparate proof strategies. Here, GPT-5 fails to produce correct or meaningful results. In Problem 4, the model reverts to stating known results without leveraging the additional structure provided by partial monotonicity, and its attempt at a smooth guarantee in $m$ and $\gamma$ is mathematically incorrect, with several unjustified steps and misapplied inequalities.

In Problem 5, GPT-5 proposes the correct algorithm for matroid intersection but fails in the analysis, miscounting the effect of removing elements and misapplying exchange properties. The errors are fundamental, indicating a lack of deep combinatorial understanding and an inability to generalize proof techniques beyond the immediate scope of the reference papers.

Evaluation of Proof Quality

Across all problems, GPT-5's proofs are often plausible and superficially convincing, but detailed examination reveals deep flaws in the more complex cases. The model's adaptation of known proofs is adequate for routine reasoning but lacks the flexibility and creativity required for integrative or novel arguments. The tendency to closely mirror reference structures, even when suboptimal, suggests a reliance on pattern matching rather than genuine mathematical insight.

Prompting Effects

The authors note that prompting strategies significantly affect model performance. Requests for full, detailed proofs elicit more complete and self-contained solutions, while minimal prompts lead to omitted steps and superficial reasoning. This sensitivity underscores the importance of prompt engineering in eliciting high-quality mathematical output from LLMs.

Implications and Future Directions

Practical Implications

The results indicate that GPT-5 has reached a level of mathematical competence comparable to a mediocre but not incompetent graduate student, capable of routine reasoning and occasional originality within specialized domains. However, the model remains limited in its ability to synthesize across papers and develop integrative proof strategies. The superficial plausibility of incorrect proofs highlights a potential risk in deploying LLMs for mathematical research without rigorous human verification.

Theoretical Implications

The Gödel Test provides a framework for evaluating genuine mathematical reasoning in AI, moving beyond rote problem-solving to the synthesis and proof of novel conjectures. The observed limitations suggest that current LLMs lack the deep combinatorial and integrative reasoning required for advanced mathematical maturity. Progress in this direction will likely require architectural innovations, improved training regimes, and integration with external tools such as computer algebra systems and proof assistants.

Speculation on Future Developments

The authors are cautiously optimistic that future model generations, possibly augmented with interactive prompting and external reasoning tools, may acquire the ability to systematically connect proof techniques and synthesize novel arguments. The Gödel Test may serve as a benchmark for tracking such progress and guiding the development of AI systems capable of genuine mathematical discovery.

Conclusion

The Gödel Test reveals meaningful progress in the mathematical reasoning abilities of frontier LLMs, with GPT-5 demonstrating competence on routine problems and occasional flashes of originality. However, significant limitations remain in cross-paper synthesis and complex proof analysis. The results underscore the need for rigorous evaluation, careful prompt design, and continued research into the development of AI systems capable of passing the Gödel Test and contributing to mathematical discovery.