PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models (2502.01584v3)

Abstract: Existing benchmarks for frontier models often test specialized, "PhD-level" knowledge that is difficult for non-experts to grasp. In contrast, we present a benchmark with 594 problems based on the NPR Sunday Puzzle Challenge that requires only general knowledge. Our benchmark is challenging for both humans and models; however correct solutions are easy to verify, and models' mistakes are easy to spot. As LLMs are more widely deployed in society, we believe it is useful to develop benchmarks for frontier models that humans can understand without the need for deep domain expertise. Our work reveals capability gaps that are not evident in existing benchmarks: OpenAI o1 significantly outperforms other reasoning models on our benchmark, despite being on par with other models when tested on benchmarks that test specialized knowledge. Furthermore, our analysis of reasoning outputs uncovers new kinds of failures. DeepSeek R1, for instance, often concedes with "I give up" before providing an answer that it knows is wrong. R1 can also be remarkably "uncertain" in its output and in rare cases, it does not "finish thinking," which suggests the need for techniques to "wrap up" before the context window limit is reached. We also quantify the effectiveness of reasoning longer to identify the point beyond which more reasoning is unlikely to improve accuracy on our benchmark.

• The paper introduces a general knowledge reasoning benchmark using nearly 600 NPR puzzles to evaluate LLM performance in a zero-shot setting.
• The paper shows that OpenAI o1 achieves 59% accuracy, outperforming other models and highlighting specific failure modes such as premature 'give ups'.
• The paper highlights that extending reasoning tokens beyond a certain threshold yields diminishing returns, suggesting an optimal token budget for these tasks.

The paper introduces a benchmark derived from the NPR Sunday Puzzle Challenge to evaluate the reasoning capabilities of LLMs. The authors posit that existing benchmarks often rely on specialized knowledge, making it difficult for non-experts to understand and verify the solutions. In contrast, the proposed benchmark uses general knowledge-based puzzles that are challenging to solve but have easily verifiable solutions.

The authors curate a dataset of nearly 600 puzzles from the "off-air challenges" of the NPR Sunday Puzzle, ensuring each puzzle has a unique or a small set of valid answers. They systematically review and edit the scraped data to add missing context, remove explanations, and address alternative solutions. The puzzles are presented to the models in a zero-shot fashion, without any formatting instructions or additional guidance. The models' answers are evaluated by ignoring capitalization and punctuation and testing whether every phrase in the ground truth answer appears in the model-generated answer.

The performance of several state-of-the-art reasoning models, including OpenAI o1, o3-mini, o1-mini, Google Gemini 2.0 Flash Thinking Experimental 01-21, and DeepSeek R1, are benchmarked. GPT-4o and Claude Sonnet 3.5 are included as non-reasoning baselines. The results indicate that OpenAI o1 significantly outperforms the other models, achieving 59% accuracy. The next best performing model is o3-mini with high reasoning effort (47%), followed by o3-mini with default settings (36%) and R1 (35%). The non-reasoning models perform considerably worse than the best reasoning models, which suggests that the benchmark effectively tests reasoning ability.

The paper identifies failure modes specific to reasoning models. For instance, DeepSeek R1 frequently outputs "I give up" before providing an answer. Two types of "give ups" are observed:

• The model produces an "out-of-thin-air" final answer that does not appear in the reasoning output.
• The model deliberately violates constraints because it must provide an answer.

The paper also reports instances where DeepSeek R1 gets stuck during reasoning, failing to output the </think> token before reaching the output token limit. In some cases, R1 may "take back" answers, proposing several wrong ones, or find a good answer early but explore other options before committing to it.

An analysis of the reasoning output from R1 and Gemini Thinking suggests that increasing reasoning length beyond a certain point does not significantly improve accuracy. Gemini Thinking reaches an accuracy plateau at approximately 10,000 tokens, while R1's accuracy continues to improve and surpasses Gemini Thinking at around 3,000 tokens. The authors suggest that setting a token budget beyond which accuracy reaches a plateau can be helpful for these tasks.

The paper identifies that the models can "give up" on a difficult problem and deliberately return an incorrect answer. Examining the reasoning outputs of R1 and Gemini Thinking, the authors quantify the effectiveness of reasoning longer, allowing them to set a token budget beyond which accuracy reaches a plateau on these tasks.

The authors include several equations and variables in the paper including the following:

• \textipa{I}: Represents the IPA symbol for the unrounded vowel, also known as the near-open front unrounded vowel.
• \textipa{@}: Represents the IPA symbol for the mid-central vowel, also known as the schwa.

The paper references several other works, including:

• "OpenAI o1 System Card" [o1-system-card] which details the safety measures implemented before the release of OpenAI o1 and o1-mini.
• "s1: Simple test-time scaling" [muennighoff:s1] which introduces a test-time scaling approach to improve LLM performance using extra compute.
• "Humanity's Last Exam" [phan:humanitys-last-exam] which presents a benchmark designed to be extremely challenging, requiring deep domain expertise.
• "Gemini 2.0 Flash Thinking Experimental" [gemini2ft] which discusses Google's enhanced reasoning model that shows its thoughts to improve performance.
• "On the Measure of Intelligence" [chollet:arc-agi] which introduces the ARC-AGI reasoning and abstraction benchmark.
• "Solving and Generating NPR Sunday Puzzles with LLMs" [zhao:puzzleqa] which evaluates LLMs on puzzles from the NPR Sunday Puzzle game show.
• "BRAINTEASER: Lateral Thinking Puzzles for LLMs" [jiang:brainteaser] which devises a multiple-choice question answering task to test lateral thinking.
• "Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLP" [rozner:cryptic-crosswords] which presents a dataset of cryptic crossword clues for NLP systems.
• "GPQA: A Graduate-Level Google-Proof QA Benchmark" [gpqa] which is a dataset of multiple-choice questions written by domain experts.
• "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" [deepseek-r1] which introduces DeepSeek-R1-Zero and DeepSeek-R1 reasoning models.
• "NNsight and NDIF: Democratizing Access to Foundation Model Internals" [fiotto-kaufman:ndif] which introduces technologies to enable scientific paper of neural networks.
• "Substance Beats Style: Why Beginning Students Fail to Code with LLMs" [lucchetti:substance-beats-style] which explores why beginning students struggle to prompt LLMs for code generation.
• "Sky-T1: Train your own O1 preview model within \$450" [noauthor_sky-t1_nodate] which introduces Sky-T1-32B-Preview, a reasoning model that performs on par with o1-preview.
• "LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement" [lee_LLM2LLM_2024] which proposes an iterative data augmentation strategy to enhance LLM performance in low-data regimes.
• "Learning Transferable Visual Models From Natural Language Supervision" [radford_learning_2021] which demonstrates learning image representations from raw text.
• "Demystifying CLIP Data" [xu_demystifying_2024] which discusses CLIP data.
• "UltraFeedback: Boosting LLMs with Scaled AI Feedback" [cui_ultrafeedback_2024] which presents a large-scale AI feedback dataset for aligning LLMs.
• "Tulu 3: Pushing Frontiers in Open LLM Post-Training" [lambert_tulu_2024] which introduces a family of open-source post-trained models and training recipes.
• "Fine-Tuning LLMs from Human Preferences" [ziegler_fine-tuning_2020] which describes reward learning for LLMs using human feedback.
• "RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning" [gehring_rlef_2024] which proposes a reinforcement learning method for code synthesis models.
• "Q3 earnings call: CEO's remarks" [noauthor_q3_2024] which is a Google blog post discussing Q3 earnings.
• "Lingma SWE-GPT: An Open Development-Process-Centric LLM for Automated Software Improvement" [ma_lingma_2024] which introduces Lingma SWE-GPT, a LLM for software improvement.
• "Training Software Engineering Agents and Verifiers with SWE-Gym" [pan_training_2024] which presents SWE-Gym, an environment for training software engineering agents.
• "Evaluating and Aligning CodeLLMs on Human Preference" [yang_evaluating_2024] which presents a benchmark for evaluating code LLMs on human preference.
• "The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization" [huang_n_2024] which reproduces reinforcement learning from human feedback for summarization.
• "SemCoder: Training Code LLMs with Comprehensive Semantics Reasoning" [ding_semcoder_2024] which introduces SemCoder, a code LLM trained with semantics reasoning.
• "Bridge-Coder: Unlocking LLMs' Potential to Overcome Language Gaps in Low-Resource Code" [zhang_bridge-coder_2024] which introduces Bridge-Coder, an approach to enhance LLM performance on low-resource programming languages.
• "Facebook - log in or sign up" [noauthor_facebook_nodate] which is a link to the Facebook login page.
• "Tuning LLMs by Proxy" [liu_tuning_2024] which introduces proxy-tuning, a decoding-time algorithm for adapting LLMs.
• "Instructor-Written Hints as Automated Test Suite Quality Feedback" [perretta:mutant-hints] which discusses instructor-written hints as automated test suite quality feedback.
• "Fully Transparent Self-Alignment for Code Generation" [wei:starcoder2-self-instruct] which proposes SelfCodeAlign, a pipeline for self-aligning code LLMs.
• "Refusal in LLMs Is Mediated by a Single Direction" [arditi_refusal_2024] which shows that refusal in LLMs is mediated by a one-dimensional subspace.
• "ChatGPT Plugins" [openai:chatgpt-plugins] which discusses ChatGPT plugins.
• "SmoLLM - blazingly fast and remarkably powerful" [benallal:smoLLM] which discusses SmoLLM.
• "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" [lewis_retrieval-augmented_2020] which explores retrieval-augmented generation models.
• "The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers" [prather:widening] which explores the impact of generative AI tools on novice programmers.
• "CS1-LLM: Integrating LLMs into CS1 Instruction" [vadaparty:cs1-LLM] which discusses integrating LLMs into introductory computer science instruction.
• "From "Ban It Till We Understand It" to "Resistance is Futile": How University Programming Instructors Plan to Adapt as More Students Use AI Code Generation and Explanation Tools such as ChatGPT and GitHub Copilot" [lau:ban-it] which discusses how programming instructors plan to adapt to student use of AI code generation tools.
• "Copilot Workspace" [copilot-workspace] which introduces Copilot Workspace.
• "OpenDevin: An Open Platform for AI Software Developers as Generalist Agents" [wang:opendevin] which introduces OpenDevin, a platform for AI software developers.
• "SWE-bench: Can LLMs Resolve Real-world Github Issues?" [jimenez:swe-bench] which introduces SWE-bench, an evaluation framework for software engineering problems.
• "Agentless: Demystifying LLM-based Software Engineering Agents" [xia:agentless] which discusses LLM-based software engineering agents.
• "OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement" [zheng_opencodeinterpreter_2024] which introduces OpenCodeInterpreter, an open-source code system for generating, executing, and refining code.
• "Data analysis with ChatGPT | OpenAI Help Center" [chatgpt-ada] which discusses data analysis with ChatGPT.
• "Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs" [li:bird] which discusses using LLMs as a database interface.
• "Eran Yahav on X: "Which should now be (obviously!) recasted as the "fundamental theorem of GenAI"" / X" [yahav:fundamental-theorem-of-genai] which is a link to a tweet.
• "Using an LLM to Help With Code Understanding" [nam:gilt] which discusses using LLMs for code understanding.
• "Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMs" [cassano:multipl-t] which discusses knowledge transfer from high-resource to low-resource programming languages for code LLMs.
• "Are mutants a valid substitute for real faults in software testing?" [just:mutants] which discusses mutants as a valid substitute for real faults in software testing.
• "Can It Edit? Evaluating the Ability of LLMs to Follow Code Editing Instructions" [cassano:canitedit] which evaluates the ability of LLMs to follow code editing instructions.
• "Predicting typeScript type annotations and definitions with machine learning" [yee:dissertation] which is a PhD dissertation on predicting typeScript type annotations and definitions with machine learning.
• "Validating AI-Generated Code with Live Programming" [ferdowsi:live-LLM] which discusses validating AI-generated code with live programming.
• "Generation Probabilities Are Not Enough: Exploring the Effectiveness of Uncertainty Highlighting in AI-Powered Code Completions" [vasconcelos:gen-probs] which explores the effectiveness of uncertainty highlighting in AI-powered code completions.
• "The Programmer’s Assistant: Conversational Interaction with a LLM for Software Development" [ross:programmer-assistant] which discusses conversational interaction with a LLM for software development.
• "Productivity assessment of neural code completion" [ziegler_productivity_2022] which assesses the productivity of neural code completion.
• "Goals of the Luau Type System, Two Years On" [brown:luau-part-two] which discusses the goals of the Luau type system.
• "TypeScript migration - Strict type of cocktails - Front End Happy Hour" [netflix:ts] which discusses TypeScript migration at Netflix.
• "Sorbet: Stripe’s type checker for Ruby" [stripe:sorbet] which discusses Sorbet, Stripe's type checker for Ruby.
• "The Road to TypeScript at Quip, Part Two" [quip:ts] which discusses the road to TypeScript at Quip.
• "TypeScript at Slack" [slack:ts] which discusses TypeScript at Slack.
• "Our journey to type checking 4 million lines of Python" [dropbox:mypy] which discusses type checking Python at Dropbox.
• "GitHub Copilot: Your AI pair programmer" [github-copilot] which introduces GitHub Copilot.
• "Generative Agents: Interactive Simulacra of Human Behavior" [park_generative_2023] which introduces generative agents for simulating human behavior.
• "The Llama 3 Herd of Models | Research - AI at Meta" [noauthor_llama_nodate] which introduces the Llama 3 Herd of Models.
• "Can It Edit? Evaluating the Ability of LLMs to Follow Code Editing Instructions" [cassano:canitedit-LLM4code] which evaluates the ability of LLMs to follow code editing instructions.
• "Lost in Translation: A Study of Bugs Introduced by LLMs while Translating Code" [pan:lost-in-translation] which studies bugs introduced by LLMs while translating code.
• "No Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPT" [liu_no_2024] which assesses the quality of code generation by ChatGPT.
• "Sparse Autoencoders Find Highly Interpretable Features in LLMs" [cunningham_sparse_2023] which discusses sparse autoencoders finding highly interpretable features in LLMs.
• "Fuzz4All: Universal Fuzzing with LLMs" [xia:universal-fuzzing] which introduces Fuzz4All, a universal fuzzing tool with LLMs.
• "Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision" [burns_weak--strong_2024] which discusses weak-to-strong generalization.
• "Magicoder: Empowering Code Generation with OSS-Instruct" [wei:magicoder] which introduces Magicoder, empowering code generation with OSS-Instruct.
• "Data Race Detection Using LLMs" [chen:drb-ml] which discusses data race detection using LLMs.
• "On the transferability of pre-trained LLMs for low-resource programming languages" [chen_transferability_2022] which discusses the transferability of pre-trained LLMs for low-resource programming languages.
• "Multilingual Code Co-evolution using LLMs" [zhang_multilingual_2023] which discusses multilingual code co-evolution using LLMs.
• "Training LLMs to follow instructions with human feedback" [ouyang:instructgpt] which discusses training LLMs to follow instructions with human feedback.
• "Deduplicating Training Data Makes LLMs Better" [lee_deduplicating_2022] which shows that deduplicating training data makes LLMs better.
• "CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X" [qinkai:codegeex] which introduces CodeGeeX, a pre-trained model for code generation.
• "Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space" [geva_transformer_2022] which discusses how transformer feed-forward layers build predictions by promoting concepts in the vocabulary space.
• "Intrinsic Evaluation of Unlearning Using Parametric Knowledge Traces" [hong_intrinsic_2024] which discusses intrinsic evaluation of unlearning using parametric knowledge traces.
• "A Critical Study of What Code-LLMs (Do Not) Learn" [anand_critical_2024] which provides a critical paper of what code-LLMs (do not) learn.
• "DreamCoder: bootstrapping inductive program synthesis with wake-sleep library learning" [ellis_dreamcoder_2021] which introduces DreamCoder, a system for inductive program synthesis.
• "An Explanation of In-context Learning as Implicit Bayesian Inference" [xie_explanation_2021] which provides an explanation of in-context learning as implicit Bayesian inference.
• "Rethinking Data-driven Networking with Foundation Models: Challenges and Opportunities" [le_rethinking_2022] which discusses rethinking data-driven networking with foundation models.
• "Eliciting Latent Predictions from Transformers with the Tuned Lens" [belrose_eliciting_2023] which discusses eliciting latent predictions from transformers with the tuned lens.
• "Towards Automated Circuit Discovery for Mechanistic Interpretability" [conmy_towards_2023] which discusses automated circuit discovery for mechanistic interpretability.
• "What Algorithms can Transformers Learn? A Study in Length Generalization" [zhou:raspl] which studies what algorithms transformers can learn.
• "Future Lens: Anticipating Subsequent Tokens from a Single Hidden State" [pal_future_2023] which discusses anticipating subsequent tokens from a single hidden state.
• "Copilot Internals" [parth_thakkar_copilot_2024] which discusses Copilot internals.
• "Are Emergent Abilities of LLMs a Mirage?" [schaeffer_are_2023] which questions whether emergent abilities of LLMs are a mirage.
• "Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in LLMs" [marks_sparse_2024] which introduces methods for discovering and applying sparse feature circuits.
• "Learning Syntax Without Planting Trees: Understanding When and Why Transformers Generalize Hierarchically" [ahuja_learning_2024] which discusses learning syntax without planting trees.
• "Compositional Generalization and Decomposition in Neural Program Synthesis" [shi_compositional_2022] which discusses compositional generalization and decomposition in neural program synthesis.
• "Locating and Editing Factual Associations in GPT" [meng_locating_2022] which discusses locating and editing factual associations in GPT.