AI-Assisted Generation of Difficult Math Questions (2407.21009v3)

Abstract: Current LLM training positions mathematical reasoning as a core capability. With publicly available sources fully tapped, there is unmet demand for diverse and challenging math questions. Relying solely on human experts is both time-consuming and costly, while LLM-generated questions often lack the requisite diversity and difficulty. We present a design framework that combines the strengths of LLMs with a human-in-the-loop approach to generate a diverse array of challenging math questions. We leverage LLM metacognition skills [Didolkar et al., 2024] of a strong LLM to extract core "skills" from existing math datasets. These skills serve as the basis for generating novel and difficult questions by prompting the LLM with random pairs of core skills. The use of two different skills within each question makes finding such questions an "out of distribution" task for both LLMs and humans. Our pipeline employs LLMs to iteratively generate and refine questions and solutions through multiturn prompting. Human annotators then verify and further refine the questions, with their efficiency enhanced via further LLM interactions. Applying this pipeline on skills extracted from the MATH dataset [Hendrycks et al., 2021] resulted in MATH$^2$ - a dataset of higher-quality math questions, as evidenced by: (a) Lower performance of all models on MATH$^2$ than on MATH (b) Higher performance on MATH when using MATH$^2$ questions as in-context examples. Although focused on mathematics, our methodology seems applicable to other domains requiring structured reasoning, and potentially as a component of scalable oversight. Also of interest is a striking relationship observed between models' performance on the new dataset: the success rate on MATH$^2$ is the square on MATH, suggesting that successfully solving the question in MATH$^2$ requires a nontrivial combination of two distinct math skills.

AI-Assisted Generation of Difficult Math Questions: An Expert Overview

The paper, "AI-Assisted Generation of Difficult Math Questions," addresses the growing need for high-quality, diverse, and challenging mathematics questions by leveraging the strengths of LLMs in conjunction with human expertise. It presents a novel design framework that integrates a human-in-the-loop approach to produce difficult math questions efficiently.

Methodology

The proposed five-step pipeline intelligently combines LLM capabilities with human verification to achieve the goal:

1. Skill Pair Validation: The model first validates randomly selected pairs of distinct mathematical skills to ensure they are not qualitatively similar, using provided skill exemplars.
2. Question Generation: Utilizing validated skill pairs, the model generates novel questions incorporating both skills and provides brief solutions.
3. Solution Attempt: The model attempts to solve the generated question with a defeatist approach, identifying potential flaws in the question.
4. Question Validation: Using a fixed rubric, the model validates questions based on several criteria (e.g., single answer requirement, computational tractability), followed by majority voting for robustness.
5. Final Solution and Re-validation: The validated question is re-solved to ensure accuracy and consistency of the final solution, discarding questions with ambiguous answers.

Experimental Setup and Results

The framework was applied to the MATH dataset, extracting 114 distinct skills, which were then filtered and used to generate 180 verified questions, creating the MATH$^2$ dataset. This dataset was rigorously tested against various open-source and proprietary models.

Key findings include:

• All models showed significantly reduced performance on MATH$^2$ compared to the original MATH set, highlighting the increased difficulty of the new questions.
• The performance of models on MATH$^2$ followed an intuitive relationship: $Y \approx X^2$, where $X$ and $Y$ are the performance scores on MATH and MATH$^2$, respectively. This suggests that solving questions in MATH$^2$ requires a successful combination of two distinct math skills.

Implications and Future Directions

The creation of MATH$^2$ dataset demonstrates the efficacy of combining AI and human expertise. The findings emphasize the potential of AI-assisted question generation in other domains requiring structured reasoning beyond mathematics.

Practically, this framework provides a blueprint for generating high-quality, diverse datasets that challenge current AI models, facilitating more robust assessment and improvement. Furthermore, the ability to produce questions that are more difficult than standard datasets suggests this approach could be applied for advanced pedagogical purposes, potentially enhancing human learning processes.

The evident degradation in performance for smaller models and the pronounced relative performance drops indicate areas for improvement in training and diversification of synthetic data.

In future developments, scaling the process to generate larger datasets and improving computational efficiency are necessary. Reducing the dependency on human verification through advancements in automated validation tools and incorporating training-based feedback loops for the model are promising directions.

Conclusion

The paper presents a sophisticated, structured framework for generating challenging math questions, combining AI capabilities with human expertise. The proposed pipeline successfully creates a substantially more difficult dataset, MATH$^2$, which meaningfully tests the composed skills of advanced models. This work exemplifies how integrating AI with human input can surpass limitations of both, providing a path forward for scalable and diverse question generation across various domains.