Mathfish: Evaluating Language Model Math Reasoning via Grounding in Educational Curricula (2408.04226v3)

Abstract: To ensure that math curriculum is grade-appropriate and aligns with critical skills or concepts in accordance with educational standards, pedagogical experts can spend months carefully reviewing published math problems. Drawing inspiration from this process, our work presents a novel angle for evaluating LLMs' (LMs) mathematical abilities, by investigating whether they can discern skills and concepts enabled by math content. We contribute two datasets: one consisting of 385 fine-grained descriptions of K-12 math skills and concepts, or standards, from Achieve the Core (ATC), and another of 9.9K math problems labeled with these standards (MathFish). We develop two tasks for evaluating LMs' abilities to assess math problems: (1) verifying whether a problem aligns with a given standard, and (2) tagging a problem with all aligned standards. Working with experienced teachers, we find that LMs struggle to tag and verify standards linked to problems, and instead predict labels that are close to ground truth, but differ in subtle ways. We also show that LMs often generate problems that do not fully align with standards described in prompts, suggesting the need for careful scrutiny on use cases involving LMs for generating curricular materials. Finally, we categorize problems in GSM8k using math standards, allowing us to better understand why some problems are more difficult to solve for models than others.

• The paper introduces novel datasets and alignment tasks to evaluate how language models understand K–12 math standards.
• The paper finds that models like GPT-4 struggle with granular verification and precise standards tagging despite strong overall performance.
• The paper underscores the importance of teacher collaboration and refined methods for developing educational tools based on LM evaluations.

Evaluating LLM Math Reasoning via Grounding in Educational Curricula

The paper presents a nuanced evaluation of LLMs' (LMs) mathematical reasoning capabilities by scrutinizing their ability to discern and align with educational content grounded in K–12 curricular standards. This approach seeks to move beyond coarse-grained assessments and towards a more granular understanding of LMs' ability to identify specific math skills and concepts as defined by educational standards.

Key Contributions

1. Datasets: The authors introduce two pivotal datasets. The first, Achieve the Core (ATC), includes 385 fine-grained descriptions of K-12 math standards. The second, MathFish, encompasses 9.9K math problems labeled according to these standards. Both datasets are designed to capture the detailed progression and interconnections of mathematical concepts and skills as per the Common Core State Standards (CCSS).
2. Curricular Alignment Tasks:
   • Standards Verification: This task evaluates whether a given math problem aligns with a specified standard. LMs were tested on their ability to verify this alignment through binary yes/no questions.
   • Standards Tagging: This task involves tagging a problem with all relevant standards by navigating a hierarchical tree structure from general domains to specific standards.
3. Teacher Collaboration: Throughout the research, the team worked closely with experienced K-12 math teachers and curriculum reviewers to ensure practical relevance and educational validity. This collaboration was crucial for both dataset creation and model evaluation.

Findings

1. Verification Performance:
   • It was observed that LMs struggled with verification, especially as the negative examples became more similar to the positive ones. The best performance was achieved by GPT-4 in a three-shot setting, yet it still did not reach expert-level accuracy.
2. Tagging Performance:
   • The hierarchical tagging task revealed that stronger models like GPT-4 and Mixtral exhibited decreasing accuracy with increasing granularity—from domains to clusters to standards. Moreover, while the models could approximate the general area in which problems fell, pinpointing exact standards remained challenging.
3. Model-Specific Idiosyncrasies:
   • Distinct weaknesses in models' performance were noted, such as difficulties with trigonometry and ratios. These findings underscore the importance of detailed and standardized datasets like ATC for pinpointing specific educational weaknesses in LMs.

Implications and Future Directions

1. Practical Applications:
   • The ability to accurately tag and verify standards alignment has significant implications for automating parts of the curriculum review process, thus providing valuable support for educators. However, given the current performance levels, LMs should not yet replace human judgment in these tasks.
2. Development of Educational Tools:
   • As LMs are integrated into educational technologies, tools that assist in problem generation and curriculum alignment can benefit from the insights provided by this paper. For instance, the tendency of GPT-4 to overestimate alignment suggests areas for model improvement, particularly in understanding deeper pedagogical goals.
3. Further Research:
   • Future directions include extending these evaluations to multimodal and interactive educational content, which are crucial in math education but were outside the scope of this paper. Additionally, refining LMs to better handle the fine-grained distinctions within educational standards could significantly enhance their utility in curricula design and review.

Conclusion

The paper provides a comprehensive and fine-grained evaluation of LMs' understanding of math content grounded in educational standards, using newly developed datasets and engaging with domain experts. The findings highlight both the current capabilities and the limitations of LMs in educational applications, offering pathways for future improvements and new research avenues. Overall, this paper significantly advances our understanding of LMs' role in math education, advocating for continued refinement and careful integration of these models into educational tools and practices.