MATHWELL: Open-Source K–8 Math Problem Generator

• MATHWELL is an open-source framework that automatically generates K–8 math word problems using teacher-validated annotations and a two-stage fine-tuning approach.
• It integrates supervised fine-tuning of Llama-2 (70B) with synthetic, human-authored, and teacher-annotated data to ensure high solvability, accuracy, and educational appropriateness.
• Quantitative evaluations show MATHWELL outperforms open-source baselines and rivals proprietary models in delivering context-free, pedagogically sound math problems.

MATHWELL is an open-source generative framework and annotated dataset for automatically producing educationally appropriate K–8 math word problems. Its primary contribution is the integration of teacher-validated annotation in the training pipeline of a LLM, establishing strict control over solvability, answer accuracy, and suitability for its designated educational cohort. Through a two-stage fine-tuning procedure atop Llama-2 (70B), MATHWELL produces problems that closely match, and in some metrics rival, outputs from proprietary models such as GPT-4, while consistently outperforming open-source baselines on both pedagogical and automatic evaluation criteria (Christ et al., 2024).

1. Model Architecture and Training

MATHWELL is realized via supervised fine-tuning of Llama-2 (70B parameters) on a combination of human-authored, synthetic, and teacher-annotated sources. Problem generation is context-free, i.e., unconstrained by any explicit curriculum tagging, and every word problem is coupled with a Program-of-Thought (PoT) solution expressed as an executable Python function.

Two-Stage Supervised Fine-Tuning

• Stage 1: General Mathematical Reasoning
  • The model is initially fine-tuned on datasets such as MathInstruct GSM8K (6,403 human-written K–8 problems with PoT solutions), MathInstruct MATH, and TheoremQA with similar solution formats.
  • Prompts are given in an 8-shot Alpaca-style structure, culminating in a directive to "Write a grade school math word problem …".
  • Optimization uses QLoRA (Dettmers et al., 2023): 4,250 steps, learning rate $1\times 10^{-4}$ (AdamW), LoRA adapters in every transformer layer, batch size 1 per GPU (2× NVIDIA A100), 3% warm-up steps.
• Stage 2: Educational Appropriateness Calibration
  • The Stage 1 model generates 3,234 synthetic question/answer pairs.
  • Domain experts with K–12 experience annotate these for: Solvability, Accuracy, and Appropriateness.
  • The MaC (Meets all Criteria) subset (1,906 examples) is used for a further 1,250 fine-tuning steps.

Annotation Schema

Annotation is performed with a custom Zooniverse interface, where each item is labeled:

• Solvability: Is the answerable with provided information?
• Accuracy: Is the solution mathematically correct?
• Appropriateness: Would a teacher assign it to a middle-schooler? If not, reasons are recorded (too hard, illogical, inappropriate, grammar).

By focusing on these criteria, MATHWELL internalizes K–8 pedagogical constraints and mathematical veracity (Christ et al., 2024).

2. Data Composition and Properties

The core synthetic corpus is the SGSM (Synthetic Grade School Math) dataset: 20,490 question/answer pairs. The annotated MaC subset contains 2,093 examples, with an additional 18,397 unannotated samples that include programmatic solutions.

Distributional Properties

• Problem Type Breakdown (approximate):
  • Addition/Subtraction: ~60%
  • Multiplication/Division: ~20%
  • Fractions/Decimals: ~15%
  • Mixed operations (≥2 operations): ~5%
• Subject Matter: Topics align with K–8 student interests (animals, superheroes, video games, sports, etc.).
• Reading Level: Measured using Flesch-Kincaid Grade Level (FKGL).
  • SGSMTrain mean FKGL = 2.50 (SD=1.76)
  • SGSMUnannotated mean FKGL = 2.68 (SD=1.97)
  • No problems are generated above grade 8 in reading accessibility.

3. Evaluation Methodology

Evaluation uses mixed methods, balancing expert human judgment with quantitative metrics.

Human-Evaluation Metrics

Defined per $N_{\text{tot}}=250$ samples/model:

• Solvability rate ($\mathrm{SolvRate}$): $N_{\mathrm{solv}}/N_{\text{tot}}$
• Accuracy rate ($\mathrm{AccRate}$): $N_{\mathrm{acc}}/N_{\text{tot}}$
• Appropriateness rate ($\mathrm{AppRate}$): $N_{\mathrm{app}}/N_{\text{tot}}$
• MaC rate ($\mathrm{MaCRate}$): $$N_{\mathrm{solv} \wedge \mathrm{acc} \wedge \mathrm{app}} / N_{\text{tot}}$$

Annotator agreement levels: 95% (solvability), 96% (accuracy), 80% (appropriateness).

Automatic-Evaluation Metrics

• Perplexity (PPL) of question text (measured using Llama-2 70B).
• BERTScore F1 (vs. MathInstruct GSM8K).
• Reading-level scores: FKGL and New Dale–Chall (NDC).

Results Table

Metric	GPT-4 T	GPT-3.5 T	Llama-2 70B	MAmmoTH 70B	LLEMMA 34B	MATHWELL
Solvable	94.8%	88.0%	84.0%	86.8%	48.8%	89.2% (±2.0%)*
Accurate	95.8%	89.5%	89.5%	94.9%	63.9%	96.9% (±1.2%)*
Appropriate	84.4%	75.5%	81.0%	67.7%	41.8%	86.5% (±2.3%)*
MaC	78.8%	62.8%	62.4%	56.8%	15.2%	74.8% (±2.8%)*

*Significantly higher ($N_{\text{tot}}=250$0) vs. next-best open-source model.

• MATHWELL achieves MaCRate = 74.8%, within 94.9% of GPT-4 T’s 78.8%.
• Model-generation PPL (MATHWELL): 2.44—lowest among open-source (Christ et al., 2024).

4. Qualitative Examples and Program-of-Thought Solutions

MATHWELL produces problems aligned to distinct grade levels. Each generated item includes a commented Python solution, reflecting Program-of-Thought structure.

• Grade 1 (Addition/Subtraction; FKGL ≈ 1.8): "A soccer team has 11 players on the field and 22 players on the bench. How many players are there in total?" $N_{\text{tot}}=250$1
• Grade 4 (Multiplication; FKGL ≈ 3.2): "Super Mario stomps on 30 Goombas, 20 Koopas, and 10 Piranha Plants. How many enemies does he stomp on in total?" $N_{\text{tot}}=250$2
• Grade 7 (Average & Subtraction; FKGL ≈ 6.5): "LeBron James has scored 12,000 points and is 4,000 points behind the all-time record. Over the next 20 games, how many points per game must he average to break the record?" $N_{\text{tot}}=250$3

These outputs illustrate adaptive complexity, content familiarity, and clear scaffolding in the corresponding PoT solutions (Christ et al., 2024).

5. Limitations

• Grade-alignment is not explicitly conditioned: MATHWELL does not directly utilize grade-level or curriculum tags during training.
• Appropriateness remains subjective: Teacher judgments of appropriateness are inherently variable.
• High cost of annotation: Human labeling, especially for appropriateness and answer correctness, incurs significant labor at scale.

A plausible implication is that further automatic measures for appropriateness and answer validity are necessary to extend the system beyond K–8 without introducing bias or drift (Christ et al., 2024).

6. Prospective Directions

• Conditioning generation on explicit grade-level or curriculum-standard topics.
• Incorporating methods such as RLHF or self-alignment to reduce annotation overhead and refine outputs.
• Development of automatic classifiers for solvability, accuracy, and appropriateness based on balanced annotated datasets.
• Extension to support multi-language output or coverage of higher-grade problems (9–12).

The systematic integration of teacher annotation and a two-stage fine-tuning architecture positions MATHWELL as a robust benchmark for context-free, educationally appropriate K–8 math word problem generation that approaches closed-source state-of-the-art in several key metrics (Christ et al., 2024).