---
title: 'MATHWELL: Open-Source K–8 Math Problem Generator'
url: https://www.emergentmind.com/topics/mathwell
type: topic
---

# MATHWELL: Open-Source K–8 Math Problem Generator

MATHWELL is an open-source generative framework and annotated dataset for automatically producing educationally appropriate K–8 math word problems. Its primary contribution is the integration of teacher-validated annotation in the training pipeline of a large language model, establishing strict control over solvability, answer accuracy, and suitability for its designated educational cohort. Through a two-stage fine-tuning procedure atop Llama-2 (70B), MATHWELL produces problems that closely match, and in some metrics rival, outputs from proprietary models such as GPT-4, while consistently outperforming open-source baselines on both pedagogical and automatic evaluation criteria [2402.15861].

## 1. Model Architecture and Training

MATHWELL is realized via supervised fine-tuning of Llama-2 (70B parameters) on a combination of human-authored, synthetic, and teacher-annotated sources. Problem generation is context-free, i.e., unconstrained by any explicit curriculum tagging, and every word problem is coupled with a Program-of-Thought (PoT) solution expressed as an executable Python function.

### Two-Stage Supervised Fine-Tuning

- **Stage 1: General Mathematical Reasoning**
    - The model is initially fine-tuned on datasets such as MathInstruct GSM8K (6,403 human-written K–8 problems with PoT solutions), MathInstruct MATH, and TheoremQA with similar solution formats.
    - Prompts are given in an 8-shot Alpaca-style structure, culminating in a directive to "Write a grade school math word problem …".
    - Optimization uses QLoRA (Dettmers et al., 2023): 4,250 steps, learning rate $1\times 10^{-4}$ (AdamW), LoRA adapters in every transformer layer, batch size 1 per GPU (2× NVIDIA A100), 3% warm-up steps.

- **Stage 2: Educational Appropriateness Calibration**
    - The Stage 1 model generates 3,234 synthetic question/answer pairs.
    - Domain experts with K–12 experience annotate these for: Solvability, Accuracy, and Appropriateness.
    - The MaC (Meets all Criteria) subset (1,906 examples) is used for a further 1,250 fine-tuning steps.

### Annotation Schema

Annotation is performed with a custom Zooniverse interface, where each item is labeled:
- *Solvability*: Is the answerable with provided information?
- *Accuracy*: Is the solution mathematically correct?
- *Appropriateness*: Would a teacher assign it to a middle-schooler? If not, reasons are recorded (too hard, illogical, inappropriate, grammar).

By focusing on these criteria, MATHWELL internalizes K–8 pedagogical constraints and mathematical veracity [2402.15861].

## 2. Data Composition and Properties

The core synthetic corpus is the SGSM (Synthetic Grade School Math) dataset: 20,490 question/answer pairs. The annotated MaC subset contains 2,093 examples, with an additional 18,397 unannotated samples that include programmatic solutions.

### Distributional Properties

- **Problem Type Breakdown** (approximate):
    - Addition/Subtraction: ~60%
    - Multiplication/Division: ~20%
    - Fractions/Decimals: ~15%
    - Mixed operations (≥2 operations): ~5%
- **Subject Matter**: Topics align with K–8 student interests (animals, superheroes, video games, sports, etc.).
- **Reading Level**: Measured using Flesch-Kincaid Grade Level (FKGL).
    - SGSMTrain mean FKGL = 2.50 (SD=1.76)
    - SGSMUnannotated mean FKGL = 2.68 (SD=1.97)
    - No problems are generated above grade 8 in reading accessibility.

## 3. Evaluation Methodology

Evaluation uses mixed methods, balancing expert human judgment with quantitative metrics.

### Human-Evaluation Metrics

Defined per $N_{\text{tot}}=250$ samples/model:
- Solvability rate ($\mathrm{SolvRate}$): $N_{\mathrm{solv}}/N_{\text{tot}}$
- Accuracy rate ($\mathrm{AccRate}$): $N_{\mathrm{acc}}/N_{\text{tot}}$
- Appropriateness rate ($\mathrm{AppRate}$): $N_{\mathrm{app}}/N_{\text{tot}}$
- MaC rate ($\mathrm{MaCRate}$): $N_{\mathrm{solv} \wedge \mathrm{acc} \wedge \mathrm{app}} / N_{\text{tot}}$

Annotator agreement levels: 95% (solvability), 96% (accuracy), 80% (appropriateness).

### Automatic-Evaluation Metrics

- Perplexity (PPL) of question text (measured using Llama-2 70B).
- BERTScore F1 (vs. MathInstruct GSM8K).
- Reading-level scores: FKGL and New Dale–Chall (NDC).

#### Results Table

| Metric         | GPT-4 T | GPT-3.5 T | Llama-2 70B | MAmmoTH 70B | LLEMMA 34B | MATHWELL        |
|---------------|---------|-----------|-------------|-------------|------------|-----------------|
| Solvable      | 94.8%   | 88.0%     | 84.0%       | 86.8%       | 48.8%      | 89.2% (±2.0%)*  |
| Accurate      | 95.8%   | 89.5%     | 89.5%       | 94.9%       | 63.9%      | 96.9% (±1.2%)*  |
| Appropriate   | 84.4%   | 75.5%     | 81.0%       | 67.7%       | 41.8%      | 86.5% (±2.3%)*  |
| MaC           | 78.8%   | 62.8%     | 62.4%       | 56.8%       | 15.2%      | 74.8% (±2.8%)*  |

*Significantly higher ($p < .01$) vs. next-best open-source model.

- MATHWELL achieves MaCRate = 74.8%, within 94.9% of GPT-4 T’s 78.8%.
- Model-generation PPL (MATHWELL): 2.44—lowest among open-source [2402.15861].

## 4. Qualitative Examples and Program-of-Thought Solutions

MATHWELL produces problems aligned to distinct grade levels. Each generated item includes a commented Python solution, reflecting Program-of-Thought structure.

- **Grade 1 (Addition/Subtraction; FKGL ≈ 1.8)**: "A soccer team has 11 players on the field and 22 players on the bench. How many players are there in total?"
    ```python
    def solution():
        on_field = 11
        on_bench = 22
        total_players = on_field + on_bench  # 11 + 22
        return total_players  # Expected output: 33
    ```
- **Grade 4 (Multiplication; FKGL ≈ 3.2)**: "Super Mario stomps on 30 Goombas, 20 Koopas, and 10 Piranha Plants. How many enemies does he stomp on in total?"
    ```python
    def solution():
        goombas = 30
        koopas = 20
        piranha_plants = 10
        total = goombas + koopas + piranha_plants  # 30 + 20 + 10
        return total  # Expected output: 60
    ```
- **Grade 7 (Average & Subtraction; FKGL ≈ 6.5)**: "LeBron James has scored 12,000 points and is 4,000 points behind the all-time record. Over the next 20 games, how many points per game must he average to break the record?"
    ```python
    def solution():
        current = 12000
        record = current + 4000  # 16000 total needed
        games = 20
        needed_per_game = (record - current) / games  # 4000 / 20
        return needed_per_game  # Expected output: 200
    ```

These outputs illustrate adaptive complexity, content familiarity, and clear scaffolding in the corresponding PoT solutions [2402.15861].

## 5. Limitations

- Grade-alignment is not explicitly conditioned: MATHWELL does not directly utilize grade-level or curriculum tags during training.
- Appropriateness remains subjective: Teacher judgments of appropriateness are inherently variable.
- High cost of annotation: Human labeling, especially for appropriateness and answer correctness, incurs significant labor at scale.

*A plausible implication is* that further automatic measures for appropriateness and answer validity are necessary to extend the system beyond K–8 without introducing bias or drift [2402.15861].

## 6. Prospective Directions

- Conditioning generation on explicit grade-level or curriculum-standard topics.
- Incorporating methods such as RLHF or self-alignment to reduce annotation overhead and refine outputs.
- Development of automatic classifiers for solvability, accuracy, and appropriateness based on balanced annotated datasets.
- Extension to support multi-language output or coverage of higher-grade problems (9–12).

The systematic integration of teacher annotation and a two-stage fine-tuning architecture positions MATHWELL as a robust benchmark for context-free, educationally appropriate K–8 math word problem generation that approaches closed-source state-of-the-art in several key metrics [2402.15861].

Source: https://www.emergentmind.com/topics/mathwell