MathQA-Python Benchmark Dataset

• MathQA-Python is a large-scale benchmarking dataset that evaluates neural models' ability to convert math word problems into executable Python functions.
• It is derived from MathQA by mechanically translating a custom DSL into Python code, resulting in 23,914 high-quality labeled instances across training, validation, and test splits.
• Benchmark results reveal that fine-tuning significantly outperforms few-shot prompting, underscoring the dataset's importance in advancing neural-symbolic reasoning and scalable program synthesis research.

MathQA-Python is a large-scale benchmarking dataset derived from MathQA for evaluating program synthesis and mathematical reasoning by neural models through direct translation of math word problems into executable Python code. It is widely used in both program synthesis and math-aware question answering contexts, providing a rigorous testbed for tasks that require converting natural language mathematical questions into correct numerical computations via code.

1. Dataset Structure and Origin

MathQA-Python originates from the MathQA dataset, which itself is an expansion of AQuA, comprising annotated math word problems with executable “operation sequencing” DSL programs. MathQA-Python is produced by mechanically translating the original MathQA operation programs into equivalent Python functions. After filtering problems whose code did not evaluate to the correct answer, the final MathQA-Python dataset contains 23,914 labeled instances (Austin et al., 2021). Each sample includes:

• A natural-language paragraph describing a mathematical problem.
• A ground-truth solution in the form of a Python function:

  def solve(): # intermediate calculations return <numeric_result>

• The dataset is split into 19,209 training, 2,822 validation, and 1,883 test problems.

Problems span a range of arithmetic and algebraic operations, including addition, subtraction, multiplication, division, exponentiation, fractional arithmetic, and multi-step reasoning (e.g., nested expressions such as "Compute the volume of acid in a solution mixing two concentrations") (Austin et al., 2021, Amini et al., 2019).

2. Formal Representation and Translation

The ground-truth programs in MathQA were originally written in a bespoke operation-based DSL: a sequence of functional operations over constants, problem numbers, and intermediate results. The translation layer for MathQA-Python maps each DSL sequence to an equivalent imperative Python code block, preserving the problem's stepwise computation and structure (Austin et al., 2021). Example translation:

• DSL: add(n1, n2)|divide(n3, #0)
• Python:

  def solve(): t0 = n1 + n2 t1 = n3 / t0 return t1

The translation maintains semantic fidelity, ensuring execution results match the labeled answer. The process discards problems whose translation does not numerically match the ground-truth, enhancing dataset quality for program synthesis evaluation.

3. Baseline Modeling Paradigms

MathQA-Python underlies the mathematical program synthesis evaluation pipeline for LLMs, particularly decoder-only Transformer architectures (Austin et al., 2021). Typical problem-solution workflows follow:

• Input: Single math word problem in English.
• Task: Synthesize a Python function that, when executed, computes the correct answer.
• Evaluation: The model generates code, which is executed. The output must numerically match the ground-truth.

Prompting and fine-tuning setups include:

• Few-shot prompting with 4 in-domain problem–code pairs per prompt.
• Fine-tuning on 19,209 training examples.
• No additional input/output assertion examples are provided at inference—only docstring-to-code mappings.

Assessment is based on the fraction of test tasks for which the model’s generated code yields the correct answer.

4. Quantitative Performance and Scaling Trends

Quantitative benchmark results demonstrate the scaling behavior of LLMs on MathQA-Python. Performance metrics (fraction of test problems solved, i.e., code-generated answer matches ground-truth) are summarized as follows (Austin et al., 2021):

Model Size	Few-shot (Python)	Fine-tuned (Python)
8B	12.5%	74.7%
68B	22.3%	79.5%
137B	33.4%	81.2%

In the DSL variant (before translation to Python), the best model achieves 83.8% accuracy. Notably, accuracy grows roughly linearly with the logarithm of the number of parameters in the few-shot regime, consistent with scaling laws observed in related coding benchmarks (e.g., MBPP) (Austin et al., 2021).

Fine-tuning yields a substantial discontinuous boost over few-shot prompting, with performance flattening for larger model sizes. Program execution-based evaluation is critical: only syntactically and semantically correct code that computes the correct numeric answer is counted as solved.

5. Comparative Analyses and Error Patterns

The MathQA-Python benchmark, when used to evaluate LLMs, highlights several key phenomena (Austin et al., 2021):

• Larger models (<137B) consistently improve over smaller ones, especially under few-shot prompting.
• Model performance on MathQA-Python is lower than on simpler program synthesis benchmarks (e.g., MBPP), reflecting increased problem and reasoning complexity.
• Human-in-the-loop dialog (studied in MBPP context) shows that supplying up to four natural-language hints or feedback can halve the model's error rate in program synthesis tasks. While not directly repeated for MathQA-Python, this suggests dialogic protocols may offer similar gains for mathematical reasoning.
• Error analysis (from MBPP findings) indicates that multi-constraint logical tasks, off-by-one failures, and uncommon mathematical sequences are among the hardest classes of problems, with prompt sensitivity capable of shifting solve rates by up to 20 percentage points.

6. Applications in Math-Aware Question Answering Systems

MathQA-Python serves as a foundation for both QA benchmarking and end-to-end system development. In math-aware question answering system architectures, such as the MathQA–Python system (Schubotz et al., 2019), related principles are employed:

• Natural-language parsing is applied to extract structured triples or parse trees from user queries.
• Formula retrieval is performed against knowledge bases (e.g., Wikidata) to obtain canonical mathematical formulae.
• These formulae are automatically translated to a computable representation (via e.g., latex2sympy2), populated with user-supplied or database-derived numerics, and evaluated (typically using Sympy or equivalent computer algebra systems).
• Integrated pipelines support multilingual input, constant retrieval, and robust formula parsing.

A plausible implication is that advanced program synthesis benchmarks such as MathQA-Python both stress-test LLMs’ mathematical reasoning abilities and inform the design of next-generation symbolic QA systems that bridge language and machine computation (Schubotz et al., 2019).

7. Reproducibility, Data Availability, and Known Limitations

MathQA-Python is available as part of the program synthesis code and data artifacts in (Austin et al., 2021). Its construction and curation rely on reproducible translation scripts derived from the MathQA operation-based DSL.

Known limitations include:

• Coverage is restricted to mathematical problems supported by the original DSL and translation logic.
• Only those instances where automatic translation to Python could be semantically verified are included, filtering out complex or ambiguous cases.
• Program generation performance remains significantly below human expert levels, indicating a persistent gap even at current LLM scales.

The benchmark has catalyzed ongoing developments in high-precision program synthesis, automated reasoning with neural-symbolic systems, and interpretable QA model evaluation (Austin et al., 2021, Amini et al., 2019, Schubotz et al., 2019).