MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education (2106.07340v5)

Abstract: Since the introduction of the original BERT (i.e., BASE BERT), researchers have developed various customized BERT models with improved performance for specific domains and tasks by exploiting the benefits of transfer learning. Due to the nature of mathematical texts, which often use domain specific vocabulary along with equations and math symbols, we posit that the development of a new BERT model for mathematics would be useful for many mathematical downstream tasks. In this resource paper, we introduce our multi-institutional effort (i.e., two learning platforms and three academic institutions in the US) toward this need: MathBERT, a model created by pre-training the BASE BERT model on a large mathematical corpus ranging from pre-kindergarten (pre-k), to high-school, to college graduate level mathematical content. In addition, we select three general NLP tasks that are often used in mathematics education: prediction of knowledge component, auto-grading open-ended Q&A, and knowledge tracing, to demonstrate the superiority of MathBERT over BASE BERT. Our experiments show that MathBERT outperforms prior best methods by 1.2-22% and BASE BERT by 2-8% on these tasks. In addition, we build a mathematics specific vocabulary 'mathVocab' to train with MathBERT. We discover that MathBERT pre-trained with 'mathVocab' outperforms MathBERT trained with the BASE BERT vocabulary (i.e., 'origVocab'). MathBERT is currently being adopted at the participated leaning platforms: Stride, Inc, a commercial educational resource provider, and ASSISTments.org, a free online educational platform. We release MathBERT for public usage at: https://github.com/tbs17/MathBERT.

• The paper introduces MathBERT, a pre-trained language model specifically adapted and fine-tuned for NLP tasks in mathematics education.
• MathBERT was trained on a large corpus of mathematical texts using a math-specific vocabulary (mathVocab) to handle specialized symbols and terminology.
• Empirical evaluations show MathBERT achieves substantial performance gains (2-8% task accuracy improvements) over base BERT on tasks like knowledge component prediction and auto-grading.

Summary of "MathBERT: A Pre-trained LLM for General NLP Tasks in Mathematics Education"

The paper introduces MathBERT, a domain-specific adaptation of the BERT LLM aimed at enhancing NLP tasks within the field of mathematics education. Recognizing the distinctive characteristics of mathematical texts, which frequently feature specialized terminology and symbols, MathBERT is engineered to better interpret these texts for educational applications. The model was developed through a collaborative effort encompassing multiple educational platforms and academic institutions.

Development and Features of MathBERT

MathBERT builds on the base BERT model, leveraging transfer learning to fine-tune the model using a comprehensive corpus sourced from mathematical texts. This corpus spans a broad spectrum, including materials suitable for pre-kindergarten up to advanced-level mathematical content from academia. An essential aspect of its development was constructing a mathematics-specific vocabulary—mathVocab—which facilitates the improved comprehension of mathematical symbols and nomenclature distinct from the general-domain texts used in base BERT pre-training.

MathBERT's training involved pre-processing the data similar to strategies employed by contemporaries like ROBERTa, ensuring robustness through large-scale pre-training with sequences maximizing length capacity for efficiency. The pre-training process effectively reached its optimal state after approximately five days of training, with meticulous attention to maximizing Masked LLMing (MLM) accuracy.

Performance Evaluation

MathBERT was subjected to a range of NLP tasks typical in mathematics education: knowledge component prediction, auto-grading of open-ended answers, and knowledge tracing correctness prediction. In empirical assessments, MathBERT demonstrated substantial performance improvements. MathBERT surpassed previous methods and attained superior metrics over BASE BERT, marking improvements between 2% to 8% on task accuracies.

Specifically,:

• In knowledge component prediction, MathBERT exhibited increased precision (F1) and accuracy (ACC) relative to established methods and BASE BERT.
• For the auto-grading task, MathBERT achieved AUC improvements, highlighting its proficiency in effectively assessing and scoring open-ended responses.
• MathBERT also showcased enhanced prediction abilities for knowledge tracing tasks, achieving higher AUC and ACC scores.

Implications and Future Directions

The practical implications of MathBERT are significant for educational platforms striving to offer automated grading and feedback systems, enhancing the scalability and depth of learning tools available to educators. In particular, platforms like ASSISTments and Stride, Inc. are in stages of integrating MathBERT to streamline their educational processes, utilizing its predictions capabilities to augment both teacher and student experiences.

Theoretically, MathBERT sets a precedent in demonstrating the value of domain-specific LLMs in educational contexts. It amplifies the discourse on applying advanced NLP methods to specific fields outside general language processing. Potential future explorations could address further domain-specific model integrations or extending MathBERT capabilities into intersecting educational domains.

The development of MathBERT opens avenues for more targeted educational interventions, providing newfound precision in understanding and responding to mathematical language, thus promising richer educational insights and more engaging learning environments.