ClozeMath: Equation Infilling Benchmark

• ClozeMath is a mathematical reasoning benchmark that uses a text-infilling objective to mask and recover equation spans in multi-step solutions.
• It employs a transformer decoder with an extended span-infilling head to reconstruct equations, achieving improved accuracy on GSM8K and MATH benchmarks.
• The approach enhances model robustness by decoupling math structure from linguistic fluency and supports both beam search and chain-of-thought decoding.

ClozeMath is a mathematical reasoning benchmark and method for training LLMs to perform stepwise mathematical problem solving by predicting masked equations in multi-step solutions. It leverages a text-infilling objective, inspired by the cognitive processes involved in human cloze exercises, to improve alignment between text-based reasoning and mathematical structure (Pham et al., 4 Jun 2025).

1. Text-Infilling Objective and Training Methodology

ClozeMath augments the standard autoregressive language modeling objective with a text-infilling task. Given a training example composed of a natural-language problem statement and an interleaved text-plus-equation model-generated solution, all equation spans $e_i = (x_{a_i}, \dots, x_{b_i})$ are first identified. In each training pass, a random subset of these spans $\mathcal{M} \subseteq \{1, \dots, K\}$ is selected (where $$|\mathcal{M}| \sim \mathrm{Uniform}\{1,\dots,\min(K,\,3)\}$$). Each masked equation is replaced with a sentinel token $[\mathtt{MATH}_i]$:

$$x_{\backslash e_i} = (x_1, \dots, x_{a_i-1}, [\mathtt{MATH}_i], x_{b_i+1}, \dots, x_T)$$

The infilling objective is to maximize the conditional likelihood of each masked equation span $e_i$ given the context with masked equations:

$$\mathcal{L}_{\mathrm{inf}}(\theta) = -\sum_{i=1}^{K} \log p_{\theta}(e_i \mid x_{\backslash e_i})$$

This objective is combined with the standard next-token language modeling loss:

$$\mathcal{L}_{\mathrm{LM}}(\theta) = -\sum_{t=1}^T \log p_{\theta}(x_t \mid x_{<t})$$

yielding the total training loss:

$$\mathcal{L}(\theta) = \mathcal{L}_{\mathrm{LM}}(\theta) + \lambda \mathcal{L}_{\mathrm{inf}}(\theta)$$

where $\lambda$ is typically set to 1.

2. Masking Protocol, Equation Detection, and Sampling

ClozeMath distinguishes itself by masking only equation spans, in contrast to approaches such as Masked Thought which mask arbitrary text or equation segments. Equation spans are detected using rule-based or LaTeX parser-based detectors. During training, a subset (up to three) of the detected equation spans is masked per example, with each span replaced by a unique sentinel token. This approach enforces reliance on textual and logical cues for equation recovery rather than superficial token patterns, reducing the risk of shortcut learning.

3. Model Architecture and Baseline Comparison

The underlying model is a transformer decoder extended with a span-infilling head. The main transformer backbone and next-token projection remain unchanged. For infilling, the model uses the hidden state at the sentinel token $\mathcal{M} \subseteq \{1, \dots, K\}$0 as input to a span decoder, which autoregressively predicts the tokens constituting the masked equation $\mathcal{M} \subseteq \{1, \dots, K\}$1 until reaching a special end-of-span marker.

Contrasts to principal baselines:

• Standard Next-Token LM: Only learns the left-to-right objective $\mathcal{M} \subseteq \{1, \dots, K\}$2; no masking or infilling.
• Masked Thought: Masks arbitrary contiguous spans—including both equations and explanatory text—whereas ClozeMath restricts infilling to equations specifically, preventing the model from sidestepping mathematical reasoning by copying nearby context.

4. Decoding Strategies at Inference

ClozeMath supports two test-time decoding procedures:

• Beam Search Decoding: Runs conventional length-normalized beam search over the entire question-to-solution sequence, invoking the infilling head whenever a sentinel token is encountered, analogous to standard LM decoding but with equation span reconstruction.
• Chain-of-Thought (CoT) Decoding: Explicitly walks through the solution skeleton, initially with all equations masked. For each equation, the model conditions on the problem and current partially-filled solution, infills one equation per step, and iterates until all masked spans have been filled. This process enforces dependency alignment reflective of human stepwise reasoning.

5. Empirical Performance and Robustness

Experiments evaluate ClozeMath on benchmarks GSM8K (grade-school arithmetic), MATH (competition-style problems), and GSM-Symbolic. Model performance is reported as exact-match accuracy and pass@k:

$\mathcal{M} \subseteq \{1, \dots, K\}$3

$\mathcal{M} \subseteq \{1, \dots, K\}$4

where $\mathcal{M} \subseteq \{1, \dots, K\}$5 is the model's probability of correctly solving the $\mathcal{M} \subseteq \{1, \dots, K\}$6-th sample.

Summary of results:

Benchmark	Standard LM	Masked Thought	ClozeMath	ClozeMath w/ CoT Decoding
GSM8K (pass@1)	75.2%	76.8%	78.9%	80.3%
MATH (pass@1)	42.1%	43.7%	45.5%	47.0%
GSM-Symbolic	71.4%	72.2%	74.0%	–

Robustness is assessed by introducing symbol renaming (e.g., $\mathcal{M} \subseteq \{1, \dots, K\}$7) and unit perturbations (e.g., "meters" → "centimeters"). ClozeMath models lose less than 2 points in absolute accuracy under such perturbations versus typical 4–5 point drops with standard LM fine-tuning.

6. Ablation Experiments

Three core components are ablated:

• Infilling Task Inclusion: The principal accuracy gain (>2 points on GSM8K) arises from incorporating equation-only infilling into the training objective.
• Type of Masking: Masking both equations and text (as in Masked Thought) provides smaller or even negative returns (–0.5 points), showing that focusing masking on equations alone is beneficial.
• Number of Masked Spans: Permitting up to three masked equations per instance yields a modest improvement (+0.7 points) over masking a single span.

These findings underline that ClozeMath's selective equation masking is the critical factor in enabling robust mathematical generalization.

7. Underlying Mechanisms and Explanatory Factors

The ClozeMath training scheme provides two primary benefits:

• Decoupling Structure from Fluency: By reconstructing entire equations from natural-language context, the model is forced to encode accurate text-to-math mappings rather than rely on spurious statistical artifacts, such as frequent next-token patterns.
• Implicit Chain-of-Thought Alignment: Although no explicit supervision of intermediate reasoning chains is provided, the span infilling task encourages attention to the hierarchical dependencies typical of human reasoning (problem → intermediate equations → final answer). CoT decoding at inference further exploits these learned dependencies for coherent multi-step solutions.

The result is improved mathematical reasoning accuracy and intermediate step quality, without requiring annotated chains of thought or exposure to gold-standard intermediate rationales (Pham et al., 4 Jun 2025).

---

A complementary direction is multi-tool integration frameworks, which combine LLMs with external modules (arithmetic tools, code execution, and chain-of-thought reasoning) for cloze-style math tasks. Such approaches achieve high empirical accuracy (e.g., 89.09% on NumGLUE Task 4) and provide complementary insights into architectural design, tool orchestration, and aggregation through self-consistency voting (Duan et al., 2024). However, unlike ClozeMath, these methods primarily address integration and tool selection rather than the core text-to-equation mapping learned solely through masked infilling.