MathIF: Evaluating Math Instruction-Following

• MathIF Benchmark is a comprehensive framework that assesses large language models' ability to follow specific mathematical instructions.
• It applies 15 different constraint types across 420 problems from varying difficulty levels to measure both reasoning accuracy and instruction compliance.
• Empirical insights reveal a trade-off where enhanced mathematical reasoning often reduces obedience to user-imposed constraints.

The MathIF Benchmark is a comprehensive framework created to systematically evaluate the instruction-following ability of LLMs in mathematical reasoning contexts. MathIF stands out for its focus on aligning mathematical intelligence with user intent, probing the trade-off between reasoning power and obedience to natural language directives, a critical but previously underexplored dimension in mathematical AI research.

1. Structural Overview and Design Rationale

MathIF comprises 420 evaluation samples, each derived from mathematically challenging problems spanning a wide range of difficulty—from primary school-level word problems to international competition-level questions. The uniqueness of MathIF lies in its programmatic injection of 15 different constraint types onto these math problems to systematically test instruction-following under controlled yet diverse real-world settings.

Constraints are grouped into four canonical types:

• Length constraints: e.g., "Answer with less than 500 words"
• Lexical constraints: e.g., "Your answer should be in Chinese" or "Include the keyword 'condition'"
• Format constraints: e.g., "Do not use any commas", "Your response must have three bullet points"
• Affix constraints: e.g., "Repeat the request word-for-word before your answer", "Finish your response with: 'Any other questions?'"

These constraints are applied singly or in combinatorial fashion (double-constraint and triple-constraint prompts), yielding a testbed that closely mirrors the complexity of user instructions encountered in practice.

Problems are sourced from GSM8K, MATH-500, Minerva, Olympiad, and AIME datasets, ensuring a full spectrum of mathematical reasoning scenarios.

2. Instruction-Following Metrics and Evaluation Protocol

MathIF is specifically architected to assess not only a model’s mathematical correctness but also its ability to comply with varied and sometimes convoluted user instructions. Two complementary evaluation metrics are introduced:

• Hard Accuracy (HAcc): Defined strictly as

$\operatorname{HAcc} = \prod_{i=1}^n I(C_i)$

where $I(C_i) = 1$ if constraint $C_i$ is met, $0$ otherwise. A response is only considered correct if all constraints are met.

• Soft Accuracy (SAcc): Defined as the mean per-constraint compliance,

$$\operatorname{SAcc} = \frac{1}{n} \sum_{i=1}^n I(C_i)$$

capturing granular per-constraint obedience even if some are missed.

Sample-level HAcc provides a binary measure of obedience per query, while SAcc quantifies constraint-level compliance, supporting nuanced diagnostics, particularly as constraints accumulate.

3. Empirical Insights and the Intelligence–Obedience Trade-off

A principal finding of MathIF is a persistent tension between scaling mathematical reasoning capability and maintaining instruction adherence. As models increase in size or are trained to employ longer chains-of-thought (CoT), their ability to solve complex math problems improves, but their fidelity to user constraints tends to degrade.

Empirical analysis demonstrates that the best-performing model in the paper (Qwen3-14B) attains only 50.71% hard accuracy; this drops further as more (double/triple) constraints are demanded. Models frequently exhibit a 10–20% drop in instruction compliance relative to unconstrained scenarios, with degradation intensifying as constraints compound in number or complexity.

The paper formalizes that extended CoT generations enlarge the gap between the occurrence of instructions and the ultimate response, thereby diluting instruction-following. When longer, more complex solutions are generated, models appear systematically less likely to obey user-imposed constraints.

4. Training Strategies and Intervention Effects

MathIF evaluates models trained under various paradigms, such as:

• Supervised fine-tuning (SFT) on distilled long-CoT reasoning traces: Enhances reasoning ability but exacerbates the obedience deficit.
• Reinforcement learning (RL) variants (with or without SFT initialization): Both can lead to reduced constraint adherence, especially as RL reward design focuses solely on solution correctness or long reasoning chains.

Notably, simple interventions such as repeating the instruction at the end of the reasoning process partially recover both hard and soft accuracy, albeit at some cost to reasoning performance. Truncating chains-of-thought (thus controlling maximum completion length) also boosts obedience, again trading off some mathematical accuracy—highlighting an inescapable, empirically validated trade-off.

5. Comparative Positioning and Theoretical Significance

MathIF distinguishes itself from prior mathematical reasoning benchmarks—which typically concentrate on answer correctness or intermediate reasoning trace quality—by making controllability under instruction the primary evaluation axis. This focus exposes limitations of existing LLM training paradigms, particularly as recent models are increasingly optimized for open-ended mathematical reasoning at the expense of fine-grained obedience.

In contrast, benchmarks such as LILA (Mishra et al., 2022) emphasize multidimensional mathematical skill and explainable (program-based) solutions, but do not directly quantify instructional controllability. The MathIF framework complements these efforts by establishing the critical importance of instruction-following as a metric co-equally important to raw reasoning prowess for real-world deployment.

6. Applications and Practical Implications

The findings from MathIF carry several significant implications for both research and application:

• Alignment and Safety: LLMs that sacrifice instruction-following for intelligence may pose safety and reliability risks in contexts (education, automated assessment, decision support) where user instructions are paramount.
• Instruction-Aware Model Development: Results motivate new lines of research into instruction-aware reasoning models—employing RL reward schemes that explicitly encode constraint obedience or architectural modifications (such as late-stage constraint revalidation layers).
• Evaluation and Benchmarking: MathIF provides an openly available code and data resource (https://github.com/TingchenFu/MathIF) for the community to reproduce, extend, or refine the benchmark for future models and training strategies.

7. Future Research Directions

The MathIF Benchmark sets the stage for ongoing advancement in instruction-following LLMs. Potential directions include:

• Development of hybrid training pipelines balancing chain-of-thought excellence with explicit instruction-controllability objectives, potentially via compositional RL rewards or auxiliary constraint-focused loss terms.
• Architectural innovations that better preserve user intent across long reasoning traces—such as memory mechanisms for constraint recall or modular answer formatting stages.
• Expanded constraint taxonomies (multilingual, cultural, or discipline-specific nuances) and integration with real-world instructional use-cases.
• Continuous tracking of trade-offs between task performance and obedience to systematically inform both dataset curation and model evaluation in mathematical AI.

In summary, the MathIF Benchmark provides a rigorous, constraint-centric lens for evaluating how mathematical reasoning prowess in LLMs interacts with, and sometimes impedes, adherence to user instructions. Its empirical findings signal both a fundamental challenge for model alignment and a roadmap for more instruction-aware LLM development in mathematical domains.