MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofs (2410.13502v2)

Abstract: LLMs can solve arithmetic word problems with high accuracy, but little is known about how well they generalize to problems that are more complex than the ones on which they have been trained. Empirical investigations of such questions are impeded by two major flaws of current evaluations: (i) much of the evaluation data is contaminated, in the sense that it has already been seen during training, and (ii) benchmark datasets do not capture how problem proofs may be arbitrarily complex in various ways. As a step towards addressing these issues, we present a framework for evaluating LLMs on problems with arbitrarily complex arithmetic proofs, called MathGAP. MathGAP generates problems that follow fixed proof specifications -- along with chain-of-thought reasoning annotations -- enabling systematic studies on generalization with respect to arithmetic proof complexity. We apply MathGAP to analyze how in-context learning interacts with generalization to problems that have more complex proofs. We find that among the models tested, most show a significant decrease in performance as proofs get deeper and wider. This effect is more pronounced in complex, nonlinear proof structures, which are challenging even for GPT-4o. Surprisingly, providing in-context examples from the same distribution as the test set is not always beneficial for performance. In particular, zero-shot prompting as well as demonstrating a diverse range of examples that are less complex than the test data sometimes yield similar or higher accuracies.

The paper introduces a framework called MathGAP. Its main goal is to create a systematic way to test how well LLMs can solve arithmetic word problems, especially when the reasoning (or “proof”) involved becomes increasingly complex.

Background and Relevance

The authors focus on a common challenge for LLMs. Even though models can solve many arithmetic problems, it is not clear how they perform when problems require several steps of reasoning that are more complex than the basic examples they might have seen during training. To address this, MathGAP builds on the idea of representing math word problems as a series of logical statements that can be connected in a “proof tree.”

Logical Forms and Proof Trees

• Logical Forms: Each sentence of a math word problem is encoded as a logical form. These logical forms capture essential facts such as the quantity an agent holds (for example, “Alice has 5 apples”) or relationships between different agents (like one having more or fewer items than another).
• Proof Trees: Once every sentence is represented, the paper organizes these logical forms into a proof tree. This tree shows the order in which statements are combined using simple inference rules until the final answer is reached. The structure of the tree (its depth, width, and the ordering of steps) exactly describes how complex the reasoning is.

Controlling Complexity

MathGAP generates synthetic arithmetic problems by deciding on the structure of the proof tree. This means that researchers can control aspects such as:

• Depth: How many layers of reasoning (or how many steps) are needed to reach the answer.
• Width: The number of basic statements or sentences the problem starts with.
• Order: The natural order in which the reasoning steps are written can also affect how the model understands the problem.

By adjusting these parameters, the framework can generate problems that range from very simple to arbitrarily complex. This is particularly useful because it highlights whether models can generalize when they only see simpler examples during training or in their context prompts.

Experiments and Findings

The authors performed various experiments using different types of problems:

• Linear Problems: Where the reasoning follows one straight path, making it somewhat simpler.
• Nonlinear Problems: Where the reasoning involves combining information from multiple steps, making it more difficult.
• Permutations: Changing the order of sentences to see if the model’s performance is affected by where certain pieces of information appear.

They compared the performance of several models, including versions of GPT-3.5, GPT-4, and other LLMs. The main findings were:

• Decrease in Accuracy with Complexity: All models showed lower performance as the number of reasoning steps (depth) or the number of starting facts (width) increased.
• Sensitivity to Order: Changing the order of sentences in the problem sometimes made a significant difference, which means the way a problem is presented can matter a lot.
• In-Context Examples: Sometimes providing examples of simpler problems (with solutions) helps; other times, simply asking the model to solve the problem without examples (zero-shot prompting) worked just as well or even better.

Why This Research Is Important

MathGAP is important because it provides a way to understand and measure the limits of current LLMs in a very detailed and controlled manner. By generating a wide range of problems that are not part of public benchmarks, researchers can be sure that the models are not simply recalling training data but are genuinely reasoning over new, unseen information. This work not only shows where the models struggle but also sets the stage for developing new methods that can improve their problem-solving and reasoning abilities.

Key Takeaways

• Math word problems are translated into logical forms and chained together into a proof tree that outlines the reasoning steps.
• By controlling the structure of the proof tree, the framework can generate problems with exactly defined levels of complexity.
• Experiments reveal that as the complexity of the reasoning increases, current LLMs generally perform worse.
• The framework highlights that providing in-context examples does not always lead to better performance, emphasizing that the diversity of examples matters.

In summary, the MathGAP framework presents a clear and systematic method for evaluating how well LLMs can handle increasingly complex arithmetic reasoning tasks. This work is a step toward better understanding the reasoning abilities of modern LLMs and guiding future improvements in their design and training.