Evaluating SageMath-Augmented LLM Agents for Computational and Experimental Mathematics

Abstract: Recent advances in AI for Mathematics have focused largely on autoformalization and theorem proving, leaving the role of Computer Algebra Systems (CAS) in agentic LLM workflows underexplored. We propose a ReAct-style agentic setup that combines LLM reasoning with verifiable feedback from SageMath, together with Context7 for the up-to-date documentation. We evaluate this agentic setup across frontier models for solving research-level mathematical problems from the RealMath benchmark in a setting that emulates a computational-mathematics research loop. We also propose a refinement to the RealMath benchmark by introducing a multi-step post-processing procedure and a multi-stage validation pipeline, both of which improve the quality and reliability of the extracted problem set. Our experiments reveal substantial performance gains from SageMath access across all evaluated models on +9.7~pp on average, the gains range from 1.5~pp to 27.8~pp and narrow the gap between open-weight and closed models. Qwen~3.7-Max benefits from SageMath the most, while GPT-5.5 achieves the highest solve rate of $75.2\%$ and the lowest token usage among tool-enabled configurations. Our findings suggest that CAS-augmented agents represent a promising direction for assisting mathematicians in computational exploration, and we believe that this work is a step towards automated conjecture discovery. The project repository is available online.

• The paper demonstrates that augmenting LLM agents with SageMath yields an average solve rate improvement of +9.7 percentage points across 15 models.
• It outlines a ReAct-style agent pipeline that leverages iterative SageMath feedback, revealing significant efficiency gains and tradeoffs in token usage.
• The study identifies domain-specific performance boosts—especially in combinatorics and classical analysis—while also highlighting persistent tool limitations.

Evaluating SageMath-Augmented LLM Agents for Computational and Experimental Mathematics

Motivation and Background

The integration of LLMs with deterministic tool backends has catalyzed a neuro-symbolic paradigm for mathematical reasoning. While substantial advances have occurred in the synthesis of LLM-based reasoning with formal proof assistants, the computational mathematics research workflow—heavily reliant on Computer Algebra Systems (CAS) for interactive exploration and executable verification—remains comparatively underexplored. "Evaluating SageMath-Augmented LLM Agents for Computational and Experimental Mathematics" (2607.06820) addresses this gap by systematically analyzing the impact of verifiable symbolic feedback—via SageMath—on the performance of LLM-based agents solving research-level mathematical problems.

Framework and Methodology

The study implements a ReAct-style agentic pipeline in which an LLM iteratively interacts with SageMath for symbolic computation and Context7 for up-to-date documentation, mimicking the experimental workflows of mathematicians.

Figure 1: Overview of the evaluation framework; a ReAct-style LLM agent leverages SageMath and Context7 iteratively, obtaining verifiable feedback and outputting answers validated by symbolic checking and LLM-as-a-Judge protocols.

The evaluation utilizes a curated subset (133 problems) of the RealMath benchmark, filtered for tasks executable and verifiable within SageMath. Each problem is processed under two matched conditions:

• Tool-free baseline: The LLM operates without access to external tools, relying exclusively on internal reasoning.
• Tool-augmented agentic setting: The LLM can call SageMath and Query Context7 throughout a multi-turn interaction loop. The external tool interface is strictly sandboxed, with constraints on compute resources and wall-clock time for each tool call.

Validation employs a hybrid pipeline: the model's output is first subjected to a symbolic equivalence check using SymPy; outputs that fail symbolic checking are then adjudicated by a panel of three state-of-the-art LLM judges (majority voting). This dual-stage ensures robustness both for numerical and symbolic answers.

Core Experimental Results

Solve Rate Improvements and Model Differentiation

Across 15 models, tool access produces a mean solve rate gain of +9.7 percentage points (pp). The range is non-uniform, from +1.5 pp (Kimi 2.7) up to +27.8 pp (Qwen 3.7-Max). Open-weight models accrue larger mean gains (+15.3 pp) compared to closed models (+6.5 pp), narrowing the traditionally wide performance gap.

Figure 2: Solve rates for each model in both settings, highlighting the effect of tool access across architectures.

GPT-5.5 attains the highest overall solve rate (75.2%) and is also the most token-efficient configuration among all tool-enabled models.

Accuracy-Cost Tradeoffs

Analyzing solve rate versus average token usage reveals that increased resource expenditure does not proportionally yield higher accuracy. For instance, while MiniMax M3 uses approximately 556k tokens per problem in the agentic setup, it underperforms GPT-5.5 (which uses only 34k tokens per problem) by 19.6 pp.

Figure 3: Accuracy versus token usage, showing significant variation in efficiency across models.

Efficiency on Universal Problems

When measuring token usage on the subset of problems solved by all models in a given setup, model ranking inverts from tool-free to tool-augmented runs. Open-weight models are more efficient for direct answering, but highly inefficient under agentic usage; closed models become preferable under tool access.

Figure 4: Per-problem token consumption on the universally solvable set—efficiency reverses between setups.

Problem-Type Specialization

Models generally perform better on numerical than on expression-type answers in the tool-free setup (mean gap up to 33 pp), but tool augmentation closes this disparity, especially for models initially weak in symbolic manipulation.

Domain-Specific Performance

Solve rate gains from tool access are most pronounced in combinatorics (+18.7 pp), classical analysis (+12.0 pp), and rings/algebras (+10.7 pp). Number theory problems show moderate improvement due to already strong tool-free baselines. Group theory and algebraic topology remain challenging despite extensive SageMath support, indicating limitations not attributable solely to tool access.

Figure 5: Category-wise solve rates with and without tools, showing heterogeneity in tool benefit across mathematical subdomains.

Error Modes and Agentic Dynamics

Execution failures are categorized by exception type, with TypeError, NameError, AttributeError, and SyntaxError accounting for 87% of all exceptions. Robust models—GPT-5.5, Opus variants, Fugu-Ultra—exhibit low failure rates and high post-failure recovery (up to 77%), whereas heavy tool users (e.g., Kimi, DeepSeek) saturate the tool-call budget and are dominated by repeated failures.

Figure 6: Distribution of Sage call failures by exception type and model, exposing stark behavioral differences.

Agentic runs are bimodal: strong models typically solve problems within 3–4 tool calls, using SageMath primarily for confirmation, while weaker models perform exhaustive, and often unproductive, searches that exhaust the tool budget.

Figure 7: Bimodal distribution of per-problem tool interactions across models; effectiveness is correlated with concise tool usage.

Case Study: Iterative Conjecture Formation

A detailed examination of the solution trajectory on a torsion order problem illustrates that SageMath-augmented LLMs can execute workflows characteristic of mathematical experimentation: computing invariants, recovering from code errors, generalizing by pattern extraction, and validating across parameter space. The model does not merely scan for direct answers but performs iterative, hypothesis-driven computation and generalization—a capability that is not widely supported by purely generative or proof-focused LLM workflows.

Theoretical and Practical Implications

The results establish that agentic LLM setups augmented with CAS significantly advance executable mathematical reasoning, especially in domains with mature symbolic software coverage. Such architectures lower the barrier for open-weight models in research-grade problem-solving and provide a substrate for future work on automated conjecturing and experimental discovery pipelines.

However, several structural limitations persist: rule-based symbolic equivalence checking is not universally robust to notational variation; tool gains derive from a complex interplay between prompt engineering, tool orchestration, and symbolic backend coverage; and many problem classes (e.g., those requiring deep theoretical insight) remain out of reach even for the strongest current agents.

Conclusion

This work rigorously evaluates the impact of integrating SageMath into LLM-based agentic mathematical reasoners. The findings demonstrate uniform but variable gains in accuracy across models, a shift in the efficiency landscape toward closed models for tool-augmented agentic use, and the emergence of authentic computational exploration behaviors in state-of-the-art models. The study substantiates CAS-augmented agents as promising collaborators for computational and experimental mathematics, highlighting crucial axes—such as robustness in error recovery and efficient hypothesis verification—that will direct future progress toward autonomous mathematical reasoning and discovery.