PHREEQC-MCQ-200: A Diagnostic Benchmark for Tool-Augmented Scientific Simulator Agents

Abstract: LLM agents are increasingly connected to scientific software, yet it remains unclear when tool access makes scientific computation more reliable rather than merely more complex. We introduce PHREEQC-MCQ-200, a benchmark for evaluating tool-augmented agents on deterministic aqueous-geochemistry simulations. The benchmark contains 200 multiple-choice questions derived from 21 validated PHREEQC scenarios, requiring agents to construct simulator inputs, execute PHREEQC, inspect structured outputs, and commit to final answers. Across multiple frontier and mid-tier model families, simulator access substantially improves aggregate accuracy, confirming that grounded execution is necessary for many scientific-computation tasks. However, the gains are not monotonic: tool-augmented agents also lose items they answered correctly without tools, revealing regressions that average accuracy alone hides. We further show that output-access protocol matters. A table-of-contents interface can reduce token cost while preserving or improving accuracy for stronger models, but it degrades performance for mid-tier models that cannot reliably navigate structured simulator outputs. PHREEQC-MCQ-200 therefore frames scientific tool use as an end-to-end diagnostic problem rather than a simple tool-calling capability. We argue that evaluations of scientific agents should report not only accuracy, but also item-level retention, output-access sensitivity, trajectory failures, and where the computation chain breaks.

• The paper introduces a diagnostic benchmark using 200 MCQs to assess tool-augmented LLM agents in geochemical simulations.
• It contrasts output-access protocols (TOC vs. Raw100k) to reveal non-monotonic effects on accuracy and resource consumption.
• Findings advocate co-designing tool interfaces with agent capabilities to improve reliability and manage token cost in complex tasks.

PHREEQC-MCQ-200: Diagnostic Benchmarking of Tool-Augmented Scientific Simulator Agents

Introduction and Motivation

PHREEQC-MCQ-200 introduces a diagnostic framework for assessing LLM agent reliability when interfaced with deterministic scientific simulators, focusing on grounded geochemical computation rather than mere tool-calling. The PHREEQC simulator, widely used in geochemistry, enables reproducible numerical experiments. The benchmark comprises 200 expert-validated multiple-choice items spanning 21 PHREEQC simulation scenarios, requiring not only generation and execution of simulator input but effective extraction and mapping of structured results to answer selections.

The study's principal aims are to distinguish when and how tool augmentation surpasses mental reasoning (including chain-of-thought), diagnose non-monotonic effects of tool use on agent performance, measure protocol-induced resource trade-offs, and localize sources of agent unreliability. The authors advocate that agent evaluation requires finer resolution than aggregate accuracy, emphasizing item-level retention, protocol ablation, and chain diagnostics.

Benchmark Design and Agent Harness

PHREEQC-MCQ-200 tasks require agents to generate PHREEQC input, invoke deterministic simulation with a pinned database, parse its section-structured output, and commit to a single multiple-choice answer. The MCQ format ensures scalable, automatable grading while maintaining scenario uniqueness per item. The harness gives the agent structured tools: file write/read/list and simulator execution, exposed in an isolated workspace with a strict 24-step limit per item.

Two output-access protocols are contrasted. Agent-TOC provides section-indexed access, returning a table-of-contents for structured retrieval, while Agent-Raw100k supplies the first and last 100k characters of the simulation's output inline. The protocols isolate the performance and cost effects of context truncation versus navigational abstraction.

Figure 1: The agent harness structure under the Agent-TOC protocol, delineating PHREEQC input synthesis, simulator execution, and navigational retrieval via a section-level TOC.

Quantitative Results and Key Findings

Chain-of-Thought Reasoning vs. Simulator Execution

Chain-of-thought (CoT) prompting produces marginal improvements over direct approaches (±8 pp), despite inflating output by 29–140× with no comparable accuracy gain. Strong numerical evidence supports that for PHREEQC-MCQ-200, computational grounding is essential; reasoning-token expansion does not approach the effect size of true tool access.

Simulator Access: Gains, Loss, and Non-monotonic Effects

Tool-augmented agents achieve substantial net accuracy increases: top-tier models (Claude Opus 4.6, Sonnet 4.6, GPT-5.4) improve by 37.5–41.5 pp, mid-tier (GPT-5.2, GPT-5.1) by 15–19.5 pp. However, these gains are not strictly additive. Across models, 10–32 items correctly answered without tool-use are lost under tool-augmentation, yielding retention rates of just 56.4–86.5%. This non-monotonicity demonstrates that tool integration introduces risks not captured by net accuracy, including regression on previously solvable items. Below-mid-tier models can regress under tool access (Gemini 3 Flash: 17 pp drop relative to CoT), primarily due to step-budget exhaustion induced by navigational overhead.

Output-Access Protocols: Capability-Dependent Trade-offs

The effectiveness of structured output access is model-dependent. For top-tier models, the TOC protocol is Pareto-optimal, achieving equivalent or superior accuracy (by 0.5–2.5 pp) at an 11–57% reduction in input token consumption. For mid-tier models, the TOC interface exacts a penalty (7.5–9.5 pp accuracy loss), with raw truncation outperforming structured retrieval. This capability-protocol interaction crosses vendor lines; GPT-5.4 aligns with Anthropic's top-tier performance, while lower GPT-5.x variants align with mid-tier behavior. Simply expanding the raw window to 500k does not alleviate degradation on long-output items and instead elevates token cost, underscoring the structural challenge of large-output navigation.

Figure 2: Accuracy and total token cost for TOC versus Raw100k output-access across agent architectures, highlighting the capability-dependent nature of navigational benefits and trade-offs.

Diagnostic Probes and Failure Analysis

Retention and Regression Decomposition

Item-level tracking reveals that gains from tool use are always offset by specific losses. In particular, GPT family models are less stable (GPT-5.2 loses 43.6% direct-correct vs. Sonnet's 13.5%), and lost-item sets are non-overlapping across architectures, indicating that regressions stem from unique model-tool interactions rather than inherent item-level pitfalls.

Trajectory and Protocol-Specific Failures

Trajectory inspection shows that top-tier models tend toward persistent, high-step retry loops when failing (median 24 steps on failure for Opus/Sonnet), while mid-tier models fail via rapid, premature commitments without leveraging additional steps. Under TOC, below-mid-tier models (Gemini 3 Flash) predominantly fail by non-commitment due to navigational step inflation. Failure taxonomy highlights that for lower-capability models, the dominant error is navigational or commit failure, while for top-tier agents, failures cluster around subtle simulator-input flaws or decision-chain interruptions.

Practical Implications and Theoretical Outlook

These results signal that the design of tool-access protocol must account for model-specific navigation and reasoning capabilities; uniform protocol deployment across heterogeneous model fleets is suboptimal. The authors' diagnostic methodology, which tracks retention, protocol sensitivity, trajectory cost, and error root causes, is mandatory for robust benchmarking of agentic scientific workflows. Benchmark discriminative power is concentrated in particular scenario types and output regimes, pointing toward the need for targeted scenario construction in next-generation diagnostics.

The findings imply that for robust, scalable scientific inference, architectural advances must address navigation and extraction in structured outputs and optimize the interaction between simulator context presentation and agent memory constraints. Token efficiency, step budgeting, and error recovery mechanisms are central to future protocol and agent design. Structured-retrieval interfaces (like TOC) are broadly applicable across scientific simulators with sectioned outputs, but extracting their benefits in mid-tier models may require advances in tool-use-coupled cognition.

Conclusion

PHREEQC-MCQ-200 reframes tool-augmented agent evaluation as a diagnostic workflow rather than a simple accuracy measurement. Simulator grounding is indispensable for scientific computation, but tool use can silently regress previously reliable solutions. Output-access protocols must be co-designed with agent capability, and aggregate accuracy obscures crucial reliability and resource trade-offs. The benchmark and diagnostic analysis set a new standard for precision evaluation of LLM science agents, offering insights directly relevant to scalable deployment and iterative improvement of such systems.

(2607.00436)