Determine whether expert calculation errors change SFBench gold labels

Determine whether the documented errors in several SFBench expert calculations—including swapped component identities, incorrect composition labeling, omitted phases, and mismatched composition totals—change the corresponding gold feasibility labels.

Background

The paper identifies problems in expert code and explanations for several materials-science claims, including swapped Mg and Zn relative to the claim, an eight-atom cell labeled as 4 at.% rather than 12.5 at.%, omitted bcc and liquid phases in Li–Al–Cu calculations, and an eutectic composition whose components sum to only 90 at.%. The authors do not establish whether these issues affect the benchmark’s gold labels, leaving the reliability of those labels unresolved for the affected claims.

References

We have not established whether any of these problems changes a gold label.

— Same Claim, Different Interpretations: How LLM Agents Resolve Ambiguity in Scientific Claims  (2610.20457 - Rao et al., 14 Oct 2026) in Appendix F.1, Agent and Expert Calculations