Effect of the Salt treatment on referee acceptance

Determine whether the dieted Salt-method treatment changes what a machine referee accepts, rather than merely changing the monetary cost of producing the artifact, relative to a plain agent across the benchmark's Lean, Verus, and Rust systems populations.

Background

The paper reports cost differences for the dieted Salt-method treatment but explicitly states that these results do not establish a capability difference. On S2-Lean, the treatment matched the control's solved-problem set at Sonnet and was nearly identical at Opus; on S2-Rust and stage C, the plain arm was at or near the available performance ceiling; and on the S3-Systems population, the treatment cost more while correctness was not assessed during the scored runs. A subsequent post-hoc correctness pass also failed to separate the arms.

The unresolved problem is therefore to obtain a benchmark comparison with sufficient headroom, an appropriate referee verdict for each cell, and a design that can distinguish an effect on referee acceptance from an effect on price or from artifacts introduced by budget caps and other measurement limitations. The paper identifies lower-tier hard-band tests, a repaired CLEVER variant, and a more complete systems population as possible ways to address it.

References

Nothing in this record shows an effect of the treatment on what a referee accepts. On S2-Lean the treatment arm matched the control's set at Sonnet and was within one problem of it at Opus, with the placebo moving with the tier; the triage attributes most of the shared failures to the reference oracle. On S2-Rust and on stage~C the plain arm is at or near the ceiling of every band the campaign can afford, so a comparison has no room. On S3-Systems the treatment arm cost more on all five problems under the registered sign test, and that is a difference in price carrying the four qualifiers of Section~\ref{sec:s3}, on a run in which no cell was checked for correctness as it went; the post-hoc pass of Section~\ref{sec:posthoc} checked the surviving cells afterwards and did not separate the arms. A price difference is not a capability difference, and nothing in that design licenses reading one from the other. The question is open.

— SaltBench: A Referee-Gated Protocol for Measuring Method Effects in Machine-Checked Software Work  (2609.11076 - Hickey, 10 Sep 2026) in Section 5, “The treatment question, open” (Section \ref{sec:open})