Audit the reliability of the B3 command-generation metric

Determine whether the VectraYX series’ lenient B3 command-generation metric accurately measures command-generation capability or, like the B4 tool-use metric, misestimates performance through keyword or substring matching.

Background

The paper identifies a false positive in the B4 tool-use metric because the scorer can credit a tool name appearing in prose without verifying a well-formed tool call. The 1B nonetheless scores higher than the 600M on B3 command generation under the same general benchmark suite.

Because B3 has not undergone the strict structural audit applied to B4, the reported advantage may reflect a comparable harness artifact. The paper explicitly leaves unresolved whether B3’s lenient matching misestimates one or both models, and in which direction.

References

Given that we discredit the same harness's B4 for the 1B, the symmetric possibility---that B3's lenient matching also misestimates one or both models, in either direction---is open.

— Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models  (2610.02142 - Santillana, 1 Oct 2026) in Section 3.4, paragraph “L8: The 1B leads on B3, and we have not audited B3”