Calibration of audit detectors across other multiple-choice formats

Determine whether the audit detectors remain calibrated across multiple-choice benchmarks with numbers of answer options other than the four-option MMStar format.

Background

The paper evaluates its audit layer primarily on MMStar, a four-option multiple-choice benchmark, and separately probes HallusionBench, a binary yes/no benchmark. On HallusionBench, the visual-gain ranking fails to recover the benchmark’s visual-dependent and visual-supplement design labels, with an AUC of 0.49, and the option-letter-based consensus machinery does not apply because yes/no answers do not parse as option letters.

The authors therefore identify a format boundary for the demonstrated calibration of the detectors. Although the binary probe provides evidence about one out-of-format case, calibration on other k-way multiple-choice benchmarks—those with a number of options different from four—has not yet been established and requires further measurement.