Per-model ordering of answer scales

Determine the per-model ordering of the verbal-binary, numeric-binary, integer ordinal, and floating-point answer scales for training-free video anomaly detection, while accounting for the confounding effects of label wording and tokenization when the number of answer levels changes.

Background

The paper evaluates several answer scales—Yes/No, numeric 0–1, integer 0–5 and 0–9, and floating-point 0–1, 0–5, and 0–9—in combination with generated and probability-based readouts for four frozen vision–LLMs. Although probability readout generally outperforms generated readout, the relative performance of answer scales is not consistent for every individual model and benchmark.

The authors note that changing the number of answer levels also changes the wording and tokenization of the response labels. Consequently, the scale comparison does not isolate granularity alone, and the per-model preference among scales remains unresolved except for the cases quantified in the bootstrap analysis of Section C. Resolving this problem requires determining the model-specific scale ordering while separating granularity from verbalizer and tokenization effects.