Measure independent annotation agreement

Measure independent inter-annotator agreement for the tempo, tonality, time-signature, and structural-interval annotations underlying the Music Understanding Benchmark.

Background

The benchmark reuses labels from an existing annotation workflow, but the paper notes that the independence and agreement of the annotators have not been established. Submission counts and revision histories do not constitute agreement across independent annotators. An independent annotation study, ideally involving stratified double annotation and adjudication, is needed to support stronger claims about label reliability.

References

MUB instead derives global labels and full-recording intervals from an existing annotation workflow; the independence and agreement of its annotators remain to be measured.

— MUB: An Identity-Masked Benchmark for Music Understanding in Multimodal Language Models  (2610.22989 - Liu et al., 16 Oct 2026) in Section 2.1, “Music Question Answering and Human Curation”