Predicting method rankings on new datasets

Determine how to predict the rank of a computer-vision method on a new dataset from its ranking on previously evaluated datasets.

Background

The evaluation discussion distinguishes performance measurement on a fixed dataset from selecting or ranking methods for deployment in a new situation. The authors note that a ranking obtained on one evaluation set need not remain valid on another dataset.

This creates an unresolved methodological problem for building damage assessment and computer vision more broadly: existing evaluation results do not provide a reliable way to anticipate how methods will rank under new data distributions.

References

Overall, predicting the rank of a method for a new dataset is a problem that, to our knowledge, has not been studied in the scientific literature.

— Revisiting the Task of Building Damage Assessment by Satellite Imaging  (2610.13857 - Cioppa et al., 9 Oct 2026) in Section 4, subsection “Theoretical elements of an evaluation procedure,” paragraph “Generalization and transferability”