Evaluation scores and dataset bias in salient object detection benchmarking

Develop evaluation measures and benchmarking protocols for salient object detection that address and mitigate dataset bias (including center bias and annotation subjectivity) and provide reliable, comparable scoring across models and datasets. Establish metrics that better reflect segmentation quality and model behavior than current PR/ROC/AUC/F-Measure variants when datasets have inherent biases.

Background

Throughout the paper, the authors discuss limitations of existing evaluation metrics, including discrepancies between PR and ROC curves, and issues with MAE and classic F-Measure, prompting use of alternatives like the weighted Fβ measure. They also analyze center bias and scene complexity, showing how dataset characteristics can skew results and rankings.

In the abstract, the authors explicitly identify evaluation scores and dataset bias as open problems and indicate the need for improved solutions to enable fairer, more informative benchmarking of salient object detection and segmentation methods.