Purpose and scope of future BDA benchmarks

Determine whether operational building damage assessment benchmarks should evaluate only fully automated artificial-intelligence systems or the overall quality of service delivery, including limited manual corrections within constrained time budgets.

Background

The chapter distinguishes scientifically oriented challenges, which primarily aim to advance methods, from operational benchmarks intended to evaluate potential service providers. These objectives imply different task formulations, evaluation criteria, and organizational designs.

The unresolved issue is whether an operational benchmark should measure the isolated performance of an automated model or assess the complete service delivered to emergency-response users, where limited human intervention may be acceptable under time constraints.

References

Finally, there is an open question regarding the nature of future benchmarks and challenges. One option is to organize scientifically oriented challenges aimed at advancing methods and driving engagement with the research community, typically over extended time frames of several months. Another option is to design more operational benchmarks focused on evaluating potential service providers. In this latter case, it remains to be determined whether the evaluation should focus solely on the performance of fully automated AI systems or instead assess the overall quality of service delivery, including the integration of limited manual corrections within constrained time budgets.

— Revisiting the Task of Building Damage Assessment by Satellite Imaging  (2610.13857 - Cioppa et al., 9 Oct 2026) in Section 2, subsection “Takeaway messages, perspectives, and open questions,” paragraph “Implications for benchmarking”