Consensus process for retiring deprecated benchmark components

Establish a formal, field-wide consensus process for retiring deprecated data-generating mechanisms, methods, and performance measures from living synthetic benchmarks, specifying governance procedures (e.g., voting mechanisms) that are appropriate to the benchmark’s organizational structure.

Background

Living synthetic benchmarks will inevitably accumulate outdated or practically irrelevant components over time. To maintain clarity of reporting and conserve computational resources, such components may need to be removed.

The paper notes that deciding how to reach consensus on deprecating components depends on how a benchmark is organized and is currently unresolved, implying the need for defined governance procedures—potentially including voting or other decision mechanisms—to guide removals in a neutral, transparent manner.