Determine the maximum attainable PIT-step localization accuracy

Determine the highest localization accuracy that any predictor can attain when evaluated against PIT-step labels generated from the paper’s finite-rollout labeling protocol.

Background

PIT steps are assigned using a small number of Monte Carlo rollouts at each evaluated step, and the two rollout collections can assign different failure types to overlapping traces. These factors introduce uncertainty into the reference labels themselves. Although the tested probes generally fail to outperform the trace-length rule, the authors state that the maximum achievable score under these noisy labels is unknown.