Statistical analysis of run-to-run variation

Establish, through repeated executor runs with different random seeds, confidence intervals and a statistical analysis of HSAD performance comparisons and reference points.

Background

The reported experiments use a single evaluation run per condition, although the executor's score on an individual input can vary between runs. The paper notes that repeated runs with different seeds could provide confidence intervals and a more reliable assessment of the reported lifts.

The authors specifically identify repeated-seed analysis as future work because averaging multiple runs or estimating pass@k would substantially increase the cost of arms collection, which is already the most expensive stage of the pipeline.