Cause of test flakiness for SWE-bench Lite instance astropy__astropy-6938

Ascertain the exact cause of the non-deterministic behavior of the unit test suite for the SWE-bench Lite instance astropy__astropy-6938 that is flaky on some machines but not others, and determine the environment and dependency conditions under which this flakiness occurs.

Background

In analyzing SWE-bench Lite, the authors identified widespread flakiness in test suites, with 34 problems exhibiting inconsistent pass/fail behavior and 30 of those affecting the dataset-provided correct solutions. One particular instance, astropy__astropy-6938, showed flakiness that varied by machine.

Although the SWE-bench authors could reproduce the issue, the paper’s authors could not, suggesting unresolved environment-specific factors. Their preliminary investigation points to unpinned dependency versions in the Docker environments used to run unit tests, but the precise cause remains to be determined.

References

An additional instance, astropy__astropy-6938, was flaky on some machines and not others. The authors of SWE-bench were able to reproduce the flakyness; however, we were unable to. Our prelimiary investigation indicates this specific issue is due to unpinned versions of dependencies in the docker environments that run the unit tests.

— Large Language Monkeys: Scaling Inference Compute with Repeated Sampling  (2407.21787 - Brown et al., 2024) in Appendix: SWE-bench Lite, Subsection “Test Suite Flakiness”

Three questions our data could not settle remain open: whether detection improves materially as the execution budget grows past the $\sim$170-rerun threshold of \citeauthor{gruber2021empirical}; whether stress-ng configurations tuned for Python's execution model, rather than inherited from Java and Android, recover the concurrency cases the default configuration misses; and whether running each test within its suite, rather than in isolation, restores part of the flakiness that did not reproduce here.

— Evaluating Shaker for Flaky Test Detection in Python Projects  (2609.25528 - Leal et al., 22 Sep 2026) in Section Conclusions