Strategic Test Selection for TDD-Guided LLM Code Generation

Determine a principled strategy for selecting a small, effective subset of test cases to guide large language models in repository-level code generation under the Test-Driven Development paradigm, subject to constraints of limited prompt context windows and computational budgets.

Background

The paper argues that Test-Driven Development (TDD) is particularly important for repository-level code generation with LLMs because tests serve as executable specifications that clarify intended behavior beyond natural language descriptions. However, real-world repositories often require validating complex functions against many diverse test cases, and providing all tests to an LLM is infeasible due to context window and efficiency constraints.

In this context, the authors explicitly state that how to strategically select a small yet effective subset of tests remains an open question. They later propose the Test Harness Mechanism (THM) as a concrete approach, but the foundational problem of devising principled, general strategies for test selection under TDD with limited resources is highlighted as a critical open issue.

References

How to strategically select tests to guide an LLM within a limited context window and computational budget remains a critical open question.

— TENET: Leveraging Tests Beyond Validation for Code Generation  (2509.24148 - Hu et al., 29 Sep 2025) in Section 1 (Introduction), paragraph following Figure 1

A limitation is that Task-CoEvolve fixes how many tasks each candidate is evaluated on before seeing any of its results. It therefore cannot stop early on a candidate that is already clearly worse, nor evaluate more tasks when two candidates are hard to tell apart. Deciding this number during evaluation is a natural next step and we leave it to future work.

— Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection  (2608.20169 - Miyai et al., 20 Aug 2026) in Section 6, Conclusion, Limitations and Future Work