Find Open Problems
Find Open Problems
Search for open problems in our database
Submit a Problem
Submit a new open problem to our database (not available yet)
Dice Question Streamline Icon: https://streamlinehq.com

Impact of advanced elicitation strategies on developer productivity

Evaluate the effect in field settings of higher test-time compute and alternative elicitation strategies (e.g., multi-trajectory sampling, LLM judges, self-consistency, or agent scaffolding) on developer productivity when using AI tools.

References

We do not provide evidence about these elicitation strategies, as developers in our study typically use Cursor and web LLMs like chatGPT, so it remains unclear how much effect these strategies would have on developer productivity in the wild.

— Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (Becker et al., 12 Jul 2025) in Factor Analysis: Factors with an unclear effect on slowdown — Subsubsection “Suboptimal elicitation”