Net effect of task selection on measured productivity

Quantify the net effect on the estimated percentage change in issue completion time of selecting smaller, better-scoped issues and excluding non-programming tasks in the randomized controlled trial measuring AI tooling’s impact on experienced open-source developers.

Background

The paper asks developers to contribute real issues but encourages breaking larger issues into shorter tasks and excludes non-programming activities such as design discussions or code review. The authors note these selection pressures may bias the task distribution toward better-scoped issues that could differentially affect humans and AI systems.

They argue this selection could plausibly either help AI (by simplifying and scoping tasks) or help experts disproportionately (by aligning tasks with their strengths), making the net impact on measured productivity ambiguous.