Develop an empirical evidence base for learning with repository agents

Establish whether and under what conditions learning with repository agents develops software-engineering competence, including task decomposition, permission design, evaluation quality, review behaviour, recovery, and security reasoning, while separating educational outcomes from agent productivity.

Background

The paper distinguishes repository-agent capability from educational benefit. Repository agents can make multi-file changes, execute tests, use tools, and pursue software tasks, but their ability to improve student learning has not been established.

The proposed research should examine agent supervision as an educational practice rather than merely measuring productivity. Relevant outcomes include students’ ability to decompose tasks, establish permission boundaries, design evaluations, review agent-produced changes, recover from failures, and reason about security. The paper also identifies autonomy, tool access, memory, execution privileges, and human intervention points as important experimental variables.

References

There is not yet a mature evidence base on learning with repository agents.

— Judgment-Centred Software Engineering Education: A Post-Hype Review and Framework for AI-Augmented Learning  (2609.29473 - Mahmoud, 24 Sep 2026) in Section 10, “Research agenda for durable competence,” subsection “Agent supervision”

The construct remains a research agenda rather than a validated scalar metric.

— Judgment-Centred Software Engineering Education: A Post-Hype Review and Framework for AI-Augmented Learning  (2609.29473 - Mahmoud, 24 Sep 2026) in Section 7.3, “Assessment as a debt probe”