Effect of demonstrations on domain-specific problem solving for SWE-agent

Determine whether including demonstration trajectories in the SWE-agent prompt (via the Demonstration Template that shows a solved trajectory) actually improves agents’ domain-specific problem solving capabilities in software engineering tasks, as opposed to primarily teaching proper command formatting and interface usage.

Background

SWE-agent optionally prepends a demonstration trajectory to the prompt to illustrate correct interface use. Ablations indicate small benefits, and the authors hypothesize demonstrations may mainly help with formatting and command usage rather than impart domain-specific expertise.

However, the authors explicitly state uncertainty about whether demonstrations help agents understand nuanced, domain-specific problem solving, leaving open the question of demonstrations’ true contribution beyond interface familiarization.