Determine the reliability of JET’s diagnostic-feedback gains

Determine whether the performance gains attributed to diagnostic feedback in Judge-Guided Evolution at Test Time (JET) persist under broader campaign replication, given that limited replication leaves the observed diagnostic-feedback differences uncertain.

Background

The primary experiments use five independent campaigns per condition. The paper reports gains from the evolved judge and analyzes textual diagnostics as an additional source of guidance for target-side program rewrites.

The conclusion cautions that limited campaign replication makes smaller differences, including diagnostic-feedback gains, uncertain. Additional independent campaigns are therefore needed to establish whether the diagnostic-feedback effect is robust rather than an artifact of limited replication.