Generalization of evolved verification skills

Establish whether verification skills evolved from development-task grader feedback generalize beyond the evaluated long-horizon workspace task distributions and improve verification performance without development grader feedback.

Background

The paper studies self-evolution of VeriHarness’s verification-skill library using development-task failures and per-item grader feedback. Candidate libraries are selected on development performance and evaluated on held-out tasks, with experiments conducted on APEX-Agents and SpreadsheetBench 2 under fixed model, protocol, and tool settings.

The reported held-out improvements demonstrate benefit within the evaluated task distributions, but the paper does not establish whether the learned procedures transfer to substantially different task distributions or whether they can improve when development-time grader feedback is unavailable. These two unresolved issues concern the external validity and supervision requirements of the self-evolution approach.

References

The held-out comparisons show that the learned libraries help on unseen tasks within the evaluated splits; generalization beyond these task distributions, and improvement without development grader feedback, remain to be established.

— VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks  (2610.00972 - Zhang et al., 1 Oct 2026) in Appendix, Section  Self-Evolution Details, subsection “What the learned skills contain”

There is no held-out task split and no temporal separation between mechanism development and evaluation, so test-set overfitting cannot be ruled out.

— Mingbird: A Local-First Agent Harness Enabling Small Open Models to Complete Real Tasks  (2610.02001 - Wang et al., 1 Oct 2026) in Section 5, “Threats to Validity,” paragraph “Development--evaluation overlap”

Connected-project reuse leaves open whether a frozen capability library helps independent tasks.

— AgentEvolver: System-Wide Self-Evolution Through Task Execution  (2610.11613 - Zhang et al., 8 Oct 2026) in Section 4.2.3, “Cross-case findings and open questions,” p. 12