Budget allocation for continual validation

Determine how to allocate budgets among calibration, admission, and regression checks for continual validation of self-evolving harnesses under distribution and evaluator drift.

Background

The paper argues that reliable admission of harness revisions requires fresh, independent tasks and that evaluator changes necessitate independent calibration of the discrepancy between evaluator loss and target risk. Calibration, candidate admission, and regression testing therefore compete for finite evaluation resources. The paper identifies the principled allocation of these resources as unresolved.