Sharper stability assumptions for classification test-error estimation

Identify sharper assumptions on the training algorithm, weaker than uniform stability, under which the leave-a-window-out estimator consistently estimates classification test error for dependent data.

Background

The paper proves its classification test-error guarantee under a uniform stability assumption on the learning algorithm. Uniform stability controls the effect of deleting observations and yields the bounded-differences property required by the general theorem.

Numerical experiments with the k-nearest-neighbor algorithm suggest that uniform stability is sufficient but not necessary: the leave-a-window-out estimator can perform well even though k-nearest neighbors does not satisfy the stated stability condition. The authors therefore leave unresolved which weaker properties of a training algorithm are sufficient for the estimator’s success.