When Does More Correct Data Hurt? Insertion-Stability and the Limits of Dimension-Based Theory

Abstract: Adding data known to be correct ought to be safe. Not always. Larsen, Pabbaraju and Shetty model the failure with a monotone adversary, which reads an i.i.d. training sample and may append as many further examples as it likes, provided the target hypothesis labels them all. Mehrotra has since settled the cost, showing that for classes of VC dimension d >= 2 no learner can guarantee expected error better than Theta((d/n)log(en/d)), a logarithmic factor above the clean PAC rate. Because that rate is a worst case over all classes, it says nothing about which classes actually suffer the penalty, and the answer turns on the learner. We call a learner insertion-stable if feeding it more correctly labeled examples can only shrink the region where it errs. Such learners are immune to the adversary, since on any given sample the risk after insertions never exceeds the risk on the clean part alone, however much is added and however cleverly it is chosen. High- probability guarantees carry over unchanged, and because Closure is insertion-stable every intersection-closed class keeps its clean rate of E[Err] <= (21d+34)/n. Immunity is not something the classical dimensions can predict. Two classes can agree on VCdim = Ldim = 2 and still split, one at Theta(1/n) and the other at Theta(log(en)/n), while intervals have unbounded Littlestone dimension and are immune anyway. On Mehrotra's hard class we prove more than the failure of a single algorithm, showing that no monotone permutation-invariant compression scheme of any finite size attains the clean rate. The question is therefore not whether a class is hard, nor whether a learner is good, but whether the two suit each other. Given an insertion-stable learner that is optimal on clean data, correct additions are free, and without one the cost belongs to the class, so changing the learner will not avoid it.

• The paper introduces insertion-stability, showing that permutation-invariant learners whose error regions shrink under correctly labeled insertions inherit clean-sample risk almost surely, including all tail and high-probability guarantees.
• The paper proves that Closure is insertion-stable and gives intersection-closed classes explicit adversarial error bounds of at most (21d+34)/n in expectation and (21d+16 ln(3/δ))/n with probability 1−δ.
• The paper separates two classes with identical VC and Littlestone dimension 2: one retains a Θ(1/n) rate while Mehrotra’s hard class incurs Θ(log(en)/n), and no finite monotone compression scheme restores the clean rate there.

Adding correctly labeled training data should, intuitively, never hurt a learner. In the monotone adversary model of Larsen, Pabbaraju, and Shetty (Larsen et al., 5 Jan 2026), this intuition fails: an adversary that observes an i.i.d. clean sample and appends any number of correctly labeled examples—chosen adaptively and then shuffled into the pool—can force every learner over a class of VC dimension $d \geq 2$ to suffer expected error $\Theta\big((d/n)\log(en/d)\big)$, a logarithmic factor above the clean PAC rate, as established by Mehrotra (Mehrotra, 6 Aug 2026). The paper under review (2608.14020) asks which classes actually pay this penalty and answers with a learner-dependent criterion: insertion-stability. Its central claims are that insertion-stable learners are exactly immune to the adversary, that Closure is insertion-stable so all intersection-closed classes retain their clean rate, and that no function of VC or Littlestone dimension can predict the monotone-insertion rate. The main technical result is a negative one: on Mehrotra's hard class, no monotone permutation-invariant compression scheme of any finite size attains the clean rate.

The model and the reduction

The monotone adversary model proceeds in three steps. Nature draws a clean ordered sample from $D^n$; the adversary, seeing it, appends $m$ points labeled by the target $t^*$ (repeats and out-of-support points allowed); nature then applies an independent uniform permutation, so the learner receives one shuffled multiset in which clean and inserted occurrences are indistinguishable. Adaptivity is the entire source of difficulty: against an oblivious adversary the clean rate is already attainable (Larsen et al., 5 Jan 2026).

The paper's first contribution is a reduction, not a new condition. A learner is insertion-stable if it is permutation-invariant and superset-monotone: for correctly labeled multisets $S \subseteq T$, its error region under $T$ is contained in its error region under $S$. The insertion-stability lemma shows that for such a learner, the adversarial risk is dominated by the clean risk almost surely on a single coupling, for every distribution, every budget $m$, and every randomized adaptive adversary. The proof quarantines the i.i.d. assumption inside the clean sample and never invokes exchangeability or concentration on the corrupted one. Because the domination is almost sure rather than merely in expectation, all moments, tail probabilities, and high-probability guarantees transfer unchanged from the clean setting to the adversarial one. Superset-monotonicity also buys uniformity in $m$: the bound passes to the unbounded-budget supremum $\Theta\big((d/n)\log(en/d)\big)$0 without change, which matters because Mehrotra's lower-bound witness uses a budget growing rapidly with $\Theta\big((d/n)\log(en/d)\big)$1, against which budget-dependent guarantees such as $\Theta\big((d/n)\log(en/d)\big)$2 are vacuous.

The paper is careful about provenance: the monotonicity condition is equivalent, for permutation-invariant learners, to Hanneke's prefix-monotonicity for consistent monotone rules (Hanneke, 2015), so his log-free clean-sample bounds apply off the shelf. The contribution is recognizing that the adversary's output is a correctly labeled superset, so a property studied for i.i.d. prefixes controls a non-exchangeable adversarial sample pointwise.

Closure and the positive half

Closure, which predicts positive exactly on the intersection of the positive sets of all hypotheses consistent with the sample, is insertion-stable for every class—intersection-closedness is not needed for stability, only to make Closure a good learner. Repeated points do not change the version space; out-of-support insertions are still valid constraints; and negatively labeled insertions can only enlarge the predicted positive set, so negatives are weakly helpful. Composing stability with Hanneke's bound yields that every intersection-closed class of VC dimension $\Theta\big((d/n)\log(en/d)\big)$3 retains, against every randomized adaptive monotone adversary at every finite budget, the explicit guarantees $\Theta\big((d/n)\log(en/d)\big)$4 in expectation and $\Theta\big((d/n)\log(en/d)\big)$5 with probability $\Theta\big((d/n)\log(en/d)\big)$6—both free of logarithmic factors. A scheme-level version holds for any consistent, permutation-invariant, superset-monotone compression scheme with permutation-invariant reconstruction.

A pointed corollary: Mehrotra's bounded-budget converse, which pads a clean lower-bound class with repeated copies of a commonly negative point, is invisible to Closure on intersection-closed classes—both features of the attack (negative labels, exact repetition) are inert against it.

Dimension independence

The paper's sharpest separation uses two classes with identical VC and Littlestone dimension, both equal to 2:

• $\Theta\big((d/n)\log(en/d)\big)$7, the class labeling at most two points positive, is intersection-closed. Closure on it labels positive exactly the observed positives; the error region is the unseen positives, of which there are at most two, so no correctly labeled insertion can be simultaneously valid and unhelpful. The adversarial rate is $\Theta\big((d/n)\log(en/d)\big)$8, with the explicit constant $\Theta\big((d/n)\log(en/d)\big)$9.
• $D^n$0, Mehrotra's Littlestone-two class built from projective-plane checker coordinates over primes, has adversarial rate $D^n$1 (Mehrotra, 6 Aug 2026). Its mechanism is that the adversary can pad two candidate targets to the identical multiset, leaving the posterior balanced on a point of test mass $D^n$2.

Thus two classes agreeing on $D^n$3 have rates differing by a $D^n$4 factor, so no function of the classical dimensions determines the monotone-insertion rate. The separation runs the other way too: intervals on $D^n$5 have unbounded Littlestone dimension (growing as $D^n$6 on a domain of $D^n$7 points) yet are intersection-closed and retain the clean rate. Bounded Littlestone dimension is neither necessary nor sufficient for immunity. The paper verified the combinatorial claims computationally, including that $D^n$8 is not intersection-closed—an essential fact, since intersection-closedness would have contradicted Mehrotra's lower bound.

A corollary worth stating precisely: on $D^n$9, no learner is both insertion-stable and clean-rate-optimal. The stronger claim that no insertion-stable learner exists would be false—the constant learner is trivially insertion-stable on every class—so insertion-stability constrains how the error region moves, never how small it is.

No monotone compression scheme certifies $m$0

The main technical result strengthens the barrier from single algorithms to a family: there is no finite $m$1 and no learner for $m$2 that is simultaneously consistent, permutation-invariant, superset-monotone, and a size-$m$3 compression scheme with permutation-invariant reconstruction. The proof is a three-way tension, each leg of which fails if weakened. The contradiction pits the constant $m$4 against the growing $m$5; a generic compression bound of order $m$6 would exceed the lower bound at every $m$7 and $m$8, collapsing the argument—so the log-free character of Hanneke's bound is load-bearing. The bound must also be budget-independent, which superset-monotonicity supplies, and $m$9 must be a single fixed class hard at every $t^*$0, which its union over primes provides. The consequence is that no would-be remedy within this natural algorithmic family can restore the clean rate on $t^*$1; the penalty is a property of the class–learner pairing, not of one unlucky algorithm.

A candidate measure, and its limits

The paper packages Corollary's hypotheses into $t^*$2, the least size of a scheme satisfying the four conditions. On intersection-closed classes it is pinned to $t^*$3, with $t^*$4, where $t^*$5 solves the entropy equation $t^*$6. The paper is explicit that this is a bookkeeping device, not a theory: the upper bound restates Hanneke's theorem by construction; every finite value comes from exhibiting Closure, every infinite value from a pre-existing rate lower bound; and the lower constant cannot be raised to 1, since a three-point full class has $t^*$7, verified exhaustively. The compressor's choice of retained points carries up to $t^*$8 bits of side information, which defeats the tempting counting argument for a tighter lower bound.

Limitations and open problems

The results are one-directional: $t^*$9 certifies the rate $S \subseteq T$0, but large $S \subseteq T$1 is not shown to force a large rate, and the $S \subseteq T$2 entries are derived from rate lower bounds rather than computed intrinsically. Three open problems structure the gap. First, is there a combinatorial quantity computable from $S \subseteq T$3 alone—ideally shattering-style, since every dimension in this literature that supports lower bounds is witness-defined—such that largeness implies every learner suffers $S \subseteq T$4 under monotone insertions? Any candidate must vanish on intersection-closed classes, a cheap and strong test. Second, which constants are optimal in $S \subseteq T$5; the known chain is $S \subseteq T$6 for the extreme ratios $S \subseteq T$7, and all four inequalities may be strict. Third, the middle of the landscape—classes neither intersection-closed nor of the $S \subseteq T$8 type, halfspaces being the paper's explicit example—is uncharted in both directions, and the paper concedes it has no method that would settle a new case. The matching lower bound for optimal $S \subseteq T$9 sample complexity under monotone insertions also remains open, though high-probability upper bounds follow here for free.

Conclusion

This paper reframes the monotone-adversary penalty as a property of the pairing between class and learner. Insertion-stable learners with clean-rate guarantees—Closure on intersection-closed classes being the canonical instance—render correct insertions free at any budget, with high-probability guarantees transferring via an almost-sure coupling. The $T$0 versus $T$1 separation shows the classical dimensions cannot predict the rate in either direction, and the no-scheme theorem shows the penalty on $T$2 cannot be escaped by any finite monotone compression scheme. What remains open is a converse: an intrinsic, witness-style quantity whose largeness forces the logarithmic penalty, which would convert a sufficient condition into a characterization.