Surprises in Proper Positive-Only Learning

Abstract: Binary classification from positive-only samples is a variant of PAC learning in which the learner receives i.i.d. samples from the positive region of an unknown target concept, but is evaluated under the original distribution (which places mass on both positive and negative regions). This model dates back to Natarajan [1987, STOC], and the characterization of improper learning is well-known -- it even appears in textbooks. The characterization of proper positive-only learning, however, has long remained open. In this work, we revisit and settle this question: a concept class is properly learnable from positive-only samples if and only if it has finite VC dimension and satisfies a new combinatorial condition, which we call uniform exterior separability. Together with several separation results, this characterization reveals a surprisingly rich landscape that differs sharply from standard PAC learning: proper and improper learning are separated, randomized and deterministic proper learning are separated, there are classes for which no ERM is a learner, and finite VC dimension does not suffice even for non-uniform learning. Along the way, we introduce new combinatorial dimensions that we believe can be of broader interest in learning theory.

• The paper's main contribution is characterizing when concept classes are properly learnable from positive-only samples using finite VC dimension and uniform exterior separability.
• It introduces new combinatorial dimensions and demonstrates that randomized strategies can succeed where deterministic methods, including ERM, fail.
• The study exposes sharp separations from classical PAC learning, emphasizing challenges in handling non-standard feedback in one-sided supervision.

Characterization and Separation Results in Proper Positive-Only Learning

Introduction

The paper "Surprises in Proper Positive-Only Learning" (2606.28309) delivers a rigorous analysis of proper positive-only PAC learning, a variant where learners receive only positive examples but are evaluated under a distribution supporting both positives and negatives. The work resolves a longstanding open problem by characterizing when concept classes can be properly learned in this restricted setting, introduces new combinatorial dimensions, and demonstrates several surprising separations from classical PAC learning theory. The theoretical results have direct implications for learning with limited or one-sided supervision, offering insight into learnability under non-standard feedback.

Proper Positive-Only Learning: Model and Challenges

The learning setup deviates from standard PAC learning by restricting supervision to i.i.d. samples labeled positive by the unknown target concept, drawn from the conditional distribution $D_+$ (the restriction of $D$ to positive region). The hypothesis error is measured on the full data distribution $D$, which includes both positives and negatives. A learner is defined as proper if it always produces hypotheses from the original concept class $H$; otherwise, it is improper.

In classical PAC learning, properness is not a constraint—the learnability is controlled entirely by VC dimension. In positive-only learning, improper learning is characterized via the intersection-closure $H_\cap$; $H$ is improperly learnable iff $H_\cap$ has finite VC dimension.

The key technical challenge for proper learning is that the regions not supported by the positive examples—i.e., the "exterior" with respect to the closure of consistent concepts—may be covered by hypotheses in $H$ in a way that generates excess false positives. Thus, simply outputting the intersection of all consistent concepts (the closure) is not generally viable for a proper learner.

Main Results: Combinatorial Characterization

The central contribution is the characterization of concept classes $H$ that are properly PAC learnable from positive-only samples:

Theorem (Main Characterization): $H$ is properly learnable from positive-only samples iff $D$0 has finite VC dimension and satisfies uniform exterior separability (UES).

Uniform exterior separability is a new combinatorial property. For every realizable set of positive examples $D$1 and every $D$2, there exists a bounded-size list of consistent hypotheses within $D$3, so that any exterior point (not forced positive by closure) is labeled as positive by at most an $D$4 fraction of hypotheses in the list. This ensures that a randomized learner can average over these lists to limit false positives to any specific exterior point.

The work further connects UES to weaker notions like distributional exterior separability (DES), showing that for classes of finite VC dimension, DES implies UES via a discretization argument using dual VC theory. Exact exterior separability (where the closure is always a member of $D$5) is strictly stronger and corresponds to stable learning.

Separation Results and Algorithmic Implications

The theoretical landscape uncovered differs sharply from ordinary PAC learning:

1. Proper/Improper Separation: There exist classes that are improperly learnable but not properly learnable in the positive-only model, contradicting classical equivalence.
2. Randomization Separation: There are concept classes learnable by randomized proper learners, but not any deterministic proper learner. For constant VC dimension, the number of random bits required for proper positive-only learning is $D$6, much less than the sample requirement.
3. ERM Non-Universality: Proper positive-only learning does not enjoy the universal applicability of ERM rules found in PAC learning; some classes cannot be learned by any deterministic proper ERM algorithm.
4. Non-Uniformity and Consistency: Even for concept classes of VC dimension 1, finite VC does not imply non-uniform or consistent learnability in the positive-only proper regime.
5. Stable Learning: Stable proper learning (output unchanged by additional consistent samples) corresponds tightly with exact exterior separability.

Technical Overview

The proofs leverage combinatorial and probabilistic reasoning. UES is shown necessary by constructing adversarial distributions that force significant false positive error unless the property is satisfied. Sufficiency is demonstrated via randomized algorithms averaging over sets of proper, consistent hypotheses. Finite VC dimension is needed for uniform convergence, and dual VC theory is crucial for discretizing distributional separability to yield uniform exterior separability.

The paper refutes Natarajan's conjecture, which predicted that exact intersection-closure is necessary for proper positive-only learning, by exhibiting learnable (under proper randomized strategies) classes not closed under intersection. It exploits anti-concentration and sample-based randomness as substitutes for external randomization in specific settings.

Implications and Future Directions

The results have direct theoretical impact: they redefine the boundaries of learnability under positive-only feedback, illuminate the need for combinatorial properties beyond VC dimension, and chart a path for analyzing learning under other supervision-adverse settings. Practically, this affects class design and algorithm development for domains where negative labels are missing or ambiguous (e.g., medical diagnosis, web engagement).

Future directions include quantitative tightness of random-bit complexity, analogous results for non-uniform learnability and consistency, and investigation into optimal sample complexity and efficient algorithms. The exterior separability properties introduced have potential utility in broader settings involving extrapolation under feedback sparsity.

Conclusion

"Surprises in Proper Positive-Only Learning" provides the first complete combinatorial characterization of proper positive-only PAC learning, reveals deep structural separations from classical theory, and opens new avenues for the study of learning under constraints on supervision. The new notions of exterior separability and their operational connections to randomization and ERM universality offer foundational tools for future theoretical and applied learning research.