---
title: Proper Positive-Only Learning
url: https://www.emergentmind.com/topics/proper-positive-only-learning
type: topic
---

# Proper Positive-Only Learning

Proper positive-only learning is the problem of learning from observations that come only from the positive side of a target while still requiring the learner to output a hypothesis in the original target class. In its narrowest formalization, the learner receives i.i.d. samples from the positive region of an unknown concept \(c\in\mathcal C\), but is evaluated under the original distribution on the full instance space, so both false negatives and false positives matter; the distinctive issue is therefore not merely recovering positive support, but controlling extrapolation outside it while remaining proper [2606.28309]. In a broader literature, the same theme appears in positive-and-unlabeled learning, symbolic inference of automata and temporal formulas, federated multi-class learning with only local positives, one-sided feedback, and positive-only policy optimization, all of which ask when a proper predictor can still be learned despite one-sided supervision [1808.08755], [2511.08431], [2209.02650], [2004.10342], [2006.04858], [2605.06650].

## 1. Formal model and exact characterization

The recent formal treatment of positive-only PAC learning fixes an instance space \(X\), a concept class \(\mathcal C\subseteq 2^X\), an unknown target concept \(c\in\mathcal C\), and an unknown distribution \(\mathcal D\) on \(X\). The learner observes a multiset
\[
S_+ \sim \mathcal D_+^n,
\]
where \(\mathcal D_+\) is the conditional distribution on the positive region \(c\), and outputs a hypothesis \(h\subseteq X\). Error is measured under the full distribution by
\[
\operatorname{err}_{\mathcal D}(h,c)\coloneqq \mathcal D(h\triangle c).
\]
A proper learner is a map
\[
A: X^{[*]}\times \Omega \to \mathcal C,
\]
so the output must lie in the same class as the target [2606.28309].

Two induced set-valued objects organize the theory. For a finite realizable set \(S\subseteq X\), the version space is
\[
\mathcal C_S\coloneqq \{h\in\mathcal C: S\subseteq h\},
\]
and the closure is
\[
\operatorname{cl}_{\mathcal C}(S)\coloneqq \bigcap_{h\in \mathcal C_S} h.
\]
Because the true target belongs to \(\mathcal C_S\), one always has \(\operatorname{cl}_{\mathcal C}(S)\subseteq c\). The closure therefore introduces no false positives, but it need not itself belong to \(\mathcal C\), which is exactly where properness becomes nontrivial [2606.28309].

The central combinatorial condition is uniform exterior separability. The class \(\mathcal C\) satisfies uniform exterior separability if for every \(\eta>0\) there exists \(M(\eta)\in\mathbb N\) such that for every finite nonempty realizable \(S\subseteq X\), there exist hypotheses \(h_{S,1},\dots,h_{S,M(\eta)}\in \mathcal C_S\) with
\[
\sup_{x\notin \operatorname{cl}_{\mathcal C}(S)} \frac{1}{M(\eta)} \left| \{i\in\{1,\dots,M(\eta)\}:x\in h_{S,i}\} \right| \le \eta.
\]
This says that although the exact closure may be improper, a bounded randomized list of proper hypotheses can approximate it while spreading false positives thinly over the exterior [2606.28309].

The resulting characterization is exact:
\[
\boxed{ \text{\(\mathcal C\) is properly learnable from positive-only samples} \iff \operatorname{VCdim}(\mathcal C)<\infty \text{ and \(\mathcal C\) satisfies uniform exterior separability.} }
\]
Finite VC dimension controls generalization from observed positives, while uniform exterior separability controls false positives outside the closure. This sharply departs from ordinary realizable PAC learning, where finite VC dimension alone suffices [2606.28309].

## 2. Consequences: separations, randomness, and non-ERM behavior

The same analysis shows that improper positive-only learning is governed by a different object. Let
\[
\mathcal C_\cap \coloneqq \left\{ \bigcap_{h\in \mathcal A} h: \mathcal A\subseteq \mathcal C \text{ finite} \right\}.
\]
Then
\[
\mathcal C \text{ is improperly learnable from positive-only samples } \iff \operatorname{VCdim}(\mathcal C_\cap)<\infty.
\]
Proper and improper learning therefore separate. A minimal example is
\[
\mathcal C=\big\{\{0,1\},\{0,2\}\big\},
\]
for which \(\mathcal C_\cap=\{\{0\},\{0,1\},\{0,2\}\}\) has VC dimension \(1\), so improper learnability holds, but proper learnability fails because the closure of \(\{0\}\) is \(\{0\}\notin\mathcal C\) [2606.28309].

Randomized and deterministic proper learning also separate. The class
\[
\mathcal C=\big\{\{1,a\}: a\in \mathbb N\setminus\{1\}\big\}
\]
has VC dimension \(1\) and satisfies uniform exterior separability, hence is properly positive-only learnable, but no deterministic proper learner exists. The obstruction is singleton closure:
\[
\forall \text{ realizable }x,\qquad \operatorname{cl}_{\mathcal C}(\{x\})\in \mathcal C,
\]
which every deterministic proper positive-only learner must satisfy, and which this class violates [2606.28309].

The model also breaks the usual universality of ERM. For
\[
\mathcal C_{\mathrm{spr}} = \big\{\{1\},\{2\}\big\} \cup \big\{\{1,2,i\}: i\ge 3\big\},
\]
the class has VC dimension \(1\) and is properly positive-only learnable, but no deterministic proper ERM learner succeeds. The reason is that consistency with observed positives does not determine how to allocate unavoidable false positives across the unseen exterior, and ERM has no mechanism for the anti-concentration strategy that the positive-only model requires [2606.28309].

Several further distinctions follow. Exact exterior separation implies uniform exterior separability, which implies distributional exterior separability, which implies finite exterior separability:
\[
\text{EES} \Longrightarrow \text{UES} \Longrightarrow \text{DES} \Longrightarrow \text{FES}.
\]
Stable proper positive-only learnability collapses back to the strongest condition:
\[
\mathcal C \text{ is properly positive-only learnable by a stable learner} \iff \operatorname{VCdim}(\mathcal C)<\infty \text{ and EES}.
\]
Finite VC dimension is not enough even for non-uniform proper positive-only learnability, and finite VC dimension does not imply consistency. At the quantitative level, if \(d=\operatorname{VCdim}(\mathcal C)\) and \(d^\star=\operatorname{VCdim}(\mathcal C^\star)\), randomized proper positive-only learning can be achieved with
\[
r(\varepsilon,\delta)=O\!\left(\log\frac{d^\star}{\delta}\right)
\]
random bits and
\[
m(\varepsilon,\delta)=O\!\left(\frac{d+\log(1/\delta)}{\varepsilon}\right)
\]
positive samples [2606.28309].

## 3. Positive-only versus positive-and-unlabeled learning

Much of the earlier literature approaches one-sided supervision through positive-and-unlabeled data rather than positive-only samples. In the SAR formulation of PU learning, each example is represented as \((x,y,s)\), where \(y\in\{0,1\}\) is the latent class label and \(s\in\{0,1\}\) is the observed label indicator with \(s=1\Rightarrow y=1\). Under SCAR,
\[
c = P(s=1\mid x,y=1)=P(s=1\mid y=1),
\]
so
\[
P(y=1\mid x)=\frac{1}{c}P(s=1\mid x).
\]
Under SAR, labeling depends on propensity attributes \(x_e\subseteq x\):
\[
P(s=1\mid x)=e(x_e)\,P(y=1\mid x),
\]
where
\[
e(x)=P(s=1\mid y=1,x)=e(x_e).
\]
The SAR-EM algorithm jointly learns \(P(y=1\mid x,\theta)\) and \(P(s=1\mid y=1,x_e,\phi)\) via EM, with unlabeled posterior
\[
P(y=1\mid x,s=0,\theta,\phi) = \frac{P(y=1\mid x,\theta)\,[1-e(x_e)]} {P(y=1\mid x,\theta)\,[1-e(x_e)] + 1-P(y=1\mid x,\theta)}.
\]
This is proper in the sense that the target remains the true posterior \(P(y=1\mid x)\), but it is not positive-only in the strict sample-only sense because unlabeled data and an explicit labeling model are essential [1808.08755].

The same distinction appears in later PU formulations. Selection biases in positive and unlabeled data are common, and an empirical-risk-based method that incorporates the labeling mechanism improves trained classifiers even when the mechanism is unknown; the explicit point is that positive and unlabeled learning becomes brittle when selection bias is ignored [1809.03207]. Variational PU learning replaces class-prior-weighted supervised-risk decompositions by the criterion
\[
\mathcal L_{\mathrm{var}}(\Phi) \triangleq \log \mathbb E_f[\Phi(x)] - \mathbb E_{f_P}[\log \Phi(x)],
\]
with
\[
\mathrm{KL}(f_P\|f_\Phi) = \mathcal L_{\mathrm{var}}(\Phi)-\mathcal L_{\mathrm{var}}(\Phi^*).
\]
Under SCAR, an anchor condition, and model realizability, the learned posterior satisfies
\[
\Phi \stackrel{p}{\to} \Phi^* \quad\text{as } M,N\to\infty,\ \lambda\to 0,
\]
so the method targets the Bayes posterior directly without explicit class-prior estimation [1906.00642].

A different route is to extract negative information only collectively from unlabeled examples. The collective PU objective uses
\[
\ell(\hat\eta,y) = \begin{cases} -\ln \hat\eta & \text{if } y=1\\
-\ln \left(1-\left|\frac{1}{|\mathsf U_b|}\sum_{x\in \mathsf U_b}\hat\eta(x)-\mu_p\right|\right) & \text{if } y=0,
\end{cases}
\]
so unlabeled data contribute through a batch-level prevalence constraint
\[
\bar{\hat\eta}_{U_b}\approx \mu_p.
\]
This avoids per-instance negative pseudo-labels, but it still depends essentially on unlabeled data and a positive-prevalence parameter [2005.03228].

## 4. Proper positive-only learning for symbolic and interpretable hypothesis classes

The proper positive-only paradigm is especially natural for symbolic classes whose outputs are already interpretable objects. In passive DFA learning from positive examples, the input is an alphabet \(\Sigma\), a finite positive sample set \(\mathcal P\subseteq \Sigma^\ast\), and a state bound \(n\). The proper hypothesis space is
\[
\mathsf{Rec}(\mathcal{P},n) := \{ \mathcal{A} \in \mathsf{DFA}(\Sigma,n) \mid \mathcal{P} \subseteq L(\mathcal{A}) \}.
\]
The learning objective is to find \(\mathcal A\in \mathsf{Rec}(\mathcal P,n)\) that is minimal under language inclusion. A finite surrogate exists: if \(h=2n-2\), then minimizing
\[
|L(\mathcal A)\cap \Sigma^{\le h}|
\]
is sufficient for language minimality. More precisely, Proposition 1 states that a DFA minimal w.r.t.
\[
\mathcal A \prec_h \mathcal A' \iff |L(\mathcal A)\cap \Sigma^{\le h}| < |L(\mathcal A')\cap \Sigma^{\le h}|
\]
is also minimal w.r.t. strict language inclusion. The associated decision problem—whether there exists \(\mathcal A\in\mathsf{Rec}(\mathcal P,n)\) with
\[
|L(\mathcal A)\cap \Sigma^{\le 2n-2}| \le k
\]
—is NP-complete, even over a binary alphabet. The paper also gives an ILP formulation and a heuristic preprocessing algorithm; the exact solver underperforms the symbolic baseline in practice, but the heuristic can improve starting points for symbolic descent [2511.08431].

A closely related line studies interpretable temporal explanations from positive traces only. For DFAs, given positive words \(P\) and size bound \(n\), the goal is to learn an \(n\)-description \(A\) with \(P\subseteq L(A)\) and no strictly smaller-language \(n\)-description. For \(\mathrm{LTL}_f\), the goal is an \(n\)-description \(\varphi\) such that \(\varphi\) is satisfied by every positive word and no strictly stronger \(n\)-description exists. The symbolic DFA algorithm uses a SAT encoding
\[
\Phi^A \coloneqq \Phi_{\mathtt{DFA}} \wedge \Phi_{P} \wedge \Phi_{\subseteq A} \wedge \Phi_{\not\supseteq A},
\]
iteratively descending in language inclusion until no smaller consistent DFA exists. For \(\mathrm{LTL}_f\), semi-symbolic and counterexample-guided algorithms combine bounded-size syntax-DAG search with implication checks and generated negatives. The guarantees are proper: Theorem 1 returns a DFA \(A\) such that for every DFA \(A'\) that is an \(n\)-description,
\[
L(A')\not\subset L(A),
\]
and Theorem 3 returns an \(\mathrm{LTL}_f\) formula \(\varphi\) such that for every bounded-size positive-consistent \(\varphi'\),
\[
\varphi' \not\rightarrow \varphi \;\text{or}\; \varphi \rightarrow \varphi'.
\]
Here properness is literal class preservation: the learner outputs a DFA or an \(\mathrm{LTL}_f\) formula, not an improper proxy [2209.02650].

## 5. Distributed, active, and sequential variants

Proper positive-only learning also appears in settings where the missing negative information is supplied by geometry, exploration, or active querying rather than by explicit negative labels. In federated multi-class learning with only positive labels, client \(i\) holds
\[
S^i = \{(x^i_1,i), \ldots, (x^i_{n_i}, i)\},
\]
can access only its own class embedding \(w_i\), and cannot access embeddings \(w_c\) for \(c\neq i\). Naively optimizing only the positive term leads to embedding collapse. FedAwS fixes this by a server-side spreadout regularizer
\[
\mathrm{reg}(W) = \sum_{c\in[C]}\sum_{c'\neq c} \big(\max\{0, \nu - d(w_c,w_{c'})\}\big)^2,
\]
combined with local positive updates. Under equal class sizes and \(\lambda=1/C\), the resulting objective equals empirical risk under
\[
\ell_{\rm sp}(f(x), y) = (1-s_y)^2 + \sum_{c\neq y} \big(\max\{0, \nu - 1 + w_y^\top w_c\}\big)^2,
\]
so a proper multi-class classifier can be learned despite strictly local positive-only supervision [2004.10342].

In one-sided feedback for generalized linear models, the learner observes labels only when it takes the positive action. The proper class is still
\[
\Pi=\{\pi^\beta:\|\beta\|_2\le M\},\qquad \pi^\beta(x)=\mathbf{1}\{\mu(x^\top\beta)\ge c\}.
\]
Naive greedy ERM on observed labels can fail permanently because rejected regions are never labeled. The adaptive solution uses optimistic querying: accept \(x_j^t\) whenever
\[
\mu(x_j^{t\top}\beta_t)-c +\rho_t(\delta)\sqrt{x_j^{t\top}A^{-1}x_j^t} >0.
\]
The resulting theorem bounds cumulative one-sided loss by
\[
R_T \le \widetilde{\mathcal O}\!\left( T_* K + \frac{L}{\eta}T_* \sqrt{Ts}\, dN \right),
\]
which implies vanishing average one-sided loss under the stated GLM assumptions [2006.04858].

Active PU learning adds adaptive querying to a one-sided reveal model. A queried example reveals a positive label only if it is positive and an independent coin with bias \(\omega\) succeeds; otherwise the learner receives \(\star\). The output is again proper, \(\hat h\in H\). When the positive class prior \(\pi_D\) is known, the paper gives a disagreement-based algorithm whose label complexity is
\[
O\!\left(
\frac{\ln(1/\varepsilon)\,\theta^2\bigl(d\ln(\theta)+\ln\ln(1/\varepsilon)+\ln(1/\delta)\bigr)}{\omega}
\right).
\]
When \(\pi_D\) is unknown, there is an additional \(\pi_D^{-2}\)-dependent overhead for estimating \(\omega\), but the final guarantee still returns \(\hat h\in H\) with
\[
err_D(\hat h,\ell)\le \varepsilon
\]
with probability at least \(1-\delta\) [2602.02081].

Positive-only optimization has also been carried into reinforcement learning with verifiable rewards. In POPO, rollouts are partitioned into
\[
S^+(x)=\{y: R(x,y)=1\}, \qquad S^-(x)=\{y: R(x,y)=0\},
\]
and the actor objective uses only \(S^+(x)\):
\[
L_{\text{POPO}} = -E_{x \sim D} \left[ \sum_{y \in S^+(x)} w_\theta(y \mid x) \cdot \log \pi_\theta(y \mid x) \right] + \alpha L_{\text{sim}} + \beta L_{\text{ent}},
\]
with
\[
w_\theta(y \mid x) = \frac{\pi_\theta(y \mid x)}{Z^+(x)}, \qquad Z^+(x) = \sum_{y' \in S^+(x)} \pi_\theta(y' \mid x).
\]
For any incorrect response \(y' \in S^-(x)\),
\[
\frac{\partial L_{\text{POPO}}}{\partial z_{y'}} = \pi_\theta(y' \mid x) \Big[ 1 + \beta \big(\log \pi_\theta(y' \mid x) + H(\pi_\theta(\cdot \mid x))\big)\Big],
\]
so minimizing the positive-only objective still pushes negative logits downward implicitly. Empirically, on Qwen-Math-7B, POPO achieves \(36.67\%\) on AIME 2025 versus GRPO’s \(30.00\%\) [2605.06650].

## 6. Limits, boundaries, and open directions

The broad literature repeatedly shows that “positive-only” is not a single statistical regime. A tensor-network PU method can learn from positive and unlabeled data and generate both positive and negative samples, but it is “not a canonical proper PU learner” in the classical statistical sense: it does not derive an unbiased PU risk estimator, estimate the class prior, or prove identifiability of the negative distribution from PU observations alone [2211.14085]. This marks an important boundary between practical one-stage positive/unlabeled methods and properness as posterior recovery or risk-consistent estimation.

The same distinction becomes sharper under distribution shift. Positive and Imperfect Unlabeled learning assumes access to positive samples from the true feature distribution and to an imperfect unlabeled distribution \(D\) satisfying generalized smoothness
\[
D^\star(S)\le \frac{1}{\sigma}\, D(S)^{1/q}.
\]
This yields strong positive-only corollaries when the learner knows a reference distribution \(D\), but the resulting learners are generally improper: the sample-efficient method outputs an intersection of \(O(1/\varepsilon)\) hypotheses, and the efficient method outputs a degree-\(k\) polynomial threshold function. The paper explicitly argues that proper PIU learning is impossible in general, and it also proves that without a reference or unlabeled distribution, positive-only learning is impossible for many natural classes even improperly [2504.10428].

A complementary general lesson comes from proper PAC theory with access to the full unlabeled marginal. In the distribution-fixed PAC model, there always exists a randomized proper learner governed by distributional regularization, and this learner can be properized by the \(\mathcal D\)-metric projection
\[
\mathcal B(\mathcal D,S) = \arg\min_{h\in\mathcal H} \operatorname{dist}_{\mathcal D}(h,\mathcal A(\mathcal D,S)).
\]
At the same time, sample complexity can shrink by only a logarithmic factor relative to classic PAC, and proper learnability in the ordinary realizable PAC model can be logically undecidable, non-monotone, and non-local [2502.10359]. This suggests that side information may restore properness more readily than it changes worst-case information-theoretic complexity.

In the narrow passive positive-only model, the exact characterization leaves several questions open: the exact random-bit complexity of proper positive-only learning, analogous characterizations for non-uniform learnability and consistency, and optimal statistical rates together with computationally efficient algorithms [2606.28309]. Taken together, these results indicate that proper positive-only learning is now well understood at the level of information-theoretic structure in its core binary setting, but still fragmented across positive-and-unlabeled, active, sequential, and structured symbolic variants, where properness, identifiability, and tractability do not generally coincide.

Source: https://www.emergentmind.com/topics/proper-positive-only-learning