Learning from Label Proportions (LLP)

• LLP is a weakly supervised learning paradigm that infers individual label distributions from aggregate class proportions in groups or bags.
• It utilizes empirical proportion risk minimization and debiasing techniques to align predicted proportions with observed statistics.
• LLP supports applications in privacy-preserving learning, epidemiology, and political polling, balancing computational complexity and statistical guarantees.

Learning from Label Proportions (LLP) is a weakly supervised learning paradigm in which only aggregate statistics—specifically, the proportions of each class within predefined groups of instances called “bags”—are provided during training, rather than individual instance labels. The learner’s objective is to induce an instance-level classifier, or more generally to recover the distribution of labels for new, unlabelled examples from the underlying population. LLP is motivated by scenarios where privacy, aggregation, or cost prohibits the collection of instance-wise labels but aggregate quantification is available, as in political polling, epidemiology, marketing, or privacy-preserving learning.

1. Formalization and Statistical Foundation

LLP is defined over a domain $\mathcal{X}$ with label set $\mathcal{Y}$ (typically $\{0,1\}$ or $\{1,\dots,C\}$). Training data consists of $m$ bags $B_1,\ldots,B_m$, each $B_j = \{x_{j,1}, \ldots, x_{j,k_j}\}$ containing $k_j$ instances. Instead of observing instance labels $y_{j,i}$, only the empirical label-proportion vector $p_j \in \Delta^\mathcal{Y}$ is provided: $$p_{j,c} = \frac{1}{k_j} \sum_{i=1}^{k_j} \mathbf{1}\{ y_{j,i} = c \}.$$ The hypothesis class $H$ contains functions $h: \mathcal{X} \to \mathcal{Y}$ (or probability vectors $\Delta^\mathcal{Y}$ for multiclass).

In the realizable setting, the goal is to find $h \in H$ whose induced label distribution over $\mathcal{X}$ matches the population-level distribution of the true labeling function $c: \mathcal{X} \to \mathcal{Y}$ to within some margin—typically measured by $|p_h - p_c|$, where $p_f = \Pr_{x \sim D}[f(x) = 1]$ in binary cases.

A central framework is Empirical Proportion Risk Minimization (EPRM), which seeks $h$ minimizing an empirical loss over bagwise predicted vs. observed proportions, e.g.,

$$\hat{h} \in \arg\min_{h \in H} \frac{1}{m} \sum_{j=1}^m L\left( \varphi_{k_j}(h)(B_j), p_j \right),$$

with $L$ a loss function such as $L_1$ or cross-entropy, and

$$\varphi_{k_j}(h)(B_j) = \frac{1}{k_j} \sum_{i=1}^{k_j} \mathbf{1}\{ h(x_{j,i}) = 1 \}.$$

Under mild regularity conditions, EPRM admits generalization guarantees on the bag-risk that are only logarithmically sensitive to the bag size and depend on the VC-dimension of $H$ (Yu et al., 2014).

2. Computational Complexity and Sample Efficiency

LLP's computational learnability diverges sharply from classical PAC learning due to the information constraints of proportion-only supervision. Key findings (Fish et al., 2020):

• Efficient LLP Learning is a Subset of Efficient PAC Learning: For finite VC-dimension $d$, if $H$ is efficiently LLP-learnable, it is also efficiently PAC-learnable (assuming $\mathrm{NP} \neq \mathrm{RP}$). However, the converse fails: some efficiently PAC-learnable classes are not efficiently LLP-learnable, e.g., learning noisy parity or monotone disjunctions is computationally hard (NP-hard) for LLP.
• Information-Theoretic Learnability: Any class with finite VC-dimension is (information-theoretically) LLP-learnable, with $m = O((d+\log 1/\delta)/\epsilon^2)$ sufficing to achieve $\epsilon$-accurate prediction of label proportions (Fish et al., 2020).
• Sample Complexity: For square loss and general hypothesis classes, the minimax-optimal sample complexity for individual-level regret $\leq \epsilon$ is $n = \Theta(k/\epsilon^2)$ in the non-realizable case, and $n = \Theta(k/\epsilon)$ in the realizable case, where $k$ is bag size and $n$ is the number of instances (Busa-Fekete et al., 8 May 2025).

Crucially, the VC-dimension suffices for sample efficiency but does not capture computational tractability: infinite-VC classes can sometimes be efficiently LLP-learned (e.g., all finite subsets of $\mathbb{N}$ via subset-sum dynamic programming), and some finite-VC classes are computationally intractable (Fish et al., 2020).

3. Algorithmic Methodologies and Practical Approaches

Multiple algorithmic strategies have been developed for LLP, typically falling into these classes:

• Direct Proportion Matching (EPRM): Minimizes a loss between predicted and observed proportions. Most foundational results and early practical algorithms use this paradigm (Yu et al., 2014, Liu et al., 2021).
• Risk-Consistent Surrogates: Recent advances introduce unbiased or low-variance de-biasing to estimate instance-wise loss from aggregate proportions, handling arbitrary loss functions and both binary and multiclass outputs. The EasyLLP estimator and its generalizations achieve only an $O(k)$ (bag size) slowdown compared to full supervision and allow for efficient SGD/ERM implementations (Busa-Fekete et al., 2023, Applebaum et al., 18 Sep 2025).
• Reduction to Learning with Label Noise: The LLPFC approach shows that LLP can be converted to a problem of learning with known label noise matrices via group-wise proportions, using forward correction losses to achieve risk consistency and finite-sample excess risk bounds (Zhang et al., 2022).
• Pseudo-Labeling and Regret Minimization: Explicit combinatorial pseudo-label assignment constrained to match bag-level proportions, with online regret-minimization (Follow-the-Perturbed-Leader) yielding robust solutions that scale to large bags (Matsuo et al., 2023).
• End-to-End Deep Learning Architectures: Incorporate bag-level cross-entropy or KL loss over softmax-averages, sometimes augmented by consistency regularization (e.g., VAT), optimal transport for pseudo-label smoothing, or contrastive objectives (Tsai et al., 2019, Liu et al., 2021, Wang et al., 2024).
• Adversarial LLP: Use of GAN (LLP-GAN) architectures to enforce consistency of discriminator predictions with bag proportions via adversarial min–max games, providing global optimality guarantees under infinite capacity (Liu et al., 2019).
• Statistical De-biasing in Large-Bag Settings: Proportion label perturbation using multivariate hypergeometric modeling for mini-bag sampling allows LLP to be feasible for bags of very large size even under GPU constraints, with loss weighting for rare-tail mini-bags (Kubo et al., 2024).

The mutual contamination model (MCM) framework provides an unbiased loss by treating each bag as a mixture of class-conditional distributions, and correcting for this to recover the balanced error rate in the original data (Scott et al., 2020).

4. Extensions, Applications, and Practical Considerations

LLP has broad applicability in settings where only group-level statistics are legally or logistically available:

• Brain–Computer Interfaces: LLP offers calibration-free unsupervised methods for ERP-based BCI spelling systems, robust to non-stationarities and with closed-form estimators for class means (Hübner et al., 2017).
• Healthcare, Market Segmentation, and Demographics: Models can infer population-level disease prevalence, voting, or consumer preference from group statistics (Yu et al., 2014, Applebaum et al., 18 Sep 2025).
• Privacy-Preserving Learning: Since individual labels need not be revealed, one can add noise to bag proportions for differentially private learning without degrading instance classification accuracy substantially (Yu et al., 2014).
• Large-Scale Vision or Tabular Data: Bag-level data augmentation methods (MixBag), contrastive architectures (TabLLP-BDC), and auxiliary instance-weighted losses further improve performance, particularly in deep learning regimes (Asanomi et al., 2023, Wang et al., 2024, Ma et al., 2024).
• Transfer Learning: Knowledge transfer between source and target LLP tasks via joint parameterization can increase label efficiency and robustness to noisy data (Xiao et al., 2019).

The effectiveness of debiasing, contrastive, or adversarial methods often depends strongly on bag size, heterogeneity of bag compositions, and class imbalance. Empirical results show that approaches employing high-confidence instance-level weighting or bag-level augmentation retain significantly higher accuracy as bag sizes increase, a regime where naïve LLP approaches often collapse (Ma et al., 2024, Asanomi et al., 2023).

5. Theoretical Characterization, Limitations, and Open Problems

LLP departs from the classical VC dimension–based characterization of learnability. Notable findings include:

• Independence from ZFC: There exist concept classes whose LLP learnability is independent of standard set theory axioms (ZFC), precluding characterization via combinatorial dimension alone; e.g., EMX-learnability of finite subsets of $[0,1]$ (Fish et al., 2020).
• Failure of VC-Dimension Alone: There exist infinite-VC classes that are LLP-learnable efficiently, and finite-VC classes that are NP-hard for LLP (Fish et al., 2020).
• Sample Complexity: Recent results provide minimax-optimal sample complexity for square loss, and extend these results to general loss functions with linear (in bag size) scaling (Busa-Fekete et al., 8 May 2025, Applebaum et al., 18 Sep 2025).
• Limitations of Direct Proportion Matching: EPRM and similar methods can fail agnostically: unless bag sizes or compositions admit sufficient “purity” or diversity, proportion matching may be exponentially sample-inefficient or even inconsistent for instance-level risk (Li et al., 2024).
• Unbiasedness and Variance Trade-offs: Risk-consistent and classifier-consistent estimators—especially those using debiased or median-of-means estimates—achieve sublinear or optimal variance but often at greater computational cost or with additional assumptions on bagging (Applebaum et al., 18 Sep 2025, Kobayashi et al., 2022).

Open problems include:

• Multi-Bag Unlabeled Supervision: Determining whether and when multiple bag observations with distinct proportions strictly increase computational or statistical learning power (Fish et al., 2020).
• Robustness to Proportion Noise: Extending robust and high-confidence pseudo-labeling or uncertainty-aware approaches to LLP with corrupted bag proportions (Ma et al., 2024).
• Structural Parameterizations: Discovering combinatorial or analytic parameters finer than VC-dimension that can predict computational hardness or generalization in LLP.
• Scaling and Hardware Efficiency: Further algorithmic refinements for efficient gradient and memory scaling to truly massive bags, especially under modern hardware constraints (Kubo et al., 2024).

6. Synthesis and Outlook

LLP represents a theoretically and practically rich weak supervision regime with unique statistical and computational challenges. While classical sample complexity analyses based on VC-dimension ensure bag-level risk generalization, the hardness of instance-level learning is governed by a confluence of hypothesis class structure, bag composition, and the form of loss function employed. Recent methodological advances, ranging from risk-consistent de-biasing, mixture and adversarial models, to data-augmentation and high-confidence filtering, have bridged many of the empirical gaps experienced by earlier direct-proportion-matching approaches, particularly for large-scale and deep architectures.

The lack of a unifying, combinatorial learnability theory for LLP—analogous to VC-dimension for PAC learning—and the persistence of hardness for even simple function classes, mark it as a frontier for future foundational work (Fish et al., 2020). Continued convergence of statistical learning theory, optimization, and practical application is required to extend LLP's flexibility and guarantees to the increasingly data-aggregated and privacy-sensitive landscape of machine learning.