Papers
Topics
Authors
Recent
Search
2000 character limit reached

PAC Learning Linear Thresholds from Label Proportions

Published 16 Oct 2023 in cs.LG and stat.ML | (2310.10098v1)

Abstract: Learning from label proportions (LLP) is a generalization of supervised learning in which the training data is available as sets or bags of feature-vectors (instances) along with the average instance-label of each bag. The goal is to train a good instance classifier. While most previous works on LLP have focused on training models on such training data, computational learnability of LLP was only recently explored by [Saket'21, Saket'22] who showed worst case intractability of properly learning linear threshold functions (LTFs) from label proportions. However, their work did not rule out efficient algorithms for this problem on natural distributions. In this work we show that it is indeed possible to efficiently learn LTFs using LTFs when given access to random bags of some label proportion in which feature-vectors are, conditioned on their labels, independently sampled from a Gaussian distribution $N(\mathbf{\mu}, \mathbf{\Sigma})$. Our work shows that a certain matrix -- formed using covariances of the differences of feature-vectors sampled from the bags with and without replacement -- necessarily has its principal component, after a transformation, in the direction of the normal vector of the LTF. Our algorithm estimates the means and covariance matrices using subgaussian concentration bounds which we show can be applied to efficiently sample bags for approximating the normal direction. Using this in conjunction with novel generalization error bounds in the bag setting, we show that a low error hypothesis LTF can be identified. For some special cases of the $N(\mathbf{0}, \mathbf{I})$ distribution we provide a simpler mean estimation based algorithm. We include an experimental evaluation of our learning algorithms along with a comparison with those of [Saket'21, Saket'22] and random LTFs, demonstrating the effectiveness of our techniques.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (35)
  1. The power of localization for efficiently learning linear separators with noise. J. ACM, 63(6):50:1–50:27, 2017.
  2. Hardness of learning noisy halfspaces using polynomial thresholds. In Proc. COLT, volume 75, pages 876–917. PMLR, 2018. URL http://proceedings.mlr.press/v75/bhattacharyya18a.html.
  3. Learnability and the vapnik-chervonenkis dimension. J. ACM, 36(4):929–965, 1989.
  4. Easy learning from label proportions. arXiv, 2023. URL https://arxiv.org/abs/2302.03115.
  5. Cost-based labeling of groups of mass spectra. In Proc. ACM SIGMOD International Conference on Management of Data, pages 167–178, 2004.
  6. Learning from aggregated data: Curated bags versus random bags. arXiv, 2023. URL https://arxiv.org/abs/2305.09557.
  7. Amit Daniely. A PTAS for agnostically learning halfspaces. In Proceedings of The 28th Conference on Learning Theory, COLT 2015, Paris, France, July 3-6, 2015, volume 40 of JMLR Workshop and Conference Proceedings, pages 484–502, 2015.
  8. S. Dasgupta. Learning mixtures of Gaussians. In FOCS, pages 634–644, 1999.
  9. N. de Freitas and H. Kück. Learning about individuals from group statistics. In Proc. UAI, pages 332–339, 2005.
  10. Weakly supervised classification in high energy physics. Journal of High Energy Physics, 2017(5):1–11, 2017.
  11. Deep multi-class learning from label proportions. CoRR, abs/1905.12909, 2019. URL http://arxiv.org/abs/1905.12909.
  12. On agnostic learning of parities, monomials, and halfspaces. SIAM J. Comput., 39(2):606–645, 2009.
  13. On the complexity of learning a class ratio from unlabeled data. J. Artif. Intell. Res., 69:1333–1349, 2020.
  14. V. Guruswami and P. Raghavendra. Hardness of learning halfspaces with noise. In Proc. FOCS, pages 543–552, 2006.
  15. Learning bayesian network classifiers from label proportions. Pattern Recognit., 46(12):3425–3440, 2013.
  16. Agnostically learning halfspaces. In 46th Annual IEEE Symposium on Foundations of Computer Science (FOCS 2005), 23-25 October 2005, Pittsburgh, PA, USA, Proceedings, pages 11–20. IEEE Computer Society, 2005.
  17. Learning halfspaces with malicious noise. J. Mach. Learn. Res., 10:2715–2740, 2009.
  18. From group to individual labels using deep features. In Proc. SIGKDD, pages 597–606, 2015.
  19. Learning from label proportions with generative adversarial networks. In Proc. NeurIPS, pages 7167–7177, 2019.
  20. Supervised learning by training on aggregate outputs. In Proc. ICDM, pages 252–261. IEEE Computer Society, 2007.
  21. Domain-agnostic contrastive representations for learning from label proportions. In Proc. CIKM, pages 1542–1551, 2022.
  22. (almost) no label no cry. In Proc. Advances in Neural Information Processing Systems, pages 190–198, 2014.
  23. Estimating labels from label proportions. J. Mach. Learn. Res., 10:2349–2374, 2009.
  24. S. Rueping. SVM classifier estimation from group probabilities. In Proc. ICML, pages 911–918, 2010.
  25. R. Saket. Learnability of linear thresholds from label proportions. In Proc. NeurIPS, 2021. URL https://openreview.net/forum?id=5BnaKeEwuYk.
  26. R. Saket. Algorithms and hardness for learning linear thresholds from label proportions. In Proc. NeurIPS, 2022. URL https://openreview.net/forum?id=4LZo68TuF-4.
  27. On combining bags to better learn from label proportions. In AISTATS, volume 151 of Proceedings of Machine Learning Research, pages 5913–5927. PMLR, 2022. URL https://proceedings.mlr.press/v151/saket22a.html.
  28. C. Scott and J. Zhang. Learning from label proportions: A mutual contamination framework. In Proc. NeurIPS, 2020.
  29. Leslie G. Valiant. A theory of the learnable. Commun. ACM, 27(11):1134–1142, 1984.
  30. S. Vempala. Learning convex concepts from gaussian distributions with PCA. In FOCS, pages 124–130, 2010.
  31. R. Vershynin. How close is the sample covariance matrix to the actual covariance matrix? J. Theor. Probab., 25:655–686, 2012.
  32. Roman Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018. doi: 10.1017/9781108231596.
  33. Using published medical results and non-homogenous data in rule learning. In Proc. International Conference on Machine Learning and Applications and Workshops, volume 2, pages 84–89. IEEE, 2011.
  34. ∝proportional-to\propto∝SVM for learning with label proportions. In Proc. ICML, volume 28, pages 504–512, 2013.
  35. On learning from label proportions. CoRR, abs/1402.5902, 2014. URL http://arxiv.org/abs/1402.5902.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up