Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Consistent Lebesgue Measure for Multi-label Learning (2402.00324v1)

Published 1 Feb 2024 in cs.LG

Abstract: Multi-label loss functions are usually non-differentiable, requiring surrogate loss functions for gradient-based optimisation. The consistency of surrogate loss functions is not proven and is exacerbated by the conflicting nature of multi-label loss functions. To directly learn from multiple related, yet potentially conflicting multi-label loss functions, we propose a Consistent Lebesgue Measure-based Multi-label Learner (CLML) and prove that CLML can achieve theoretical consistency under a Bayes risk framework. Empirical evidence supports our theory by demonstrating that: (1) CLML can consistently achieve state-of-the-art results; (2) the primary performance factor is the Lebesgue measure design, as CLML optimises a simpler feedforward model without additional label graph, perturbation-based conditioning, or semantic embeddings; and (3) an analysis of the results not only distinguishes CLML's effectiveness but also highlights inconsistencies between the surrogate and the desired loss functions.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. Hype: An algorithm for fast hypervolume-based many-objective optimization. Evolutionary Computation, 19(1):45–76, 2011.
  2. Faster hypervolume-based search using Monte Carlo sampling. In Multiple Criteria Decision Making for Sustainable Energy and Transportation Systems: Proceedings of the 19th International Conference on Multiple Criteria Decision Making, Auckland, New Zealand, 7th-12th January 2008, pp.  313–326. Springer, 2010.
  3. Disentangled variational autoencoder based multi-label classification with covariance-aware multivariate probit model. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI’20, 2021. ISBN 9780999241165.
  4. Demšar, J. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research, 7:1–30, 2006.
  5. A many-objective feature selection for multi-label classification. Knowledge-Based Systems, 208:106456, 2020.
  6. On the consistency of ranking algorithms. In Proceedings of the 27th International Conference on Machine Learning, ICML’10, pp.  327–334, Madison, WI, USA, 2010. Omnipress. ISBN 9781605589077.
  7. Dunn, O. J. Multiple comparisons among means. Journal of the American Statistical Association, 56:52–64, 1961.
  8. A review of the application of multiobjective evolutionary fuzzy systems: Current status and further directions. IEEE Transactions on Fuzzy systems, 21(1):45–65, 2012.
  9. On the consistency of multi-label learning. In Kakade, S. M. and von Luxburg, U. (eds.), Proceedings of the 24th Annual Conference on Learning Theory, volume 19 of Proceedings of Machine Learning Research, pp.  341–358, Budapest, Hungary, 09–11 Jun 2011. PMLR.
  10. Contrastive audio-visual masked autoencoder. In The Eleventh International Conference on Learning Representations, 2022.
  11. A survey of multi-label classification based on supervised and semi-supervised learning. International Journal of Machine Learning and Cybernetics, 14(3):697–724, 2023.
  12. Collaborative learning of label semantics and deep label-specific features for multi-label classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12):9860–9871, 2022a.
  13. Dual perspective of label-specific feature learning for multi-label classification. In International Conference on Machine Learning, pp. 8375–8386. PMLR, 2022b.
  14. Adapting arbitrary normal mutation distributions in evolution strategies: The covariance matrix adaptation. In Proceedings of IEEE International Conference on Evolutionary Computation, pp.  312–317. IEEE, 1996.
  15. Completely derandomized self-adaptation in evolution strategies. Evolutionary computation, 9(2):159–195, 2001.
  16. Learning label specific features for multi-label classification. In 2015 IEEE International Conference on Data Mining, pp. 181–190, 2015.
  17. Joint feature selection and classification for multilabel learning. IEEE Transactions on Cybernetics, 48(3):876–889, 2017.
  18. Covariance matrix adaptation for multi-objective optimization. Evolutionary Computation, 15(1):1–28, 2007.
  19. Laplace, P. S. Théorie analytique des probabilités. Courcier, 1814.
  20. Deep learning for extreme multi-label text classification. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.  115–124, 2017.
  21. The emerging trends of multi-label learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):7955–7974, 2021.
  22. Accurate detection of arrhythmias on raw electrocardiogram images: An aggregation attention multi-label model for diagnostic assistance. Medical Engineering & Physics, 114:103964, 2023. ISSN 1350-4533.
  23. Predicting label distribution from tie-allowed multi-label ranking. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp.  1–15, 2023.
  24. Topic-based instance and feature selection in multilabel classification. IEEE Transactions on Neural Networks and Learning Systems, 33(1):315–329, 2020.
  25. Warm starting CMA-ES for hyperparameter optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.  9188–9196, 2021.
  26. Evolutionary multi-objective multi-tasking for fuzzy genetics-based machine learning in multi-label classification. In 2022 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), pp.  1–8. IEEE, 2022.
  27. Modeling label space interactions in multi-label classification using box embeddings. In The Eleventh International Conference on Learning Representations, 2022.
  28. Learning symbolic model-agnostic loss functions via meta-learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp.  1–15, 2023.
  29. Classifier chains for multi-label classification. Machine Learning, 85:333–359, 2011.
  30. Learning representations by back-propagating errors. Nature, 323(6088):533–536, 1986.
  31. Explicit gradient learning for black-box optimization. In Proceedings of the 37th International Conference on Machine Learning, ICML’20, pp.  8480–8490, 2020.
  32. On the stratification of multi-label data. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011, Proceedings, Part III 22, pp.  145–158. Springer, 2011.
  33. Multi-task learning as multi-objective optimization. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  34. Multi-label classification based on multi-objective optimization. ACM Transactions on Intelligent Systems and Technology (TIST), 5(2):1–22, 2014.
  35. Revisiting facial age estimation with new insights from instance space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(5):2689–2697, 2020.
  36. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
  37. Multi-label classification with label graph superimposing. Proceedings of the AAAI Conference on Artificial Intelligence, pp.  12265–12272, 2020.
  38. Multi-label classification: do hamming loss and subset accuracy really conflict with each other? Advances in Neural Information Processing Systems, 33:3130–3140, 2020.
  39. Label-specific document representation for multi-label text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp.  466–475, 2019.
  40. Learning deep latent spaces for multi-label classification. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, pp.  2838–2844. AAAI Press, 2017.
  41. A multi-label feature selection algorithm based on multi-objective optimization. In 2015 International Joint Conference on Neural Networks (IJCNN), pp.  1–7. IEEE, 2015.
  42. Cross-modality attention with semantic graph embedding for multi-label classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.  12709–12716, 2020.
  43. Graph attention transformer network for multi-label image classification. ACM Transactions on Multimedia Computing, Communications and Applications, 19(4):1–16, 2023.
  44. Lift: Multi-label learning with label-specific features. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(1):107–120, 2014.
  45. ML-k𝑘kitalic_kNN: A lazy learning approach to multi-label learning. Pattern Recognition, 40(7):2038–2048, 2007.
  46. Zhang, T. Statistical analysis of some multi-category large margin classification methods. The Journal of Machine Learning Research, 5:1225–1251, 2004. ISSN 1532-4435.
  47. Multi-objective classification based on NSGA-II. International Journal of Computing Science and Mathematics, 9(6):539–546, 2018.
  48. Deep semantic dictionary learning for multi-label image classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.  3572–3580, 2021.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now