Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SemSup-XC: Semantic Supervision for Zero and Few-shot Extreme Classification (2301.11309v2)

Published 26 Jan 2023 in cs.CL

Abstract: Extreme classification (XC) involves predicting over large numbers of classes (thousands to millions), with real-world applications like news article classification and e-commerce product tagging. The zero-shot version of this task requires generalization to novel classes without additional supervision. In this paper, we develop SemSup-XC, a model that achieves state-of-the-art zero-shot and few-shot performance on three XC datasets derived from legal, e-commerce, and Wikipedia data. To develop SemSup-XC, we use automatically collected semantic class descriptions to represent classes and facilitate generalization through a novel hybrid matching module that matches input instances to class descriptions using a combination of semantic and lexical similarity. Trained with contrastive learning, SemSup-XC significantly outperforms baselines and establishes state-of-the-art performance on all three datasets considered, gaining up to 12 precision points on zero-shot and more than 10 precision points on one-shot tests, with similar gains for recall@10. Our ablation studies highlight the relative importance of our hybrid matching module and automatically collected class descriptions.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. Multi-label learning with millions of labels: recommending advertiser bid phrases for web pages. Proceedings of the 22nd international conference on World Wide Web, 2013.
  2. Dismec: Distributed sparse machines for extreme multi-label classification. Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, 2017.
  3. Supervised semantic indexing. In Proceedings of the 18th ACM conference on Information and knowledge management, pp.  187–196, 2009.
  4. Extreme classification (dagstuhl seminar 18291). In Dagstuhl Reports, volume 8. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2019.
  5. Sparse local embeddings for extreme multi-label classification. In NIPS, 2015.
  6. Large-scale multi-label text classification on eu legislation. In ACL, 2019.
  7. A modular deep learning approach for extreme multi-label text classification. ArXiv, abs/1905.02331, 2019.
  8. Taming pretrained transformers for extreme multi-label text classification. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp.  3163–3171, 2020.
  9. Siamesexml: Siamese networks meet extreme classifiers with 100m labels. In International Conference on Machine Learning, pp. 2330–2340. PMLR, 2021a.
  10. Deepxml: A deep extreme multi-label learning framework applied to short text documents. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pp.  31–39, 2021b.
  11. Zero-shot learning and clustering for semantic utterance classification. In ICLR (Poster), 2014.
  12. BERT: pre-training of deep bidirectional transformers for language understanding. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 4171–4186. Association for Computational Linguistics, 2019a. doi: 10.18653/v1/n19-1423. URL https://doi.org/10.18653/v1/n19-1423.
  13. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019b.
  14. Splade: Sparse lexical and expansion model for first stage ranking. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021.
  15. Friedland, B. profanity: A python library to check for (and clean) profanity in strings, 2013. URL https://github.com/ben174/profanity.
  16. COIL: revisit exact lexical match in information retrieval with contextualized inverted list. In Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tür, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., and Zhou, Y. (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp.  3030–3042. Association for Computational Linguistics, 2021a. doi: 10.18653/v1/2021.naacl-main.241. URL https://doi.org/10.18653/v1/2021.naacl-main.241.
  17. Simcse: Simple contrastive learning of sentence embeddings. ArXiv, abs/2104.08821, 2021b.
  18. Grandury, M. roberta-base-finetuned-sms-spam-detection, 2021. URL https://huggingface.co/mariagrandury/roberta-base-finetuned-sms-spam-detection.
  19. Generalized zero-shot extreme multi-label learning. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021.
  20. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pp.  1735–1742. IEEE, 2006.
  21. Semantic supervision: Enabling generalization over output spaces. ArXiv, abs/2202.13100, 2022.
  22. Slice: Scalable linear extreme classifiers trained on 100 million labels for related searches. Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, 2019.
  23. Lightxml: Transformer with dynamic negative sampling for high-performance extreme multi-label text classification. In AAAI, 2021.
  24. Bonsai: diverse and shallow trees for extreme multi-label classification. Machine Learning, pp.  1 – 21, 2020.
  25. Zero-data learning of new tasks. In AAAI, volume 1, pp.  3, 2008.
  26. Latent retrieval for weakly supervised open domain question answering. ArXiv, abs/1906.00300, 2019.
  27. Deduplicating training data makes language models better. In ACL, 2022.
  28. Multi-label classification via feature-aware implicit label space encoding. In ICML, 2014.
  29. Deep learning for extreme multi-label text classification. Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2017.
  30. Decoupled weight decay regularization. In ICLR, 2019.
  31. Hidden factors and hidden topics: understanding rating dimensions with review text. Proceedings of the 7th ACM conference on Recommender systems, 2013.
  32. Extreme classification in log memory using count-min sketch: A case study of amazon search with 50m products. Advances in Neural Information Processing Systems, 32, 2019.
  33. Decaf: Deep extreme classification with label features. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pp.  49–57, 2021.
  34. All-in text: Learning document, label, and word representations jointly. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
  35. Gile: A generalized input-label embedding for text classification. Transactions of the Association for Computational Linguistics, 7:139–155, 2019.
  36. Parabel: Partitioned label trees for extreme classification with application to dynamic search advertising. pp.  993–1002, 04 2018. ISBN 978-1-4503-5639-8. doi: 10.1145/3178876.3185998.
  37. Exploring the limits of transfer learning with a unified text-to-text transformer, 2019. URL https://arxiv.org/abs/1910.10683.
  38. A cluster-based approach for improving isotropy in contextual embedding space. In ACL, 2021.
  39. Sentence-bert: Sentence embeddings using siamese bert-networks. ArXiv, abs/1908.10084, 2019.
  40. Few-shot and zero-shot multi-label learning for structured label spaces. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, volume 2018, pp.  3132. NIH Public Access, 2018.
  41. Open vocabulary extreme classification using generative models. ArXiv, abs/2205.05812, 2022.
  42. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. ArXiv, abs/2104.08663, 2021.
  43. Well-read students learn better: On the importance of pre-training compact models. arXiv: Computation and Language, 2019.
  44. Joint embedding of words and labels for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.  2321–2331, 2018.
  45. Eda: Easy data augmentation techniques for boosting performance on text classification tasks, 2019. URL https://arxiv.org/abs/1901.11196.
  46. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp.  1112–1122. Association for Computational Linguistics, 2018. URL http://aclweb.org/anthology/N18-1101.
  47. A no-regret generalization of hierarchical softmax to extreme multi-label classification. In NeurIPS, 2018.
  48. Extreme zero-shot learning for extreme text classification. ArXiv, abs/2112.08652, 2022.
  49. mt5: A massively multilingual pre-trained text-to-text transformer. In NAACL, 2021.
  50. Ppdsparse: A parallel primal-dual sparse method for extreme classification. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2017.
  51. Attentionxml: Label tree-based attention-aware deep model for high-performance extreme multi-label text classification. In NeurIPS, 2019.
  52. Fast multi-resolution transformer fine-tuning for extreme multi-label text classification. Advances in Neural Information Processing Systems, 34:7267–7280, 2021.
  53. Metadata-induced contrastive learning for zero-shot multi-label text classification. In Proceedings of the ACM Web Conference 2022, pp. 3162–3173, 2022.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now