Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Modelling Commonsense Commonalities with Multi-Facet Concept Embeddings (2403.16984v2)

Published 25 Mar 2024 in cs.AI and cs.CL

Abstract: Concept embeddings offer a practical and efficient mechanism for injecting commonsense knowledge into downstream tasks. Their core purpose is often not to predict the commonsense properties of concepts themselves, but rather to identify commonalities, i.e.\ sets of concepts which share some property of interest. Such commonalities are the basis for inductive generalisation, hence high-quality concept embeddings can make learning easier and more robust. Unfortunately, standard embeddings primarily reflect basic taxonomic categories, making them unsuitable for finding commonalities that refer to more specific aspects (e.g.\ the colour of objects or the materials they are made of). In this paper, we address this limitation by explicitly modelling the different facets of interest when learning concept embeddings. We show that this leads to embeddings which capture a more diverse range of commonsense properties, and consistently improves results in downstream tasks such as ultra-fine entity typing and ontology completion.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. A mixture-of-experts model for learning multi-facet entity embeddings. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5124–5135, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  2. Learning conceptual spaces with disentangled facets. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 131–139, Hong Kong, China. Association for Computational Linguistics.
  3. Genericskb: A knowledge base of generic statements. CoRR, abs/2005.00660.
  4. Automated generation of multilingual clusters for the evaluation of distributed representations. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net.
  5. Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4758–4781, Online. Association for Computational Linguistics.
  6. One of these words is not like the other: a reproduction of outlier identification using non-contextual word representations. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 120–130, Online. Association for Computational Linguistics.
  7. José Camacho-Collados and Roberto Navigli. 2016. Find the word that does not belong: A framework for an intrinsic evaluation of word vector representations. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pages 43–50, Berlin, Germany. Association for Computational Linguistics.
  8. Cabbage sweeter than cake? analysing the potential of large language models for learning conceptual spaces. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11836–11842, Singapore. Association for Computational Linguistics.
  9. Isolating sources of disentanglement in variational autoencoders. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 2615–2625.
  10. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 2172–2180.
  11. Ultra-fine entity typing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 87–96, Melbourne, Australia. Association for Computational Linguistics.
  12. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  13. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55–65, Hong Kong, China. Association for Computational Linguistics.
  14. Do neural language representations learn physical commonsense? In Proceedings of the 41th Annual Meeting of the Cognitive Science Society, CogSci 2019: Creativity + Cognition + Computation, Montreal, Canada, July 24-27, 2019, pages 1753–1759. cognitivesciencesociety.org.
  15. What do deck chairs and sun hats have in common? uncovering shared properties in large concept vocabularies. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10587–10596, Singapore. Association for Computational Linguistics.
  16. Modelling commonsense properties using pre-trained bi-encoders. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3971–3983, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  17. Is information extraction solved by chatgpt? an analysis of performance, evaluation criteria, robustness and errors. CoRR, abs/2305.14450.
  18. Acquiring and modelling abstract commonsense knowledge via conceptualization. CoRR, abs/2206.01532.
  19. An unsupervised neural attention model for aspect extraction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 388–397, Vancouver, Canada. Association for Computational Linguistics.
  20. beta-vae: Learning basic visual concepts with a constrained variational framework. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
  21. Unified semantic typing with meaningful label inference. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2642–2654, Seattle, United States. Association for Computational Linguistics.
  22. Learning disentangled representations of texts with application to biomedical abstracts. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4683–4693, Brussels, Belgium. Association for Computational Linguistics.
  23. Microsoft concept graph: Mining semantic concepts for short text understanding. Data Intell., 1(3):238–270.
  24. Hyunjik Kim and Andriy Mnih. 2018. Disentangling by factorising. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 2654–2663. PMLR.
  25. Novel views on novels: Embedding multiple facets of long texts. In WI-IAT ’21: IEEE/WIC/ACM International Conference on Web Intelligence, Melbourne VIC Australia, December 14 - 17, 2021, pages 670–675. ACM.
  26. Ontology completion using graph convolutional networks. In The Semantic Web - ISWC 2019 - 18th International Semantic Web Conference, Auckland, New Zealand, October 26-30, 2019, Proceedings, Part I, volume 11778 of Lecture Notes in Computer Science, pages 435–452. Springer.
  27. Ultra-fine entity typing with prior knowledge about labels: A simple clustering based strategy. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11744–11756, Singapore. Association for Computational Linguistics.
  28. Distilling semantic concept embeddings from contrastively fine-tuned language models. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023, Taipei, Taiwan, July 23-27, 2023, pages 216–226. ACM.
  29. Fast, effective, and self-supervised: Transforming masked language models into universal lexical and sentence encoders. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1442–1459, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  30. MirrorWiC: On eliciting word-in-context representations from pretrained language models. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 562–574, Online. Association for Computational Linguistics.
  31. Challenging common assumptions in the unsupervised learning of disentangled representations. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 4114–4124. PMLR.
  32. Don’t miss the labels: Label-semantic augmented meta-learner for few-shot text classification. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2773–2782, Online. Association for Computational Linguistics.
  33. Label semantics for few shot named entity recognition. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1956–1971, Dublin, Ireland. Association for Computational Linguistics.
  34. Taxoref: Embeddings evaluation for AI-driven taxonomy refinement. In Machine Learning and Knowledge Discovery in Databases. Research Track - European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part III, volume 12977 of Lecture Notes in Computer Science, pages 612–627. Springer.
  35. Semantic feature production norms for a large set of living and nonliving things. Behavior research methods, 37(4):547–559.
  36. A property induction framework for neural language models. CoRR, abs/2205.06910.
  37. Automatic noisy label correction for fine-grained entity typing. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 4317–4323. ijcai.org.
  38. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  39. Multifaceted domain-specific document embeddings. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations, pages 78–83, Online. Association for Computational Linguistics.
  40. Alex Rosenfeld and Katrin Erk. 2023. An analysis of property inference methods. Nat. Lang. Eng., 29(2):201–227.
  41. Sascha Rothe and Hinrich Schütze. 2016. Word embedding calculus in meaningful ultradense subspaces. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 512–517, Berlin, Germany. Association for Computational Linguistics.
  42. Subsumption prediction for e-commerce taxonomies. In The Semantic Web - 20th International Conference, ESWC 2023, Hersonissos, Crete, Greece, May 28 - June 1, 2023, Proceedings, volume 13870 of Lecture Notes in Computer Science, pages 244–261. Springer.
  43. Enriching taxonomies with functional domain knowledge. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, pages 745–754. ACM.
  44. LexFit: Lexical fine-tuning of pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5269–5283, Online. Association for Computational Linguistics.
  45. Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7222–7240, Online. Association for Computational Linguistics.
  46. CAR: Conceptualization-augmented reasoner for zero-shot commonsense question answering. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13520–13545, Singapore. Association for Computational Linguistics.
  47. CAT: A contextualized conceptualization and instantiation framework for commonsense reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13111–13140, Toronto, Canada. Association for Computational Linguistics.
  48. Adaptive cross-modal few-shot learning. In Proceedings of the Annual Conference on Neural Information Processing Systems, pages 4848–4858.
  49. Imposing label-relational inductive bias for extremely fine-grained entity typing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 773–784, Minneapolis, Minnesota. Association for Computational Linguistics.
  50. Aligning visual prototypes with BERT embeddings for few-shot learning. In Proceedings of the International Conference on Multimedia Retrieval, pages 367–375.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets