Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
91 tokens/sec
GPT-4o
12 tokens/sec
Gemini 2.5 Pro Pro
o3 Pro
5 tokens/sec
GPT-4.1 Pro
15 tokens/sec
DeepSeek R1 via Azure Pro
33 tokens/sec
Gemini 2.5 Flash Deprecated
12 tokens/sec
2000 character limit reached

Multi-Relational Hyperbolic Word Embeddings from Natural Language Definitions (2305.07303v5)

Published 12 May 2023 in cs.CL and cs.LG

Abstract: Natural language definitions possess a recursive, self-explanatory semantic structure that can support representation learning methods able to preserve explicit conceptual relations and constraints in the latent space. This paper presents a multi-relational model that explicitly leverages such a structure to derive word embeddings from definitions. By automatically extracting the relations linking defined and defining terms from dictionaries, we demonstrate how the problem of learning word embeddings can be formalised via a translational framework in Hyperbolic space and used as a proxy to capture the global semantic structure of definitions. An extensive empirical analysis demonstrates that the framework can help imposing the desired structural constraints while preserving the semantic mapping required for controllable and interpretable traversal. Moreover, the experiments reveal the superiority of the Hyperbolic word embeddings over the Euclidean counterparts and demonstrate that the multi-relational approach can obtain competitive results when compared to state-of-the-art neural models, with the advantage of being intrinsically more efficient and interpretable.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. Multi-relational poincaré graph embeddings. Advances in Neural Information Processing Systems, 32.
  2. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26.
  3. Tom Bosc and Pascal Vincent. 2018. Auto-encoding dictionary definitions into consistent word embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1522–1532.
  4. Multimodal distributional semantics. Journal of artificial intelligence research, 49:1–47.
  5. Learning disentangled representations for natural language definitions. Findings of the European chapter of Association for Computational Linguistics (Findings of EACL).
  6. Improving distributed representation of word sense via wordnet gloss composition and context clustering. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 15–20.
  7. Danilo Silva de Carvalho and Minh Le Nguyen. 2017. Building lexical vector representations from concept definitions. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 905–915.
  8. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186.
  9. Problems with evaluation of word embeddings using word similarity tasks. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pages 30–35.
  10. Christiane Fellbaum. 2010. Wordnet. In Theory and applications of ontology: computer applications, pages 231–243. Springer.
  11. Knowledge graph embedding by flexible translation. In Fifteenth International Conference on the Principles of Knowledge Representation and Reasoning.
  12. Placing search in context: The concept revisited. In Proceedings of the 10th international conference on World Wide Web, pages 406–414.
  13. Conditional generators of words definitions. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 266–271.
  14. Hyperbolic neural networks. Advances in neural information processing systems, 31.
  15. Simverb-3500: A large-scale evaluation set of verb similarity. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2173–2182.
  16. Learning to understand phrases by embedding the dictionary. Transactions of the Association for Computational Linguistics, 4:17–30.
  17. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695.
  18. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 873–882.
  19. Matthias Leimeister and Benjamin J Wilson. 2018. Skip-gram word embeddings in hyperbolic space. arXiv preprint arXiv:1809.01498.
  20. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  21. Daniel Loureiro and Alipio Jorge. 2019. Language modelling makes sense: Propagating representations through wordnet for full-coverage word sense disambiguation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5682–5691.
  22. Semeval-2022 task 1: Codwoe–comparing dictionaries and word embeddings. arXiv preprint arXiv:2205.13858.
  23. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
  24. George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41.
  25. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1864–1874.
  26. Maximillian Nickel and Douwe Kiela. 2017. Poincaré embeddings for learning hierarchical representations. Advances in neural information processing systems, 30.
  27. Definition modeling: Learning to define word embeddings in natural language. In Thirty-First AAAI Conference on Artificial Intelligence.
  28. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.
  29. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1–67.
  30. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992.
  31. Herbert Rubenstein and John B Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633.
  32. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
  33. Improving word embedding compositionality using lexicographic definitions. In Proceedings of the 2018 World Wide Web Conference, pages 1083–1093.
  34. Drg2vec: Learning word representations from definition relational graph. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–9. IEEE.
  35. Building a knowledge graph from natural language definitions for interpretable text entailment recognition. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
  36. Categorization of semantic roles for dictionary definitions. In Proceedings of the 5th Workshop on Cognitive Aspects of the Lexicon (CogALex-V), pages 176–184.
  37. Exploring knowledge graphs in an interpretable composite approach for text entailment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7023–7030.
  38. Recognizing and justifying text entailment through distributional navigation on definition graphs. In Thirty-Second AAAI Conference on Artificial Intelligence.
  39. Mpnet: Masked and permuted pre-training for language understanding. Advances in Neural Information Processing Systems, 33:16857–16867.
  40. Poincare glove: Hyperbolic word embeddings. In International Conference on Learning Representations.
  41. Dict2vec: Learning word embeddings using lexical dictionaries. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 254–263.
  42. Defsent: Sentence embeddings using definition sentences. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 411–418.
  43. Abraham A Ungar. 2001. Hyperbolic trigonometry and its application in the poincaré ball model of hyperbolic geometry. Computers & Mathematics with Applications, 41(1-2):135–147.
  44. Qitong Wang and Mohammed J Zaki. 2022. Hg2vec: Improved word embeddings from dictionary and thesaurus based heterogeneous graph. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3154–3163.
  45. Using wiktionary for computing semantic relatedness. In AAAI, volume 8, pages 861–866.
  46. Manifold learning-based word representation refinement incorporating global and local information. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3401–3412.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube