2000 character limit reached
Block-Diagonal Orthogonal Relation and Matrix Entity for Knowledge Graph Embedding (2401.05967v3)
Published 11 Jan 2024 in cs.CL and cs.IR
Abstract: The primary aim of Knowledge Graph embeddings (KGE) is to learn low-dimensional representations of entities and relations for predicting missing facts. While rotation-based methods like RotatE and QuatE perform well in KGE, they face two challenges: limited model flexibility requiring proportional increases in relation size with entity dimension, and difficulties in generalizing the model for higher-dimensional rotations. To address these issues, we introduce OrthogonalE, a novel KGE model employing matrices for entities and block-diagonal orthogonal matrices with Riemannian optimization for relations. This approach enhances the generality and flexibility of KGE models. The experimental results indicate that our new KGE model, OrthogonalE, is both general and flexible, significantly outperforming state-of-the-art KGE models while substantially reducing the number of relation parameters.
- Pierre Ablin and Gabriel Peyré. 2022. Fast and accurate optimization on the orthogonal manifold without retraction. In International Conference on Artificial Intelligence and Statistics, pages 5636–5657. PMLR.
- Trust-region methods on riemannian manifolds. Foundations of Computational Mathematics, 7:303–330.
- Optimization algorithms on matrix manifolds. Princeton University Press.
- P-A Absil and Jérôme Malick. 2012. Projection-like retractions on matrix manifolds. SIAM Journal on Optimization, 22(1):135–158.
- Multi-relational poincaré graph embeddings. Advances in Neural Information Processing Systems, 32.
- Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247–1250.
- Silvere Bonnabel. 2013. Stochastic gradient descent on riemannian manifolds. IEEE Transactions on Automatic Control, 58(9):2217–2229.
- Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26.
- Minhyung Cho and Jaehyung Lee. 2017. Riemannian approach to batch normalization. Advances in Neural Information Processing Systems, 30.
- Convolutional 2d knowledge graph embeddings. Proceedings of the AAAI conference on artificial intelligence, 32(1).
- Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7).
- An end-to-end model for question answering over knowledge base with cross-attention combining global knowledge. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 221–231.
- Orthogonal recurrent neural networks with scaled cayley transform. In International Conference on Machine Learning, pages 1969–1978. PMLR.
- Knowledge graph embedding via dynamic mapping matrix. In Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: Long papers), pages 687–696.
- Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
- Geoopt: Riemannian optimization in pytorch. arXiv preprint arXiv:2005.02819.
- Mario Lezcano-Casado and David Martınez-Rubio. 2019. Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group. In International Conference on Machine Learning, pages 3794–3803. PMLR.
- Learning entity and relation embeddings for knowledge graph completion. Proceedings of the AAAI conference on artificial intelligence, 29(1).
- Estelle Massart and Vinayak Abrol. 2022. Coordinate descent on the orthogonal group for recurrent neural network training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7744–7751.
- George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41.
- Riemannian bfgs algorithm with applications. In Recent Advances in Optimization and its Applications in Engineering: The 14th Belgian-French-German Conference on Optimization, pages 183–192. Springer.
- Yago: a core of semantic knowledge. In Proceedings of the 16th international conference on World Wide Web, pages 697–706.
- Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv preprint arXiv:1902.10197.
- Poincar\\\backslash\’e glove: Hyperbolic word embeddings. arXiv preprint arXiv:1810.06546.
- Kristina Toutanova and Danqi Chen. 2015. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd workshop on continuous vector space models and their compositionality, pages 57–66.
- Complex embeddings for simple link prediction. In International conference on machine learning, pages 2071–2080. PMLR.
- On orthogonality and learning recurrent networks with long term dependencies. In International Conference on Machine Learning, pages 3570–3578. PMLR.
- Knowledge graph embedding by translating on hyperplanes. Proceedings of the AAAI conference on artificial intelligence, 28(1).
- Full-capacity unitary recurrent neural networks. Advances in neural information processing systems, 29.
- Explicit semantic ranking for academic search via knowledge graph embedding. In Proceedings of the 26th international conference on world wide web, pages 1271–1279.
- Bishan Yang and Tom Mitchell. 2019. Leveraging knowledge bases in lstms for improving machine reading. arXiv preprint arXiv:1902.09091.
- Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575.
- Collaborative knowledge base embedding for recommender systems. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 353–362.
- Hongyi Zhang and Suvrit Sra. 2016. First-order methods for geodesically convex optimization. In Conference on Learning Theory, pages 1617–1638. PMLR.
- Quaternion knowledge graph embeddings. Advances in neural information processing systems, 32.
- Yihua Zhu (4 papers)
- Hidetoshi Shimodaira (45 papers)