2000 character limit reached
Are Large Language Models Geospatially Knowledgeable? (2310.13002v1)
Published 9 Oct 2023 in cs.CL
Abstract: Despite the impressive performance of LLMs (LLM) for various natural language processing tasks, little is known about their comprehension of geographic data and related ability to facilitate informed geospatial decision-making. This paper investigates the extent of geospatial knowledge, awareness, and reasoning abilities encoded within such pretrained LLMs. With a focus on autoregressive LLMs, we devise experimental approaches related to (i) probing LLMs for geo-coordinates to assess geospatial knowledge, (ii) using geospatial and non-geospatial prepositions to gauge their geospatial awareness, and (iii) utilizing a multidimensional scaling (MDS) experiment to assess the models' geospatial reasoning capabilities and to determine locations of cities based on prompting. Our results confirm that it does not only take larger, but also more sophisticated LLMs to synthesize geospatial knowledge from textual information. As such, this research contributes to understanding the potential and limitations of LLMs in dealing with geospatial information.
- Ingwer Borg and Patrick JF Groenen. 2005. Modern multidimensional scaling: Theory and applications. Springer Science & Business Media.
- Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
- Analyzing commonsense emergence in few-shot knowledge models. arXiv:2101.00297.
- Evaluating the effectiveness of embeddings in representing the structure of geospatial ontologies. In International Conference on Geographic Information Science. Springer, 41–57.
- Curdin Derungs and Ross S Purves. 2016. Mining nearness relations from an n-grams Web corpus in geographical space. Spatial Cognition & Computation 16, 4 (2016), 301–322.
- Bert: Pre-training of deep bidirectional transformers for language understanding.
- Measuring Causal Effects of Data Statistics on Language Model’sFactual’Predictions. arXiv:2207.14251.
- Scaling laws for neural language models. arXiv:2001.08361 (2020).
- Anouk Lang. 2019. Spatial dialectics: Pursuing geospatial imaginaries with word embedding models and mapping. Modernism/Modernity Print Plus (2019).
- Xiang Lisa Li and Percy Liang. 2021. Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online, 4582–4597. https://doi.org/10.18653/v1/2021.acl-long.353
- Do Language Models Know the Way to Rome?. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP. Association for Computational Linguistics, Punta Cana, Dominican Republic, 510–517. https://doi.org/10.18653/v1/2021.blackboxnlp-1.40
- P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks.
- GPT understands, too.
- Efficient estimation of word representations in vector space.
- Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825–2830.
- Dissecting Contextual Word Embeddings: Architecture and Representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Brussels, Belgium, 1499–1509. https://doi.org/10.18653/v1/D18-1179
- Language Models as Knowledge Bases?. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, 2463–2473. https://doi.org/10.18653/v1/D19-1250
- python visualization. [n. d.]. Folium. https://python-visualization.github.io/folium/
- Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. https://nlp.stanford.edu/pubs/qi2020stanza.pdf
- Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
- Tara Safavi and Danai Koutra. 2021. Relational World Knowledge Representation in Contextual Language Models: A Review. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 1053–1067. https://doi.org/10.18653/v1/2021.emnlp-main.81
- Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca.
- Waldo R Tobler. 1970. A computer movie simulating urban growth in the Detroit region. Economic geography 46, sup1 (1970), 234–240.
- Llama: Open and efficient foundation language models.
- Shinji Umeyama. 1991. Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions on Pattern Analysis & Machine Intelligence 13, 04 (1991), 376–380.
- Attention is all you need. Advances in neural information processing systems 30 (2017).
- An overview of Microsoft Web N-gram corpus and applications. Proceedings of the NAACL HLT 2010 Demonstration Session (2010), 45–48.
- Self-Instruct: Aligning Language Model with Self Generated Instructions.
- CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data. In Proceedings of the Twelfth Language Resources and Evaluation Conference. European Language Resources Association, Marseille, France, 4003–4012. https://aclanthology.org/2020.lrec-1.494
- Huggingface’s transformers: State-of-the-art natural language processing. arXiv:1910.03771.
- Opt: Open pre-trained transformer language models.
- Prabin Bhandari (6 papers)
- Antonios Anastasopoulos (111 papers)
- Dieter Pfoser (17 papers)