Evaluating the Effectiveness of Large Language Models in Representing Textual Descriptions of Geometry and Spatial Relations (2307.03678v1)

Abstract: This research focuses on assessing the ability of LLMs in representing geometries and their spatial relations. We utilize LLMs including GPT-2 and BERT to encode the well-known text (WKT) format of geometries and then feed their embeddings into classifiers and regressors to evaluate the effectiveness of the LLMs-generated embeddings for geometric attributes. The experiments demonstrate that while the LLMs-generated embeddings can preserve geometry types and capture some spatial relations (up to 73% accuracy), challenges remain in estimating numeric values and retrieving spatially related objects. This research highlights the need for improvement in terms of capturing the nuances and complexities of the underlying geospatial data and integrating domain knowledge to support various GeoAI applications using foundation models.

• The paper demonstrates that LLMs achieve 100% accuracy in geometry type classification using WKT descriptors.
• The study applies a structured GIS workflow combined with LLM embeddings to assess area, centroid, and spatial predicate tasks.
• The results reveal strong performance in categorical tasks but significant limitations in precise numeric and spatial computations.

The paper "Evaluating the Effectiveness of LLMs in Representing Textual Descriptions of Geometry and Spatial Relations" investigates the capability of LLMs, specifically GPT-2 and BERT, in encoding geometrical and spatial relation data expressed in text. The primary focus of this research is on evaluating how well these models can represent geometries encoded in a Well-Known Text (WKT) format and how effectively they can facilitate downstream tasks related to geospatial data.

Methodology

The paper adopts a structured workflow combining traditional GIS tools and LLM-based embeddings for evaluation. The workflow is divided into three key stages:

1. Extraction of Geometrical Attributes: Using GIS tools to extract relevant attributes such as geometry type, centroid, area, and spatial relations (e.g., predicates and distances).
2. Encoding with LLMs: The geometries are encoded using LLMs, with an emphasis on maintaining the integrity of the WKT format, which specifies geometry type and ordered coordinates.
3. Task-Specific Evaluation: The embeddings generated by LLMs are subjected to various classification and regression tasks to evaluate their effectiveness in encoding and representing geometric attributes.

The following main evaluation tasks are conducted:

• Geometric Attributes:
  • Geometry type classification.
  • Area computation.
  • Centroid derivation.
• Spatial Relations:
  • Spatial predicate classification.
  • Distance measure.
  • Location prediction through spatial predicates.

Experimentation

The experiments utilize a multi-sourced dataset of geospatial objects from Madison, Wisconsin, with samples from OpenStreetMap, SLIPO POIs, and Microsoft Building Footprints. The paper employs common predicates as defined by the Open Geospatial Consortium (OGC) for spatial relations and tests on a dataset split into training, validation, and test sets.

Results and Analysis

• Geometry Type Classification: Both GPT-2 and BERT achieve 100% accuracy in classifying geometry types, as this task highly aligns with tokenization of known geometry descriptor words.
• Area Computation and Centroid Derivation: LLMs reveal limitations in accurate numeric representation, showing a high Mean Absolute Percentage Error (MAPE) for area computation, especially when including LineString and Point types, which should have an area of zero. Centroid derivation suffers from geographical inaccuracies, as reflected by Root Mean Square Error (RMSE).
• Spatial Predicate Classification: Including geometry types enhances accuracy to over 73%, suggesting LLMs can leverage geometry attributes to inform spatial relations.
• Distance Measure: Encounters difficulty due to the complexity of numeric estimations within embeddings, particularly highlighted when limited to "disjoint" predicates.
• Location Prediction: The precision of spatial context retrieval remains inadequate, emphasizing deficiencies in retrieving spatially related objects based solely on embeddings.

The results underscore the LLMs' capability to detect geometry types and some relations but exhibit profound challenges in quantifying geometric and spatial features. The intrinsic tokenization mechanism of LLMs may cause a dilution of geometric details essential for precise computation. The paper suggests room for improvement in LLM architecture to enhance performance in applications concerning GeoAI by perhaps integrating more explicit spatial reasoning processes or domain-specific modifications like alternative notations and cognitive-influenced prompting methods.

In conclusion, while LLMs provide a promising foundation for semantic understanding of geospatial data, they fall short in high-precision spatial and geometric computation tasks, necessitating continued refinement to incorporate intricate geospatial information accurately. The findings indicate crucial avenues for R&D in developing more adept models for geospatial AI tasks.