GeoJSEval: LLM Evaluation in WebGIS

• GeoJSEval is a multimodal, function-level, automated framework that quantitatively assesses LLM capabilities in generating accurate geospatial JavaScript code.
• It integrates standardized test suites, automated submission, execution, and judging pipelines to evaluate accuracy, stability, and resource efficiency.
• The framework systematically addresses WebGIS challenges by testing core libraries like Turf.js, Leaflet, and OpenLayers under diverse operational conditions.

GeoJSEval is a multimodal, function-level, automated evaluation framework for quantitatively assessing the ability of LLMs to synthesize JavaScript code in the domain of geospatial computation and visualization. Developed to meet the complex demands of WebGIS development, GeoJSEval systematically measures LLM capabilities across core JavaScript geospatial libraries, testing the robustness, efficiency, and semantic correctness of code outputs at execution time. GeoJSEval is characterized by rigorous test case design, a fully automated execution and judging pipeline, and comprehensive multidimensional metrics, establishing it as the foundational resource for evaluating LLM-driven spatial computing in frontend geospatial environments (Chen et al., 28 Jul 2025).

1. Motivation and Context

The proliferation of LLMs for automated code generation has introduced new paradigms for spatial computing and visual analytics, yet standard code evaluation benchmarks—such as HumanEval and MBPP—do not capture the intricate semantic, topological, and execution complexities characteristic of JavaScript-based WebGIS development. Geospatial computation and visualization tasks require correct handling of spatial reference systems, topological integrity, orchestration of fragmented JavaScript geospatial libraries (e.g., Turf.js for analysis, Leaflet and OpenLayers for map rendering), and strict integration with event-driven, asynchronous browser environments. These conditions frequently induce “hallucinations” in LLM-generated code, such as incorrect API usage, parameter misplacement, scale errors, and non-executable outputs. Without systematic, automated evaluation, AI-generated WebGIS applications risk instability and semantic error propagation, creating urgent demand for a domain-specialized assessment framework (Chen et al., 28 Jul 2025).

2. Framework Architecture and Core Components

GeoJSEval comprises three tightly integrated modules, each designed for a specific phase of the evaluation pipeline:

• GeoJSEval-Bench (Standardized Test Suite): A rigorously constructed suite of 432 unit-level tasks, derived from the APIs of five widely used JavaScript geospatial libraries (Turf.js, JSTS, Geolib, Leaflet, OpenLayers). Tasks are parameterized into 2,071 test cases, spanning 25 data types (primitive, structured, and complex geospatial objects including GeoJSON geometries, Features, FeatureCollections, and Map/Layer/Source objects). Each test case encapsulates three substructures:
  • Function Semantics: Specification of function headers and parameter types, plus hidden standard implementation for reference output.
  • Execution Configuration: Parameterization, expected output type, boundary (edge) case tagging, and I/O configuration.
  • Evaluation Parsing: Answer extraction and equivalence checking routines (direct value matching, feature extraction, topology comparison).
• Submission Program (Code Generation & Execution): Automates the interaction with LLMs, feeding only function signatures and JSDoc templates to elicit code completion (“Return ONLY the complete JavaScript function code.”). Generated code is executed in controlled Node.js/browser sandboxes, with specific parameters injected. Outputs may be captured as raw data or screenshots, supporting both computational and visual verification.
• Judge Program (Automated Evaluation): Applies output-type-dependent metrics to generated results, including exact comparison for primitives, set-based and structural matches for arrays/objects, and GeoJSON topological congruency using Shapely for spatial outputs. Resource metrics (token usage, response latency, code length) and comprehensive error logs (syntax, type, output type, logical/semantic mismatches, runtime errors) are centrally recorded (Chen et al., 28 Jul 2025).

3. Evaluation Methodology and Metrics

GeoJSEval implements a multidimensional set of quantitative metrics tailored to spatial code generation:

• Accuracy (pass@n): Probability at least one of $n$ generated candidates is correct, formalized as $$\mathrm{pass}@n = 1 - \frac{\binom{N-c}{n}}{\binom{N}{n}}$$, and reported for $n=1,3,5$.
• Output Stability (CV): Coefficient of variation $$\mathrm{CV} = \frac{\sigma(\mathrm{scores})}{\mu(\mathrm{scores})}$$ across multiple runs, penalizing volatility.
• Stability-Adjusted Accuracy (SA): Combines robustness and accuracy via $$\mathrm{SA} = \frac{\mathrm{pass}@5}{1+\mathrm{CV}}$$.
• Resource Consumption: Aggregates average token count per test (Tok), inference latency in seconds (In.T), and valid code length (Co.L).
• Operational Efficiency: Normalizes accuracy by resource use:
  • Inference Efficiency: $$E_\mathrm{time} = \frac{\mathrm{pass}@5}{\mathrm{In.T}}$$
  • Token Efficiency: $$E_\mathrm{token} = \frac{\mathrm{pass}@5}{\mathrm{Tok}}$$
  • Code-Line Efficiency: $$E_\mathrm{code} = \frac{\mathrm{pass}@5}{\mathrm{Co.L}}$$
• Error Type Distribution: Catalogues errors into SyntaxError, Type/AttributeError, OutputType mismatch, InvalidAnswer (logic error), RuntimeError, and “Other” (non-compliance with instructions) (Chen et al., 28 Jul 2025).

Boundary testing is employed for robustness evaluation; cases marked as edge_test synthesize extreme parameter values to verify LLM resilience under spatial or API-defined boundary conditions.

4. Dataset and Coverage

GeoJSEval-Bench stands out for its comprehensive breadth and expert-driven validation:

• Coverage: 432 canonical API tasks span the functional core of JavaScript geospatial libraries, including operations such as buffer, union, clip, distance computation, and advanced map/layer/view manipulation.
• Parameterization: 2,071 test cases, split among typical and boundary inputs, generated by a combination of multi-round LLM prompting (Claude Sonnet 4) and manual validation by GIS experts.
• Data Type Breadth: 25 geospatial types, encompassing primitive scalars, GeoJSON object hierarchies, and dynamic map and layer constructs.
• Boundary Robustness: Automated synthesis of edge cases such as empty feature sets, zero/maximum parameter values, and nonsensical call contexts to trigger failure modes (Chen et al., 28 Jul 2025).

This systematic approach enables both broad library coverage and deep semantic stress-testing within a machine-readable YAML schema for rapid extension and community reuse.

5. Experimental Results and Analysis

Comprehensive evaluation of 18 LLM instances (including general, reasoning-enhanced, code-centric, and geospatial-specialized) on GeoJSEval reveals:

• Performance Variance: Reasoning-enhanced models (DeepSeek-R1, o4-mini) achieve highest pass@5 (83.8%, 80.0%). General-purpose LLMs (GPT-4.1, mini) reach ∼74%; specialized GeoCode-GPT (fine-tuned on GEE) trails at ∼55%, indicating domain transfer issues.
• Stability: GPT-4.1 and mini variants show low output variance (CV ≈ 0.03); small models (e.g., Qwen-2.5-3B) are less consistent (CV > 0.12).
• Efficiency Trade-offs: Reasoning models incur high resource usage (15–40× token cost, tens of seconds latency). Code-centric models (Qwen-2.5-Coder-7B/32B) offer superior efficiency at moderate accuracy cost.
• Composite Efficiency Ranking: Qwen-3-32B and Qwen-2.5-Coder-32B present optimal trade-offs across efficiency metrics.
• Error Landscape: “InvalidAnswer” (logic/semantic mismatch) dominates (>50% of all failures), while syntax and type errors are relatively scarce, demonstrating that model bottlenecks are primarily semantic rather than syntactic.
• Boundary Pass Rates: Reasoning models outperform on boundary cases (∼71%) versus non-reasoning (61%), code-centric (58%), and GeoCode-GPT (41%), underlining the specificity of semantic reasoning for edge stability (Chen et al., 28 Jul 2025).

Multi-round generation (pass@3 vs. pass@1) yields diminishing returns after $n=3$. Model scale (e.g., parameters from 3B→32B) positively correlates with accuracy and robustness.

6. Extensibility and Related Frameworks

GeoJSEval’s modular task modeling (function-level with structured YAML definitions) allows for rapid integration of new libraries, extensions to composite workflow testing, and adaptation to non-JavaScript domains. Planned enhancements include automated DOM-based visual diffing for map rendering validation and backend language expansion (e.g., to Python/GeoPandas or PostGIS environments), establishing a cross-language benchmarking paradigm.

GeoJSEval is the conceptual forerunner of similar frameworks such as AutoGEEval++ (Hou et al., 12 Jun 2025) for GEE/Python and GeoSQL-Eval (Hou et al., 28 Sep 2025) for PostGIS GeoSQL query evaluation. These systems generalize the GeoJSEval pipeline—standardized function/task-level benchmarking, generation-to-execution-to-judging automation, multidimensional quantitative reporting, and fine-grained boundary/error analysis—to other vertical geospatial modalities and environments.

7. Real-World Applications and Impact

GeoJSEval supports:

• Automated QA for AI-generated WebGIS: Integration into CI/CD pipelines for libraries or geospatial apps, supporting continuous validation by triggering GeoJSEval tests on new code or model deployments.
• Smart-City and Environmental Portals: Routine verification of LLM-generated spatial analytics or visualization components.
• LLM Fine-tuning and Research: Ground-truth code and test-case datasets for supervised fine-tuning or retrieval-augmented generation in domain-specific models.
• Display Performance Benchmarking: Extension to 3D geospatial visualization stack assessment (e.g., 3D Tiles, Mapbox Vector Tiles) via standardized performance and user-experience metrics, as exemplified in (Seto et al., 27 Feb 2026).

GeoJSEval defines the state-of-the-art for LLM evaluation in frontend spatial computing and sets a cross-domain standard for future interoperable geospatial code benchmarks (Chen et al., 28 Jul 2025, Hou et al., 12 Jun 2025, Hou et al., 28 Sep 2025, Seto et al., 27 Feb 2026).