GeoJSEval-Bench: JS Benchmark for Geospatial Code

• GeoJSEval-Bench is a comprehensive benchmark that evaluates LLM-generated JavaScript functions for geospatial computation and visualization tasks.
• It employs a tri-modular architecture with a test suite, submission engine, and a judge module to automate function-level evaluation across diverse geospatial libraries.
• The framework’s extensive coverage of 432 functions and 2,071 test cases ensures reproducible, fine-grained assessment of code accuracy, efficiency, and robustness.

GeoJSEval-Bench is the function-level unit test suite within the GeoJSEval framework, designed to systematically evaluate LLMs on the generation of JavaScript code for geospatial computation and visualization tasks. GeoJSEval-Bench addresses the need for standardized, reproducible, and fine-grained benchmarking of LLMs in this technically demanding application domain, characterized by complex library APIs, heterogeneous data types, and stringent requirements for correctness, robustness, and efficiency (Chen et al., 28 Jul 2025).

1. Design Objectives and Scope

GeoJSEval-Bench was developed with the explicit goal of enabling fine-grained, automated assessment of LLM-generated code targeting real-world JavaScript geospatial APIs. The suite focuses on both spatial analysis functions (such as buffering, clipping, and distance calculations) and map visualization operations (such as layer management and view control), covering end-to-end scenarios typical in geospatial information systems (GIS) and web-mapping environments.

Key attributes:

• Function-Level Granularity: Each test targets a single API function, isolating capabilities in code synthesis, parameterization, and output handling.
• Extensive Coverage: The suite comprises 432 core function-level tasks, spanning five widely adopted JavaScript geospatial libraries—Turf.js, JSTS, Geolib, Leaflet, and OpenLayers.
• Diverse Inputs and Data Types: 2,071 structured test cases are parameterized with both typical and edge-case inputs, covering 25 mainstream geospatial data types, including GeoJSON geometries, map objects, and primitives.
• Full Automation and Extensibility: Designed for seamless integration with the GeoJSEval submission and evaluation infrastructure to support continuous benchmarking and future expansion.

This focus on breadth, automation, and multidimensional granularity establishes GeoJSEval-Bench as a foundational evaluation resource for geospatial code generation model development and assessment.

2. Architecture and Key Components

GeoJSEval-Bench is embedded within a tri-modular system architecture, consisting of the following principal modules:

A. GeoJSEval-Bench (Test Suite)

• Test Case Definition: Each case specifies a FunctionHeader (JSDoc signature), canonical StandardCode, a parameterized ParameterList, expected OutputType, optional EdgeTest flag, auxiliary EvalMethods for result extraction, prescribed OutputPath, and precise ExpectedAnswer.
• Task Modularization: Tasks are organized via three sub-modules handling function semantics, execution configuration, and evaluation parsing, facilitating systematic test execution and result logging.

B. Submission Program (Code Submission Engine)

• Prompting Protocol: LLMs are prompted using only the JSDoc-style function header and a fixed-generation template.
• Input Injection: Values from ParameterList are substituted at invocation time; any required auxiliary method calls (as specified in EvalMethods) are appended before execution.
• Sandboxed Execution: Returned code is executed in a controlled JavaScript runtime, with outputs persisted to OutputPath.

C. Judge Program (Evaluation Module)

• Flexible Evaluation: Judge logic dynamically selects direct comparison, set-based, GeoJSON-aware, or indirect evaluation strategies based on OutputType and EvalMethods.
• Pass/Fail Aggregation: Success is determined by equivalence with ExpectedAnswer (which may involve additional output parsing or method application), and extensive logs are maintained for error categorization and metric computation.

This modular design enables precise, scalable testing while ensuring compatibility with new libraries, datasets, and future code-generation paradigms.

3. Test Suite Breakdown

3.1. Function-Level Tasks

GeoJSEval-Bench systematically samples 432 API functions from the documentation of Turf.js, JSTS, Geolib, Leaflet, and OpenLayers, covering both geospatial computation and visualization primitives.

Each function is modeled as a mapping:

$$M: \mathrm{FunctionHeader}\ H \rightarrow \mathrm{Code}\ C \rightarrow y = \mathrm{Exec}(C) \rightarrow \mathrm{Judge}(y, a)$$

with indicator $L(q)$:

$$L(q) = \begin{cases} 1, & \text{if } \mathrm{Exec}(M(H),i)\equiv a\ 0, & \text{otherwise} \end{cases}$$

3.2. Structured Test Cases

Each function is tested on 2,071 structured test cases:

• Normal Inputs: 3 canonical parameterizations per function.
• Boundary/Edge Inputs: 2 adversarial cases per function when EdgeTest=true, designed to probe error handling and robustness.

3.3. Functional Categories and Data Types

• Computation/Analysis: Tasks using Turf.js, JSTS, Geolib—examples include geometric transformations, measurements, and spatial predicates.
• Visualization/Interaction: Tasks with Leaflet and OpenLayers—focusing on map state, layer/style management, view adjustment, and event handling.
• Data Types: 25 supported types, including primitives (number, boolean), arrays, GeoJSON (Geometry, Feature, FeatureCollection), and complex library-specific objects (leaflet.Map, ol.Map, etc.).

Table: Data Type Coverage (Excerpt)

Output Type	Description
Number, Boolean, String	Primitives
Array	Arrays of numbers/objects
GeoJSON Geometry	Points, lines, polygons, etc.
Feature/FeatureCollection	GeoJSON features/collections
leaflet.Map/Layer	Complex Leaflet objects
ol.Map/View/Layer	OpenLayers objects

4. Evaluation Metrics and Error Diagnosis

GeoJSEval-Bench implements a multidimensional analytical protocol:

4.1. Accuracy and Stability

• pass@n: Probability that at least one of $n$ generated codes is correct

$$\mathrm{pass@}n = 1 - \left(1 - \frac{c}{N}\right)^n$$

• Coefficient of Variation (CV): Consistency across multiple runs

$\mathrm{CV} = \sigma/\mu$

• Stability-Adjusted Accuracy (SA):

$\mathrm{SA} = \mathrm{pass@5}\;/\;(1+\mathrm{CV})$

4.2. Efficiency and Resource Consumption

• Inference Efficiency:

$$\text{Inference Efficiency} = \frac{\mathrm{pass@}n}{T_{\mathrm{inf}}}$$

• Token Efficiency:

$\frac{\mathrm{pass@}n}{\text{Tokens}}$

• Code-Line Efficiency:

$\frac{\mathrm{pass@}n}{\text{LinesOfCode}}$

4.3. Error Type Distribution

Automatically categorized as:

1. Syntax Error
2. Attribute/Parameter Error
3. OutputType Error
4. Invalid Answer (logical mismatch)
5. Runtime Error (timeout/infinite loop)
6. Other (e.g., redundant imports)

Each type's frequency is captured as:

$L(q)$0

4.4. Boundary Testing

Cases flagged as edge_test dynamically trigger boundary-value parameterizations (e.g., zero-length geometries, out-of-range coordinates), with model robustness quantified by accuracy under these conditions.

5. Workflow: Test Submission and Judging

The evaluation pipeline operates as follows:

1. For each test case, the function header is used to prompt the LLM using a strict template.
2. The returned code is invoked with fixed parameters (normal or edge).
3. For complex outputs, auxiliary evaluation methods extract features or properties for comparison.
4. Results are parsed and compared to the canonical expected output; verdicts and errors are logged.

Pseudocode (excerpt)

$L(q)$1

This fully automated process supports continuous evaluation and scaling to new tasks, libraries, or LLM backends.

6. Sample Test Cases and Auxiliary Evaluation

GeoJSEval-Bench employs both direct and indirect evaluation:

• Direct: Output primitives are directly compared (e.g., a numerical return value).
• Indirect: For objects (e.g., map states), auxiliary methods extract features (e.g., zoom level, layer opacity) for judgment.

Auxiliary Methods Table (Excerpt)

Object	Method	Extracted Feature
map	map.getCenter()	Center coordinates
map	map.getZoom()	Zoom level
layer	layer.getOpacity()	Opacity value
source	source.getFeatureCount()	Number of features

This flexible strategy generalizes across diverse output types and libraries.

7. Significance and Applications

GeoJSEval-Bench constitutes the first comprehensive, multimodal, and automated function-level benchmark for evaluating LLMs on JavaScript-based geospatial code generation (Chen et al., 28 Jul 2025). Its principal contributions are:

• Establishing standardized, reproducible protocols and metrics for function-level geospatial code benchmarking.
• Enabling identification of performance disparities, robustness gaps, and persistent bottlenecks in LLM spatial understanding and code synthesis.
• Supporting extensibility to new libraries, data types, and task categories, facilitating practical optimization of LLMs for real-world GIS and web-mapping contexts.

A plausible implication is that frameworks like GeoJSEval-Bench will drive advances not just in code generation quality, but also in LLM interpretability, safety, and deployment for scientific, engineering, and enterprise geospatial applications.