GeoQuery: A Semantic Parsing Benchmark

Updated 4 July 2026

GeoQuery is a semantic-parsing benchmark that maps natural language geography questions to formal queries using Prolog, FunQL, and SQL representations.
It supports diverse configurations and evaluation protocols that assess compositional generalization, data augmentation strategies, and structured inference methods.
GeoQuery has driven innovations in semantic parsing, including in-context prompting, certified abstention, and multitask learning for improved performance.

Searching arXiv for recent and relevant papers on GeoQuery and associated semantic parsing work. GeoQuery is a semantic-parsing benchmark centered on questions about U.S. geography, in which a natural-language utterance is mapped to a formal query or logical form. Across the literature, the benchmark appears in several closely related forms: GeoQuery questions paired with Prolog queries, variable-free FunQL trees, or SQL over a fixed database schema; it is also evaluated under standard train/test splits, compositional splits, multilingual settings, few-shot prompting regimes, and incomplete-question variants such as prefix-to-SQL (Ziai, 2019, Jie et al., 2018, Yang et al., 2022, Rajkumar et al., 2022). The literature therefore presents GeoQuery not as a single immutable artifact, but as a compact, high-leverage testbed for semantic representation, structured inference, data augmentation, compositional generalization, and prompt-based adaptation.

1. Core task and representational variants

In its most direct formulation, GeoQuery asks for translation from a geography question into a machine-interpretable meaning representation. One line of work describes the dataset as GeoQuery questions paired with Prolog queries, with examples such as “what states border Texas?” mapped to a logical form beginning answer(NV, state(V0), next_to(V0,NV), const(V0, stateid(texas))) (Ziai, 2019). Another line uses variable-free FunQL, treating the meaning representation as a tree and reconstructing Prolog only after semantic parsing; an English example is “What rivers do not run through Tennessee?” with gold meaning representation answer(exclude(river(all), traverse(stateid('tn')))) (Jie et al., 2018). A third line reformulates GeoQuery as text-to-SQL over a single, fixed database, where questions such as “What is the population of Austin?” are translated into executable SQL (Rajkumar et al., 2022).

The representational diversity is technically significant. FunQL-based work emphasizes tree structure, semantic units, and composition of predicates such as largest, state, and population (Yang et al., 2022). SQL-based work emphasizes schema conditioning, joins, aggregation, and execution correctness, with one paper explicitly listing eight tables such as STATE, CITY, RIVER, and BORDER_INFO (Rajkumar et al., 2022). Prefix-oriented work further extends the task from full-question parsing to incomplete-question prediction, defining prefix-to-SQL as a setting in which only a user question prefix is available and the system predicts the intended SQL (Deng et al., 2021).

A recurring misconception is to treat GeoQuery as a single fixed formalism. The literature instead uses GeoQuery as a stable domain paired with multiple output languages and multiple supervision regimes, which makes results comparable only when the representation, split, and metric are specified.

2. Dataset configurations and evaluation protocols

Reported dataset sizes and splits depend on the formulation. The following configurations are explicitly reported in the literature.

Configuration	Split / size reported	Representation
Multilingual semantic parsing	880 total utterance–MR pairs in eight languages; standard 600/280 train/test split	FunQL (Jie et al., 2018)
Logical-form generation	600 examples for training; 280 for testing	Prolog queries (Ziai, 2019)
Compositional SQL parsing	536 train / 159 dev / 182 test	SQL (Yang et al., 2022)
Question and Query splits	513/57/256 and 519/54/253	FunQL (Yang et al., 2022)
Few-shot text-to-SQL	880 English↔SQL pairs in total; 549 used for training/few-shot support	SQL (Rajkumar et al., 2022)

Evaluation is correspondingly heterogeneous. For logical-form generation, papers report parsing or sequence accuracy and token accuracy, with sequence accuracy defined as the fraction of test logical forms exactly matching the gold string (Ziai, 2019). For multilingual FunQL parsing, reported metrics include exact answer accuracy and query-execution $F_1$ score (Jie et al., 2018). Text-to-SQL work often reports execution accuracy or test-suite accuracy, reflecting semantic equivalence despite surface differences in SQL (Wang et al., 2018, Rajkumar et al., 2022). Prefix-to-SQL introduces SAVE, with $\text{Save@K}$ defined as the fraction of the full question not typed once the intended SQL appears in the top- $K$ suggestions (Deng et al., 2021).

This metric diversity matters for interpretation. Exact-match improvements on one version of GeoQuery are not automatically comparable with execution gains on another, and SQL settings may reward denotational equivalence that string-level FunQL evaluation does not.

3. Structured inference and symbolic bias

GeoQuery has been a focal benchmark for parsers that encode strong structural constraints. One early direction introduced a type-driven incremental parser with three explicit claims: a linear-time incremental shift-reduce-style semantic parsing algorithm, type-driven rather than syntax-driven reduction, and an enriched type system using subtype polymorphism and parametric polymorphism; the abstract states that the system learns very accurate parses in GeoQuery, Jobs and Atis domains (Zhao et al., 2014). The technical emphasis is on type checking as the mechanism that determines reduction direction, removing the need for a syntactic grammar such as CCG (Zhao et al., 2014).

Later work made structure explicit in the latent alignment between words and semantics. Dependency-based Hybrid Trees interpret each semantic unit as a latent dependency label between words, perform exact dynamic-programming inference with complexity $O(N^3\!\cdot M)$ , and report on the multilingual GeoQuery dataset that English $F_1$ rises from 86.8 for DepHT to 89.3 for DepHT+NN (Jie et al., 2018). A graph-based reentrancy-free parser proves that both MAP inference and latent tag anchoring are NP-hard, then uses constraint smoothing and conditional gradient to approximately solve inference; on GeoQuery it reports 90.7 on IID, 86.2 on Template, 69.3 on Length, and 82.2% exact match (Petit et al., 2023).

GeoQuery has also supported work on certified abstention. The unanimity principle predicts only when all models consistent with the training data agree on the output, yielding 100 % precision under a well-specified model family (Khani et al., 2016). On GeoQuery, predicate-atom recall grows up to 70 % on the 600-example set, and 84 % of those yield a unique FunQL reconstruction, giving approximately 59 % full logical-form recall (Khani et al., 2016). This makes GeoQuery a benchmark not only for accuracy maximization, but also for the study of precision–coverage tradeoffs under abstention.

4. Data augmentation, recombination, and synthetic supervision

A major GeoQuery research line treats the benchmark as a compositional data-generation problem. Data recombination induces a synchronous context-free grammar from the training set, samples recombinant utterance–logical-form pairs, and trains an attentional seq2seq model with copying on both original and synthetic examples (Jia et al., 2016). On the standard GeoQuery test split, the reported exact-match accuracy improves from 75.1 for seq2seq + attention without recombination to 80.4 with abstract entities, 82.2 with abstract whole phrases, and 83.0 with AbsAll (Jia et al., 2016).

Compositional pre-training extends that idea to a two-stage recipe: unsupervised pre-training on an augmented corpus followed by task-specific fine-tuning on the original GeoQuery data (Ziai, 2019). The best reported configuration on GeoQuery uses “nesting, entity, concat” augmentation with pre-training and reaches 74.3 sequence accuracy and 87.8 token accuracy, compared with 73.6 and 88.2 without pre-training under the same augmentation (Ziai, 2019). The same study reports that the “co-occurrence only” strategy performs substantially worse, and explicitly attributes degradation to noise from semantically unrelated token replacements (Ziai, 2019).

A broader synthesis approach learns a database-specific PCFG over SQL and a BART-based SQL-to-utterance generator, then pre-trains a neural parser on the synthesized pairs before fine-tuning on the real data (Wang et al., 2021). On GeoQuery, the reported execution accuracy for RAT-SQL rises from 70.9% to 74.6% on question split and from 49.5% to 62.1% on query split (Wang et al., 2021). Another augmentation method, SUBS, exchanges subtrees with the same semantic function label. On the GeoQuery Query split, 519 original training examples yield approximately 29 039 augmented examples, and BART accuracy rises from 0.85 to 0.88 while LSTM accuracy rises from 0.58 to 0.79 (Yang et al., 2022).

Taken together, these results indicate that GeoQuery is unusually responsive to augmentation methods that preserve compositional validity. The strongest gains are reported when the synthetic distribution explicitly respects semantic structure rather than relying on purely surface-level perturbation.

5. Compositional generalization and LLMs

GeoQuery has become a standard probe for compositional generalization under pretrained generative models. SeqZero decomposes SQL generation into five sub-clauses—FROM, SELECT, WHERE, GROUP-BY, and ORDER-BY—generates each clause by sequential prompting, and ensembles a few-shot BART model with a constrained zero-shot model (Yang et al., 2022). On the GeoQuery compositional split, the reported exact-match accuracy is 74.7 for SeqZero $_{Large}$ , compared with 72.5 for BART $_{Large}$ and 71.4 for the ablation without the zero-shot component (Yang et al., 2022).

Prompt-only text-to-SQL with Codex shows that GeoQuery can be highly learnable in-context when schema and exemplars are supplied in the prompt. Using a “Create Table + Select 3” prompt format and in-domain examples, davinci-codex reaches 87.6% test-suite accuracy at 0-shot, 94.1% at 10-shot, 96.0% at 20-shot, and 97.1% at 40-shot, while the reported T5-3B baseline remains at 85.7% for the shot counts shown (Rajkumar et al., 2022). The paper attributes a large fraction of the few-shot gain to learning GeoQuery-specific SQL conventions such as aliasing and argmax style from the demonstrations (Rajkumar et al., 2022).

Grammar prompting pushes the same idea toward explicit formal-language control. In the 32-shot GeoQuery setting with Codex-davinci-002, standard prompting gives 81.5 execution accuracy, grammar prompting without constraint gives 87.5, grammar prompting with grammar-subset constraint gives 88.6, and grammar prompting with grammar+program constraint gives 88.9; an oracle grammar reaches 96.1 (Wang et al., 2023). On out-of-distribution compositional splits, the reported execution accuracies are 95.7 on Template, 86.6 on TMCD, 88.6 on Length, and 90.8 on NewFunc, all above the corresponding standard-prompt results (Wang et al., 2023).

The in-context-learning literature further analyzes the gap between in-distribution and out-of-distribution GeoQuery performance. One study reports a trend of decreasing relative generalization gap as model size increases across OPT, BLOOM, CodeGen, and Codex, with code-specialized families achieving the strongest GeoQuery out-of-distribution results (Hosseini et al., 2022). This suggests that GeoQuery remains useful even in the LLM era because it exposes whether high in-context accuracy reflects genuine recomposition or only template-level matching.

6. Extensions: interaction, multi-task transfer, and execution feedback

GeoQuery has also been adapted to interactive and transfer-oriented settings. Prefix-to-SQL defines a benchmark, PAGSAS, containing 124K user question prefixes across Advising, GeoQuery, Scholar, ATIS, and Spider, and the GeoQuery sub-task contains 3.1 K prefixes, 6.8 K gold SQL queries grouped under those prefixes, average full question length 8.3 tokens, average prefix length 6.6 tokens, and average 3.7 omitted tokens per prefix (Deng et al., 2021). On GeoQuery question split, T5 + curriculum learning reaches Recall@5 = 42.7%, MRR@5 = 40.0%, and Save@5 = 14.7%, compared with 37%, 34%, and 11% for T5; on SQL-template split, T5 + CL reaches 24.0%, 17.6%, and 6.4% (Deng et al., 2021). The same work states that the primary difficulty factor is the number of omitted tokens rather than absolute prefix or SQL length (Deng et al., 2021).

Multi-task learning treats GeoQuery as one dataset among several semantic-parsing tasks. A fully shared seq2seq model trained jointly on GeoQuery, NLMaps, TOP, Overnight, and AMR reports 79.8% exact-match accuracy on GeoQuery, compared with 77.6% for the single-task baseline, while reducing the total number of parameters by 68% relative to training separate models (Damonte et al., 2021). This result is notable because it indicates positive transfer despite heterogeneous meaning representations across datasets (Damonte et al., 2021).

Execution-guided decoding provides a complementary, inference-time perspective. On GeoQuery, post-hoc execution filtering raises a seq2seq model’s test execution accuracy from 72.5 to 75.0, while the template-based model sees only marginal change from 55.2 to 55.6 at the largest beam reported (Wang et al., 2018). The mechanism prunes candidates that fail to parse, cause runtime type errors, or return an empty answer set (Wang et al., 2018). In practical terms, GeoQuery thus supports both interaction-oriented formulations, where predictions are made before the full question is available, and execution-aware formulations, where semantic validity is enforced during or after decoding.

7. Nomenclature and later name reuse

In recent arXiv usage, “GeoQuery” is no longer unique to the U.S.-geography semantic-parsing benchmark. A 2026 paper uses the name for a zero-shot retrieval system for global Sentinel-2 imagery in crisis response, reporting 31.6% accuracy within 50 km on 76 disaster-location queries and 50% within 50 km on UK floods (Walsh et al., 6 May 2026). Another 2026 paper uses “GeoQuery” for a geometry-guided diffusion framework for sparse-view reconstruction, reporting on DL3DV 3-view a PSNR improvement from 15.20 to 15.98 and an SSIM improvement from 0.579 to 0.614 (Cao et al., 12 May 2026).

These works are unrelated to the classic semantic-parsing benchmark except by title. A plausible implication is that modern citation practice should disambiguate “GeoQuery” by arXiv id, task description, and output formalism rather than by name alone. Within semantic parsing, however, GeoQuery remains a compact benchmark on which structural bias, compositional augmentation, exact inference, in-context prompting, certified abstention, and incomplete-question modeling can all be studied under controlled domain conditions.