Natural Language Queries

• Natural language queries are defined as unconstrained linguistic inputs that enable users to interact with databases and APIs in everyday language.
• They operate via multi-stage pipelines—preprocessing, entity recognition, schema mapping, and query generation—to translate human language into structured queries.
• NLQ systems employ rule-based, deep learning, and hybrid methods, with applications spanning finance, visualization, code retrieval, and more.

A natural language query (NLQ) is an unconstrained linguistic utterance—spoken or written—that expresses information needs as a user would in human conversation, with the goal of retrieving or manipulating data within structured (e.g., relational, knowledge-graph, or API-driven) systems. NLQs are the central abstraction in natural language interfaces (NLIs) to databases, search engines, analytics platforms, scientific repositories, and domain-specific systems. NLQs bypass technical barriers such as structured query language (SQL) or other formalisms, aiming to democratize data access by letting users interact with data in their preferred language modality. The design, processing, and deployment of NLQ-driven systems span a spectrum of architectures and methodologies, incorporating rule-based parsing, deep learning, hybrid techniques, domain ontologies, and intelligent ranking mechanisms. This article surveys major NLQ system architectures, technical challenges and methodologies, practical instantiations across domains, key evaluation metrics, and ongoing research issues.

1. System Architectures and Processing Pipelines

Natural language query systems are most commonly realized as multi-stage pipelines, each stage addressing a well-defined subproblem in NLQ interpretation and execution (Amavi et al., 2024, Quamar et al., 2022). The canonical pipeline is:

• Preprocessing: Tokenization, lemmatization, part-of-speech (POS) tagging, and syntactic parsing (dependency or constituency), often using libraries such as spaCy or Stanford CoreNLP (Amavi et al., 2024, Montgomery et al., 2020).
• Entity Identification: Lexicon- or embedding-driven extraction of entities (surface values, named entities, attribute mentions) from the token stream, possibly augmented with sequence-tagging models (e.g., BiGRU+CRF) or transformer-based embeddings (Usta et al., 2022, Quamar et al., 2022).
• Type and Context Mapping: Mapping extracted entities to their database or schema types, using hand- or automatically-generated ontologies, lookup tables, or neural encoders (Amavi et al., 2024, Jamil, 2017).
• Query Intent and Semantic Linking: Identifying the intended query type, aggregating constraints, and assembling a coherent schema-level or knowledge-graph representation via rule-based or attention-based linking, cost-based graph algorithms, or learned neural scoring (Khabiri et al., 18 Oct 2025, Quamar et al., 2022).
• Query Generation: Translating the intermediate interpretation into a structured query language (SQL, SPARQL, Cypher, DSL, or declarative API calls), with handling of disjunction, conjunction, and operator attachment (Amavi et al., 2024, Khabiri et al., 18 Oct 2025, Limpanukorn et al., 2 Jul 2025).
• Query Execution and Postprocessing: Running the generated query, sometimes with runtime or pre-execution validation (syntactic, semantic), followed by data retrieval, ranking, and natural language generation (NLG) for presenting results (Fotso, 2024).

Some systems bifurcate this architecture into manual (domain-specific, rule-based) and end-to-end neural variants (Quamar et al., 2022), while others, such as xDBTagger, adopt a hybrid approach to negotiate efficiency, explainability, and precision (Usta et al., 2022). Domain knowledge integration via ontologies is prominent in scientific (BioSmart), engineering (BIM), and industrial contexts (Jamil, 2017, Yin et al., 2023).

2. Technical Foundations: Entity Enrichment, Logical Formulation, and Disambiguation

The entity enrichment paradigm (Amavi et al., 2024) anchors modern NLQ systems by treating query interpretation as enrichment and transformation of surface-level "simple entities" (tuples of surface values and lexical types) into "enriched entities" (tuples containing value, database type/class/property, lexical type, and attached operator). These serve as the formal substrate for generating logical database queries (under conjunctive query or Datalog semantics):

$E = (\mathcal{V}, \mathcal{T}, m)$

$$E_e[\text{EntityValue}, \text{DBType}, \text{LexType}, op]$$

Where $\mathcal{V}$ is the set of extracted surface values, $\mathcal{T}$ the set of candidate lexical types, and $m$ the mapping from values to types. Context enrichment merges contextually related entities, supporting complex queries by conjoining constraints from different parts of the user's utterance (Amavi et al., 2024).

Ambiguity resolution (lexical or semantic) is addressed via:

• Ontological Filtering: Ambiguous terms are mapped to domain concepts through ontologies; context clues are extracted from neighboring tokens or syntactic relations (Al-Harbi et al., 2017).
• WordNet and Domain Context: Manual or semi-automatic labeling of word senses with context labels (e.g., Money, Transport) enables filtering of candidate senses to those consistent with context and ontology (Al-Harbi et al., 2017).
• Rule- or Graph-Based Disambiguation: Minimum-cost Steiner tree or highest-degree node selection in schema graphs supports resolution of entity matches (Quamar et al., 2022, Montgomery et al., 2020).

3. Methodological Spectrum: Rule-Based, Deep Learning, and Hybrid Approaches

Three methodological paradigms dominate NLQ research (Quamar et al., 2022):

• Rule-Based Systems: Utilize explicit lexicons, ontologies, and grammatical templates. Index-based or ontology-driven mapping directs parsing and structured query generation, favoring high precision and interpretability at the expense of coverage and paraphrase robustness (Amavi et al., 2024, Usta et al., 2022, Yin et al., 2023).
• Deep Learning Text-to-SQL: Neural encoder–decoder models—typically utilizing BiLSTM, BERT, or other transformers—map input utterances (plus schema context) to output query tokens or ASTs, often with pointer networks for entity linking and execution-guided decoding to maximize validity and faithfulness (Quamar et al., 2022, Fotso, 2024).
• Hybrid Systems: Combine ML-based taggers for entity recognition/schema linking with rule-based interpreters or symbolic logic for assembly and validation (Usta et al., 2022, Amavi et al., 2024). Hybrid architectures address the trade-off between robustness and domain adaptation, and can include stages such as ML-driven tagging, graph-based join path inference, and rule- or heuristic-driven query clause extraction.

Declarative architectures have become prominent in multi-source/heterogeneous settings, where NLQ translation targets extended SQL or declarative languages supporting embedded API calls and function evaluation (Khabiri et al., 18 Oct 2025).

4. Domain-Specific Instantiations and Applications

NLQ methodologies are instantiated across a range of application domains:

• Relational and Graph Databases: Core NLQ engines process unconstrained queries over relational schemas, often with vector-driven schema matching and iterative validation (Fotso, 2024, Usta et al., 2022, Montgomery et al., 2020).
• Financial Knowledge Search: NLQ systems for finance integrate dense vector models, hybrid retrieval (BM25+ANN), domain-specific NER (e.g., for CUSIP, ISIN, regulations), dynamic freshness requirements, compliance/ranking constraints, and LLM-powered response generation (Pant et al., 24 Jan 2026).
• E-Commerce and Information Retrieval: Datasets of NLQs for e-commerce support sequence labeling for key facts, detection of vague/subjective phrasing, and attribute mapping for product search and recommenders (Papenmeier et al., 2023).
• Spatio-Temporal and Scientific Data: NLQ4ST maps spatial and temporal linguistic expressions to operator plans for trajectory and POI data, using layered knowledge bases, entity linking, operator selection, and optimizer-driven plan generation (Wang et al., 22 Jan 2026). BioSmart orchestrates multi-level knowledge-based inference via Datalog rule synthesis and execution, integrating domain inference and tool invocation (Jamil, 2017).
• Media and Code Retrieval: NLQ-driven retrieval extends to cross-modal domains (audio, video) with embedding-based similarity architectures, and to codebases via translation of NLQ to structural DSLs (e.g., Semgrep, GQL) using RAG-augmented LLMs and benchmark-driven evaluation (Oncescu et al., 2021, Limpanukorn et al., 2 Jul 2025).
• Visualization: NLQ-to-visualization synthesis uses type-directed program synthesis to generate well-typed, design-guideline compliant visualization programs from intent-and-slots parsed NLQ specifications (Chen et al., 2022, Zhang et al., 15 Jun 2025).
• BIM and Engineering Models: Modular ontologies enable detailed NLQ-based querying over IFC/OWL BIM models, parsing multi-constraint natural language into SPARQL with logical AND/OR, attribute, and relational constraints (Yin et al., 2023).

5. Evaluation Protocols, Benchmarks, and Empirical Findings

Standardized evaluation frameworks drive NLQ research:

• Accuracy Metrics: Reporting includes structured query generation accuracy (exact match), execution accuracy (does SQL/SPARQL result match ground truth), and component-level metrics (entity, WHERE, JOIN correctness) (Fotso, 2024, Usta et al., 2022).
• Benchmarks: Notable datasets include WikiSQL, Spider, SParC, CoSQL, domain-specific benchmarks for finance and code, and e-commerce NLQ corpora (Quamar et al., 2022, Limpanukorn et al., 2 Jul 2025, Papenmeier et al., 2023).
• Specialized Metrics: In code search, precision/recall/F1 at the line level; in media retrieval, Recall@K and mAP; in visualization, top-k accuracy for correct chart specification (Limpanukorn et al., 2 Jul 2025, Oncescu et al., 2021, Chen et al., 2022).

Experimental analyses consistently demonstrate that:

• Rule-based and hybrid systems exhibit high precision but may suffer in recall and coverage, particularly where lexicon or grammar overlap is incomplete (Amavi et al., 2024).
• Deep learning systems outperform rules for paraphrase robustness and generalization, provided substantial annotation exists; performance drops over complex, multi-table queries and out-of-vocabulary phenomena (Fotso, 2024, Quamar et al., 2022).
• In code and financial retrieval, NLQ-based translation to structured search (DSL, SQL) vastly outperforms keyword and vector-based baselines, with reported F1 gains of 14–57 points in code search (Limpanukorn et al., 2 Jul 2025) and improved qualitative relevance in hybrid search (Pant et al., 24 Jan 2026).
• Declarative NLQ-to-SQL+API architectures are more robust to data-source heterogeneity than agentic or imperative LLM approaches (Khabiri et al., 18 Oct 2025).

6. Challenges, Limitations, and Future Directions

Open issues in NLQ research and deployment span:

• Scalability and Domain Transfer: Manual lexicon/ontology extension and rule/heuristic coverage remain large bottlenecks for vertical or cross-domain scaling (Quamar et al., 2022, Amavi et al., 2024).
• Ambiguity and Vagueness Handling: Determining and clarifying vague expressions or resolving underspecified constraints (e.g., "good battery," "by region") are significant challenges. Deferment of disambiguation until downstream disjunction (lazy enumeration) is one practical mitigation (Amavi et al., 2024).
• Complex Query Classes: Most deployed systems handle conjunctive and facet-style queries; support for multi-class, nested, aggregate, and windowed queries is limited and is a core focus for future methodology (Amavi et al., 2024, Montgomery et al., 2020).
• Explainability and Auditability: Black-box deep-learning approaches provide limited insight into entity mapping and query decisions; explicit labeling, schema graph highlighting, and local explanation models (e.g., LIME) are emerging techniques (Usta et al., 2022, Deutch et al., 2020).
• Low-Data and Cross-Schema Generalization: Reducing labeled data requirements (through few-shot, self-supervised, or transfer learning) and developing schema-agnostic architectures are pressing needs (Quamar et al., 2022, Tang et al., 2024).

Anticipated directions include: multi-turn and conversational NLQs for analytics workflows, robust multi-modal and multi-lingual support, hybrid neuro-symbolic reasoning, dynamic adaptive ranking for changing corpora, and enhanced human-centered auditability and clarifications (Quamar et al., 2022, Fotso, 2024, Pant et al., 24 Jan 2026, Amavi et al., 2024).

7. Summary Table of Representative NLQ Systems and Domains

System/Domain	Key Approach	Notable Features
Entity Enrichment NLQ (Amavi et al., 2024)	Lexicon+grammar, rule-based	Entity enrichment, high precision, logical query mapping
xDBTagger (Usta et al., 2022)	Hybrid (BiGRU+CRF + graph)	Explainable tagging, join path inference, scalable
NLQ to SQL (General AI) (Fotso, 2024)	LLM-based encoder/decoder	Syntactic/semantic validation, iterative refinement
NLQ for Finance (Pant et al., 24 Jan 2026)	Embeddings, hybrid retrieval, NER	Data freshness, audit, LLM-backed explainability
NLQ for Visualization (Chen et al., 2022Zhang et al., 15 Jun 2025)	Type-directed synthesis, slot-filling	Chart-type constraints, denoising, axis matching
Code Search (Limpanukorn et al., 2 Jul 2025)	LLM→DSL translation via RAG	High F1, code pattern matching, DSL-agnostic
Ontology-aided BIM (Yin et al., 2023)	Modular OWL2 ontology, semantic parsing	SPARQL, multi-constraint, logic-based grouping

Detailed system descriptions, notation, and empirical results can be found in their respective references. This synthesis covers the technical landscape and outstanding challenges in NLQ research as exemplified in the recent literature.