--- title: Structured & Unstructured Query Language (SUQL) url: https://www.emergentmind.com/topics/structured-and-unstructured-query-language-suql type: topic --- # Structured & Unstructured Query Language (SUQL) A Structured and Unstructured Query Language (SUQL) enables unified, declarative access to both structured databases (e.g., relational tables, property graphs) and unstructured data (e.g., free text, images, video, vector embeddings). SUQL systems integrate the logic and algebraic rigor of classical query languages with the flexibility, retrieval, and reasoning capabilities of large language models (LLMs) operating over unstructured assets. Recent work establishes formal grammars, algebraic semantics, query-planning toolchains, and cost models that endow SUQL with compositionality, extensibility, and strong empirical performance in conversational assistants, scientific query systems, and hybrid retrieval settings [2311.09818, 2312.13264, 2509.00277, 2510.18029, 2601.09036]. ## 1. Formal Syntax, Semantics, and Algebraic Extensions SUQL design extends foundational query grammars (SQL, Cypher, Relational Algebra) to admit unstructured data access and reasoning operators: - **Extended Grammar**: SUQL queries augment SQL or Cypher with special operators or predicates, e.g. ANSWER/ SUMMARY functions invoking LLMs over a text field [2311.09818], SEM_WHERE/ SEM_SELECT/ SEM_JOIN for semantic processing [2509.00277], or sub-property extraction in property graphs (CypherPlus) [2107.01963]. - **Algebraic Semantics**: The SABER algebra [2509.00277] generalizes selection (σ), projection (π), join (⋈), grouping (γ), intersection, and difference to semantic variants (σ^{sem}, π^{sem}, ⋈^{sem}), where selection is performed via embedding similarity or LLM-based inference over text or multimedia fields. - **BNF/Pseudocode Fragments**: Canonical SUQL grammars include both structured operators (SELECT, WHERE, JOIN) and explicit unstructured/semantic operators, e.g., ```ebnf ::= SELECT FROM [ WHERE ] [ SEM_WHERE() ] [ SEMANTIC_FILTER ] ``` The semantic operators invoke LLMs or embedding-based retrieval for row-wise or join predicates, and the results are combined with structured query execution. ## 2. System Architectures and Execution Pipelines Core system architectures reflect a planner–compiler–executor pipeline augmented for hybrid data access [2509.00277, 2312.13264, 2510.18029]: - **Parser and Planner**: The query is parsed and decomposed into structured and unstructured fragments that are planned via combined schema and NL predicate analysis. In DynaQuery [2510.18029] and dIR [2312.13264], a Schema Introspection and Linking Engine (SILE) and a text-→columns LLM pipeline, respectively, organize schema discovery and mapping. - **Execution Engine**: Structured sub-queries are handled by native DBMS or graph processing operators; unstructured fragments are executed via LLM calls, embedding-based ANN search, or semantic vector retrieval (e.g., HNSW, IVF_SQL8) [2207.07940, 2501.05006, 2107.01963]. - **Combiner/Join**: Results are combined via user-specified or default fusion operators, e.g. outer joins, intersect, aggregate, or answer synthesis via an LLM [2304.13010, 2601.09036]. A common pattern is "plan–execute–synthesize": LLMs parse and plan, then DBMS and vector search engines retrieve, with final answer synthesis and grounding via LLMs [2601.09036]. ## 3. SUQL Operators, Semantic Primitives, and Illustrative Queries Leading SUQL systems exhibit the following operator patterns: - **Free-Text Primitives**: Functions such as ANSWER(text_col, question) and SUMMARY(text_col) invoke LLMs to extract information from a row’s text field, which can then be filtered, sorted, or projected as in SQL [2311.09818]. - **Embedding-Based Semantic Filters**: SEM_WHERE('text about X') evaluates to rows where the encoded text field is similar (above threshold θ) to the prompt embedding [2509.00277]. - **Multi-Modal Predicates**: Extensions for cross-modal joins and retrieval (e.g. associating a user's face in an image with their structured records) via SIMILARITY or sub-property extraction [2107.01963]. - **Hybrid Query Pattern Examples**: - Return orders where the product description mentions "organic apples": ```sql SELECT o.order_id, o.customer_id FROM customer_orders AS o JOIN product_descriptions AS p ON o.product_id = p.product_id WHERE SEM_WHERE('text is about organic apples') ``` - Hybrid conversational queries (natural-language to full SUQL program), as in Yelp restaurant search with constraints on cuisine, reviews, and location [2311.09818]. These operators are integrated into classical logical plans with selection pushdown, join-ordering, and cost-based optimization now extended with semantic cost terms [2509.00277, 2501.05006]. ## 4. Optimization, Indexing, and Cost Models for Hybrid Queries Efficient SUQL execution relies on multi-modal index design, semantic-aware cost models, and logical optimization: - **Indexing Strategies**: Semantic vector indexes (e.g., HNSW, IVF), inverted text indexes, and BLOB metadata indexes support efficient retrieval over unstructured modalities [2207.07940, 2107.01963]. - **Cost Models**: Hybrid cost formulas incorporate: - Embedding extraction costs - Vector similarity computation costs (e.g. O(ef) for ef search width in HNSW [2207.07940]) - LLM invocation and token consumption (critical under per-token pricing and context window constraints) [2304.13010] - Operator selectivity over unstructured subspaces and predicted speeds from semantic cache [2107.01963] - **Plan Rewriting and Pushdown**: Structured predicates and topology operators are pushed ahead of semantic filters where selectivity estimates justify, minimizing expensive LLM or vector operations [2509.00277, 2107.01963, 2501.05006]. CHASE leverages semantic-analysis passes, logical plan rewriting, vectorized code generation, and inlined runtime to achieve $13\times$-$7500\times$ speedups over non-native hybrid query engines [2501.05006]. ## 5. Empirical Evaluation, Metrics, and Application Domains SUQL approaches have been evaluated across conversational QA, scientific data access, and industry-scale workloads: - **Quantitative Performance**: - dIR achieves $0.85$ recall and $0.80$ precision on direct hybrid queries, outperforming dense IR and SQL-only baselines [2312.13264]. - SUQL (few-shot GPT-4) attains $59.0\%$ EM and $68.4\%$ F1 on HybridQA, within $7.1$ F1 of the SOTA (trained on $62$K data), and $90.3\%$ entity return accuracy on Yelp conversations [2311.09818]. - DynaQuery’s SILE pipeline suppresses SCHEMA_HALLUCINATION failures from $50.7\%$ (RAG) to $6.8\%$, and achieves $80.0\%$ execution accuracy on Spider, $58.6\%$ on BIRD [2510.18029]. - SABER’s semantic algebra supports SQL-compatible queries with semantic operator costs on the order of $|r|\cdot(c_{emb}+c_{sim})$ per selection, invoking LLMs or embedding services [2509.00277]. - Systems such as HQANN accelerate hybrid top-$k$ retrieval by leveraging fused distance metrics and attribute-aware graph navigation, e.g. $99\%$ recall@$10$ in $50$μs [2207.07940]. PandaDB sustains $>5300$ QPS, and mixed queries complete in tens–hundreds of ms [2107.01963]. - SpectraQuery attains $80\%$ SQL correctness, $93$–$97\%$ groundedness in answer synthesis, and expert satisfaction scores $4.1$–$4.6$/$5$ [2601.09036]. - **Application Areas**: - Open-domain and domain-specific QA, exploratory scientific search (e.g., battery science), recommendation engines, graph-based entity resolution, fraud detection, multi-hop dialogue [2601.09036, 2312.13264, 2107.01963]. ## 6. Limitations, Open Problems, and Future Directions SUQL systems face several technical and operational frontiers: - **Scalability**: LLM call costs, context window limits, and schema explosion (thousands of columns from discretization) constrain practical deployments [2312.13264]. - **Explainability and Provenance**: While structured traceability is well-understood, reliable provenance over LLM-driven retrieval and answer chains remains an open research challenge [2304.13010]. - **Semantic Ambiguity**: Zero-shot classification of large enumerations and value-to-column grounding remain brittle [2311.09818, 2510.18029]. - **Optimization**: Budget-aware plans—balancing DBMS and LLM costs, retrieval/filter cascades, lazy materialization—remain active research topics [2304.13010]. - **Generality**: Most SUQL systems require fine-grained schema awareness, and evaluation outside curated scientific or QA datasets is limited [2601.09036]. - **Planned Extensions**: Advanced multimodal support (images, XRD, audio), interactive disambiguation, ontology-driven key consolidation, reinforcement learning for interaction policies, debug/verification modes, dynamic operator insertion, and streaming data support are all articulated as practical next steps [2312.13264, 2601.09036, 2510.18029]. ## 7. Selected Comparative Table of SUQL System Properties The following table summarizes representative SUQL systems and their principal contributions: | System | Key SUQL Innovations | Empirical Highlights / Domain | |------------------|-------------------------------------|---------------------------------------| | dIR [2312.13264] | Text→columns discretization + few-shot text-to-SQL + ReAct conversational planner | 0.85 recall/0.80 precision; complex multi-hop QA | | SUQL [2311.09818] | SQL extension with ANSWER/SUMMARY LLM operators | 90.3% accuracy on real-world conversations | | SABER [2509.00277] | Extended Relational Algebra with semantic selection/join/group-by | Costed semantic plans; SQL-compatibility | | DynaQuery [2510.18029] | SILE schema-linking, per-row multimodal LLM filtering | 80.0% execution accuracy; robust to schema hallucination | | HQANN [2207.07940] | Attribute-filtered HNSW for fused queries | 99% recall@10 in <100μs (hybrid ANN) | | PandaDB [2107.01963] | CypherPlus: subproperty operators for property graphs | 100–1000x faster than pipeline baselines | | CHASE [2501.05006] | Native relational algebra extensions + compilation to MLIR | 13×–7500× speedup vs. plugin ANN SQL | | SpectraQuery [2601.09036] | Domain-adapted SUQL for scientific structured+literature search | 93–97% groundedness; >4/5 expert rating | Comprehensive technical depth, cost-aware operatorization, and tight integration of DBMS and LLM-based unstructured reasoning are hallmarks of modern SUQL research. The evolving landscape is marked by increasing formal rigor, operator modularity, and empirical validation, with significant progress on scalability and composability for hybrid data access in practical, high-value domains. Source: https://www.emergentmind.com/topics/structured-and-unstructured-query-language-suql