Synthetic Query Creation

• Synthetic query creation is the automated generation of queries using algorithms, LLMs, and templates to simulate real-user inputs.
• Key methodologies include schema-aware synthesis, template-based generation, and interactive sketching to ensure semantic fidelity and diversity.
• Empirical studies reveal that high-quality synthetic queries enhance performance in IR, semantic parsing, privacy-preserving learning, and RAG systems.

Synthetic query creation is the algorithmic generation of queries—expressed in natural or formal language—that do not originate from end users but are instead constructed automatically for the purposes of training, evaluation, or adaptation of downstream systems. Synthetic queries are widely used in information retrieval (IR), semantic parsing (e.g., Text-to-SQL or Text-to-Graph Query tasks), code search, privacy-preserving learning, ontology engineering, and retrieval-augmented generation (RAG). The technical landscape encompasses both generative models (e.g., LLMs, LLMs) and rule- or synthesis-template-based procedural strategies. The design and deployment of high-quality synthetic queries require methodological rigor to ensure semantic fidelity, diversity, schema-awareness, and alignment with real-world information needs.

1. Motivations and Applications

Synthetic query creation is motivated by the need to overcome limitations in real data availability, annotator cost, privacy constraints, and schema adaptation. Key use cases include:

• Benchmark Construction and IR Evaluation: Synthetic queries enable the scalable assembly of test collections for IR evaluation when user logs or hand-crafted queries are unavailable or unrepresentative. LLM-based generation supports both query and synthetic relevance judgment creation, providing test collections that can achieve high system-ordering fidelity with real benchmarks (Rahmani et al., 2024).
• Training Data Augmentation: In semantic parsing tasks such as Text-to-SQL or Text-to-Cypher, synthetic queries allow the bootstrapping of model training in low-data domains, adaptation to new database schemas, and the balancing of unlabeled databases for column and operator coverage (Zhao et al., 2022, Caferoğlu et al., 30 Sep 2025, Tiwari et al., 2024, Zhong et al., 2024).
• Domain Adaptation and Diversity: For IR ranker fine-tuning and generalization, synthetic queries generated through clustering and sampling over document corpora provide representative and diverse training data, crucial for domain transfer scenarios (Chandradevan et al., 2024).
• Privacy-Preserving Retrieval Systems: Differentially private LLMs (DP-LMs) can be trained to generate synthetic queries from documents without exposing sensitive input data, enabling privacy-guaranteed training of dual-encoder retrieval models (Carranza et al., 2023).
• Code Search and Ontology Engineering: Representative synthetic queries for code pattern search and synthesis of ontology competency questions (CQs) facilitate interpretable, systematic evaluation in software engineering and knowledge graph querying (Wang et al., 2023, Wiśniewski et al., 2021).
• RAG Systems and Query Rewriting: Synthetic query rewrites, especially when supervised by positive documents or answers, have been shown to bridge the intent gap more successfully than human rewrites in multi-turn RAG applications (Zheng et al., 26 Sep 2025).

2. Core Methodologies for Synthetic Query Construction

Synthetic queries can be generated via a variety of algorithmic and machine learning approaches, which include:

• Generative LLMs: Sequence-to-sequence transformers (e.g., T5, GPT-4, Llama-family, Flan-T5) are commonly used in zero-shot, few-shot, or fine-tuned modes. Prompts incorporate document passages, schema context, or sample queries, with decoding strategies (temperature, top-p, beam search) controlling output diversity and specificity (Rahmani et al., 2024, Breuer, 2024, Sannigrahi et al., 2024, Chandradevan et al., 2024).
• Schema- or Type-Aware Synthesis: Effective synthetic data generation for structured query tasks relies on explicit schema-graph modeling, strong typing, and primary/foreign-key constraints to sample only semantically valid column combinations and joins. For SQL, this includes schema-distance-weighted column sampling and key-tagmatching to prevent illogical queries (Zhao et al., 2022). For Cypher or SPARQL, schema constraints and slot-filling are essential (Tiwari et al., 2024, Zhong et al., 2024, Wiśniewski et al., 2021).
• Template- and Grammar-Based Synthesis: Template filling, often based on parameterizable domain-specific languages (DSLs) or canonical query patterns, ensures full coverage of query constructs (aggregation, filtering, joins, etc.), while enabling scalable instantiation across databases, ontologies, and domains (Zhong et al., 2024, Wiśniewski et al., 2021).
• Interactive Sketching and Program Synthesis: In programming-by-example (PBE) or interactive query design, users provide incomplete query "sketches" with optional soft constraints; automated refinement and symbolic deduction (e.g., with worklists, abstract interpretation) fill in the holes, guided by feedback or by input/output examples (Bastani et al., 2019, Yaghmazadeh et al., 2017, Liu et al., 2024).
• Diversity/Cluster Allocation: To promote representativeness, clustering-based document selection and probabilistic/MRR sampling (e.g., in DUQGen) ensure that training queries are drawn from all semantic regions of a collection (Chandradevan et al., 2024).
• Rewriting and Pairwise Generation: For nuanced relevance prediction or query clarification in multi-turn contexts, pairwise or relative generation (first generate a positive/relevant query, then an explicitly conditioned negative/contrastive one) improves the utility of synthetic pairs for downstream discriminative tasks (Chaudhary et al., 2023, Zheng et al., 26 Sep 2025).

3. Challenges in Quality Assurance and Validation

A critical concern in synthetic query generation is the risk of introducing ill-formed, illogical, or unfaithful queries. Major technical remedies include:

• Schema and Key-Aware Sampling: Enforcing that component selection (columns, tables) respects both type constraints and foreign-key structure. Arbitrary sampling leads to nonsensical SQL and ineffective augmentation (Zhao et al., 2022).
• Intermediate Representation (IR) for NLQ Generation: To mediate the gap between logic forms (e.g., SQL) and natural language queries, the design of an intermediate representation streamlines NLQ synthesis via rule-based linearization and rewrite steps, improving BLEU/ROUGE alignment with user intent (Zhao et al., 2022).
• Human-in-the-Loop and LLM-as-Judge Validation: Synthetic queries are filtered through both expert manual review (for clarity, specificity, diversity), and automated LLM-based judges rating faithfulness, logicality, and executability. For Text-to-SQL/Graph tasks, this often involves running queries against databases and discarding or repairing failures (Tiwari et al., 2024, Caferoğlu et al., 30 Sep 2025, Zhong et al., 2024).
• Automatic Repair and Deduction: For sketches or template-driven pipelines, SMT-backed deduction, automatic fault localization, and repair tactics (e.g., join/disjunction insertion, predicate splitting) ensure candidates are schema-consistent and non-trivial, boosting coverage and accuracy (Yaghmazadeh et al., 2017, Liu et al., 2024).
• Statistical Agreement for IR Test Sets: Kendall's τ, Cohen's κ, and other metrics are used to assess system ordering and agreement between system rankings under real and synthetic queries/judgments (Rahmani et al., 2024).

4. Empirical Findings and Performance Impact

Synthesizing high-quality queries is empirically validated to yield substantial gains in multiple downstream tasks:

• Semantic Parsing for Databases: High-quality synthetic data (with schema-aware constraints and IR) enables T5-3B (with PICARD or similar decoding) to outperform prior state-of-the-art by up to +6.9 EM on Spider (Zhao et al., 2022). In domain-specialized settings, frameworks such as SING-SQL and SynthCypher report absolute gains of 12–40 pts over strong baselines on execution accuracy and F1, across in-domain and cross-domain splits (Caferoğlu et al., 30 Sep 2025, Tiwari et al., 2024).
• Information Retrieval and Test Collections: Synthetic LLM-generated queries allow system evaluation with τ > 0.8 against real query baselines, and have been shown to be unbiased with respect to different retrieval architectures (Rahmani et al., 2024). Fusion of multiple LLM-generated query variants (using RRF or similar) increases nDCG@10 by 27–49% over BM25 and PRF baselines across standard TREC benchmarks (Breuer, 2024).
• Privacy-Preserving Retrieval: DP synthetic queries produced via DP-LM fine-tuning recover 70% of non-private model utility at strong privacy settings (ε=3), outperforming direct DP training by an order of magnitude (Carranza et al., 2023).
• Label-Conditioned and Pairwise Setting: For nuanced relevance or zero-shot settings, label-conditioned and pairwise QGen mitigate—but do not fully resolve—the challenge of label leakage and distinction, with relative/pairwise methods providing higher discrimination and ranking performance in hard-negative constructions (Chaudhary et al., 2023, Chaudhary et al., 2023).
• RAG Query Rewriting: Synthetic rewrites can yield +7.5 ROUGE-1, +23.6 MRR@5 improvements over manual rewrites in conversational QA datasets, supporting superior intent capture and retrieval/generation (Zheng et al., 26 Sep 2025).

5. Methodological Best Practices and Recommendations

The literature develops a set of methodological guidelines for synthetic query construction:

• Incorporate full schema, type, and key information in template instantiation and sample selection to avoid pathological training signals.
• Use principled cluster-based document selection and Maximal Marginal Relevance to ensure query diversity and centrality in domain adaptation contexts (Chandradevan et al., 2024).
• Employ LLMs in both generation and validation, with prompt engineering to inject context and examples (zero-shot, few-shot, chain-of-thought), and downstream LLM-based quality control for logical and semantic soundness (Rahmani et al., 2024, Tiwari et al., 2024, Zhong et al., 2024).
• Fuse top-N LLM-generated query variants for robust retrieval (usually N≈10–20 suffices), and prefer rich topic context in prompts (Breuer, 2024).
• Maintain rigorous human-in-the-loop or LLM-as-judge validation, prioritizing both functional correctness and domain specificity; drop or repair ill-formed candidates (Caferoğlu et al., 30 Sep 2025, Tiwari et al., 2024).
• In relevance prediction and IR, label-conditioning alone is insufficient for nuanced label spaces: consider pairwise/relative task setups and auxiliary losses to increase discriminative power (Chaudhary et al., 2023, Chaudhary et al., 2023).

6. Generalization, Limitations, and Extensions

Synthetically generated queries generalize to a wide spectrum of query types and application domains:

• Schema-agnostic pipelines can be instantiated for new database schemas, graph query languages or ontologies, provided type metadata and key relationships can be extracted and injected into prompt or template logic (Zhao et al., 2022, Tiwari et al., 2024, Wiśniewski et al., 2021).
• Hybrid approaches (LLM-generation plus template coverage) are recommended to combine semantic diversity with coverage of complex constructs (Zhong et al., 2024).
• Limitations include risk of domain or label drift (especially with over-complex template-filling in mismatched evaluation pools (Zhong et al., 2024)), computational cost in DP or SMT-based synthesis, and residual challenges in capturing highly nuanced or multifaceted user-intent in complex RAG or multi-label settings (Chaudhary et al., 2023, Zheng et al., 26 Sep 2025).
• Future directions focus on more sophisticated label and schema conditioning, learning-to-rank fusion of synthetic demonstration candidates, robust multi-modal extensions, and declarative validator pipelines (e.g., semantic coherence, schema adherence, answer-consistency).

In sum, synthetic query creation is a foundational technique for scalable, high-fidelity training and evaluation in modern IR, semantic parsing, code search, knowledge graph integration, and RAG systems, with ongoing advances in schema-aware synthesis, robust validation, and domain transfer. Rigorous quality control and schema/type awareness are paramount for ensuring the utility of synthetic data (Zhao et al., 2022, Rahmani et al., 2024, Breuer, 2024, Caferoğlu et al., 30 Sep 2025, Tiwari et al., 2024, Liu et al., 2024).