Zero-Shot & Retrieval-Augmented Schema Generation

• The paper presents embedding-based two-tower architectures with synthetic query generation that boost zero-shot recall metrics, such as achieving 44.33% Recall@1 on Natural Questions.
• The schema-guided paradigm decouples language understanding from policy structure using graph-based representations to improve action prediction in dialog systems.
• Retrieval-augmented generation integrates dense retrieval and generative models to enhance slot filling accuracy and support robust domain adaptation across varied applications.

Zero-shot and retrieval-augmented schema generation refers to the development of models and frameworks that can infer, generate, or adapt structural representations (schemas) in data-rich tasks—such as dialog systems, knowledge graph induction, slot filling, and document understanding—without direct supervision or manual labeling in the target domain, by leveraging external retrieval and synthetic data augmentation. Recent advances have integrated powerful neural architectures, generative modeling, and retrieval techniques, enabling robust handling of semantic variation, disjoint domains, and evolving information structures.

1. Embedding-Based Architectures and Synthetic Data for Zero-Shot Retrieval

A foundational approach employs embedding-based two-tower architectures, in which queries and candidate schema elements (such as passages, table columns, or policy steps) are independently encoded in vector space, typically using BERT-base encoders. The relevance between query $q$ and candidate $d$ is quantified by their inner product:

$$\mathrm{sim}(q, d) = f_\mathrm{Q}(q)^{\top} f_\mathrm{D}(d)$$

One key contribution is the generation of massive synthetic training data using a fine-tuned seq2seq model (BART) that produces diverse, naturalistic queries from passages, populating datasets such as WikiGQ. This synthetic augmentation leads to strong zero-shot performance; on Natural Questions, models trained on WikiGQ achieve Recall@1 of 44.33% compared to BM25's 30.67%. The architecture supports efficient inference by precalculating and indexing passage embeddings for similarity search (Liang et al., 2020).

In schema generation, the same paradigm enables the matching of textual descriptions to schema elements, or generating schema templates, without lexical overlap, by learning semantic similarity in embedding space. Synthetic query generation can be extended: for every schema-related passage, diverse queries mimicking real user utterances can be produced, aiding automatic schema induction. The method outperforms BM25 and even some human-labeled baselines, illustrating the critical role of synthetic data diversity.

2. Schema-Guided Paradigm for Zero-Shot Dialog and Action Prediction

Zero-shot dialog adaptation demands explicit decoupling of language understanding from policy structure. The schema-guided paradigm formalizes dialog policies as schema graphs, which encode both expected system actions and possible user utterances. During inference, models receive the explicit policy as a graph, guiding action prediction even in unseen domains.

The Schema Attention Model (SAM) employs word-level attention mechanisms between the dialog context and schema node representations (concatenating preceding and current steps). This attention is propagated to system actions; by dispensing with task-specific linear classifiers, the system generalizes to actions not seen during training. Improved schema graphs (with user nodes and richer context transitions) further bolster robustness. In zero-shot experiments on the STAR corpus, SAM achieves up to +22 F1 improvement over prior work (Mehri et al., 2021).

The schema-guided paradigm connects directly to schema generation: incorporating explicit, graph-based representations empowers models to generate and align schemas even when transferring to new tasks, and evidences the value of fine-grained structural modeling.

3. Retrieval-Augmented Generation for Slot Filling and Knowledge Graph Schema Induction

Retrieval-augmented generation (RAG) tightly couples retrieval and generation in an end-to-end trainable system. In KGI, dense passage retrievers—enhanced by dense negative sampling—locate supporting passages for slot-filling queries; a BART generator then produces slot values marginalizing over retrieved evidence:

$P(t_i) = \sum_j P(t_i | s_j) \cdot P(s_j)$

This yields marked improvements in slot-filler accuracy, recall, and provenance metrics on datasets like T-REx and zsRE. The same architecture adapts to new domains, as demonstrated on the TACRED variant using zero- or few-shot adaptation.

Applied to schema generation and knowledge graph induction, these techniques allow for automatic schema extraction and slot value prediction with supporting evidence, crucial for robust knowledge base construction (Glass et al., 2021).

4. Query Augmentation and Training-Free Schematization via External Knowledge Retrieval

Methods such as QZero reformulate the input query by retrieving supporting categories from Wikipedia. The augmented query—concatenating top Wikipedia categories—amplifies contextual knowledge for embedding-based classification models (both static and contextual). In news classification, QZero boosts even large OpenAI models by 5–13% and allows small word embeddings (Word2Vec) to match larger model performance, acting as a "knowledge amplifier" (Editor's term).

In schema generation, this retrieval of granular, structured categories from dynamic resources furnishes schema templates with up-to-date context, enhancing adaptability in rapidly evolving domains (Abdullahi et al., 21 Jun 2024).

5. Schema Augmentation and Robust Domain Adaptation

Schema Augmentation introduces variations (e.g., synonyms, encoded names) to the schema surface form during fine-tuning, forcing models to attend to slot descriptions and possible values rather than memorized identifiers. In dialogue state tracking, this leads to over twofold accuracy improvements in zero-shot adaptation to held-out domains, as measured by Target Goal Accuracy (TGA):

$\mathrm{TGA} = |\mathcal{H}_t| / |\mathcal{X}_t|$

This technique is extensible to other schema-generation contexts, especially where retrieval-augmented methods can supply dynamic variations at inference time. The model becomes robust to schema changes, enhancing generalization without sacrificing in-domain performance (Richardson et al., 31 Oct 2024).

6. Retrieval-Augmented Generation in Multimodal, Time Series, and Knowledge Graph Domains

Complex domains require multimodal or longitudinal schema comprehension. FilterRAG, for example, integrates multimodal encoders (BLIP-VQA) with external Wikipedia and DBpedia retrieval for Visual Question Answering, grounding answers to reduce hallucination (grounding score $\sim$70% in both in-distribution and OOD settings) (Sarwar, 25 Feb 2025).

In time series forecasting, TimeRAF and TS-RAG frameworks augment pre-trained TSFMs with retrieval from curated knowledge bases. Candidate histories are retrieved and adaptively mixed (via MLPs, multi-head attention, gating mechanisms) with query embeddings, producing improved zero-shot forecasts (up to 6.84% MSE reduction on ETTh1, Weather, and other benchmarks) (Zhang et al., 30 Dec 2024, Ning et al., 6 Mar 2025). The mixture-of-experts approach parallels retrieval-augmented schema fusion for complex generation tasks.

Walk&Retrieve traverses knowledge graphs via walks (Random Walks, BFS), verbalizes sequences into textual descriptions using LLM prompts, and anchors retrieval in a shared LLM space for zero-shot generation. It achieves lower hallucination, higher accuracy, and seamless adaptation to dynamic graph updates—establishing itself as a lightweight baseline for future KG-informed RAG systems (Böckling et al., 22 May 2025).

7. Evaluation, Multilinguality, and Efficient Schema Retrieval

The IRSC benchmark was developed for zero-shot evaluation of embedding models in RAG across multilingual corpora and varied schema-related retrieval tasks (query, title, part-of-paragraph, keyword, and summary retrieval). New metrics—Similarity of Semantic Comprehension Index (SSCI) and Retrieval Capability Contest Index (RCCI):

$$\mathrm{SSCI} = \frac{1}{Q} \sum_{q=1}^Q \frac{|m_{1_q} - m_{2_q}|}{n}$$

$$\mathrm{RCCI} = \frac{1}{Q} \sum_{q=1}^Q \frac{m_{1_q} - m_{2_q}}{n}$$

allow for comparison of semantic retrieval capability, guiding the selection of robust models for schema generation in mixed-language, diverse input formats (Lin et al., 24 Sep 2024).

DocsRay advances training-free doc-schema understanding by generating pseudo-TOCs via multimodal prompt-driven chunking, merging, and summarization. Hierarchical retrieval drastically improves query latency while maintaining accuracy, enabling real-world applications in document understanding where schema boundaries are latent and heterogeneous (Jeong et al., 31 Jul 2025).

Summary

Zero-shot and retrieval-augmented schema generation integrates neural encoders, synthetic query/data generation, schema augmentation, multimodal fusion, and external retrieval. These advances enhance system adaptability, reduce dependency on labeled data, and achieve quantifiable improvements in recall, precision, and generalizability—across text, dialogue, multimodal, time series, and graph-based domains. Ongoing research seeks more robust filtering, diversified synthetic sample creation, dynamic multi-agent schemas, and efficient retrieval in ever-evolving knowledge ecosystems, as evidenced across leading benchmarks and multifaceted real-world evaluations.