Unknown Schema: Structural Uncertainty

• Unknown schema scenarios are conditions where the data structure is incomplete, implicit, or evolving, requiring dynamic methods for schema discovery.
• Approaches include externalizing schema as machine-readable objects through techniques like JSON graphs, BERT embeddings, and ontology mappings.
• Techniques for retrieval, exploration, and evolution emphasize verifying metadata via active feedback loops, cross-validation, and structured checkpoints.

Searching arXiv for the cited works to ground the article in current metadata. An unknown schema scenario is a setting in which a model, tool, or modelling process cannot assume that the operative schema is known, fixed, complete, or preloaded. In the cited literature, this condition appears in task-oriented dialogue with unseen task schemas, enterprise text-to-SQL without full schema injection, analysis of unfamiliar relational databases, schema-on-read NoSQL systems whose structure is implicit in code or data, ontology-mediated access to XML for users who do not know the XML Schema, and live or local-first systems where schema expectations are partly encoded in running code and must evolve together with data (Mosig et al., 2020, Jian et al., 17 Mar 2026, Christopher et al., 2021, Bikakis et al., 2013, Edwards et al., 2023). The common technical problem is not simply missing metadata; it is the need to discover, represent, retrieve, verify, or evolve structure under partial observability.

1. Scope and meanings of “schema” across domains

The term schema has domain-specific meanings, but the underlying role is consistently structural. In STAR, a schema is a graph of the ideal dialog flow for a task, with nodes corresponding to system or user action descriptions or response templates and edges encoding the next step in the interaction (Mosig et al., 2020). In human event-schema curation, a schema specifies a recurring real-world scenario through events, participants, participant types, and relations, and may include linear temporal order, arbitrary temporal order, or mutually exclusive branches (Weber et al., 2021). In SchemaDB, the relevant object is a relational database schema represented in SQL and graph form, with tables, columns, and foreign keys as typed components of a directed heterograph (Christopher et al., 2021).

Other works broaden the notion further. The NoSQL study defines the schema as the structure implied by entity-class declarations in the Java domain model, making schema an artifact recovered from annotated code rather than from DDL (Scherzinger et al., 2020). SkiQL treats schemaless NoSQL systems as having implicit schemas that remain queryable once extracted into the U-Schema metamodel, where entity types, relationship types, and structural variations are explicit (Candel et al., 2022). The live/local-first challenge paper adopts the broadest definition: schema change includes any change to the expected shape of data, whether expressed in a database schema, a type system, or only in the behavior of code (Edwards et al., 2023). A related conceptual-modelling critique distinguishes the conceptual schema from the information base and argues that treating the schema as the whole conceptual model is a historically contingent “schema turn,” not a necessity (Partridge et al., 1 Sep 2025).

This diversity matters because unknown-schema reasoning is not limited to one representation family. A schema may be explicit and formal, implicit and reconstructable, partially documented, or still emerging from interaction with the data or environment.

2. Representing unfamiliar structure explicitly

A recurrent response to unknown-schema conditions is to externalize structure into machine-readable objects rather than leaving it latent in model parameters or application code. STAR operationalizes this most directly. Each task is paired with a schema flow chart, a KB interface, and suggested response templates in a Wizard-of-Oz collection setup, and each schema is stored as JSON with "replies" and "graph" so that inference can condition on an explicit task description (Mosig et al., 2020). The schema node texts are encoded with BERT, the dialog history is encoded with BERT, the history representation is $\mathbf{h}_{CLS}$, the schema node embeddings are

$K = \{\kappa_1, \kappa_2, \dots, \kappa_n\},$

and the corresponding next-action labels are

$V = \{v_1, v_2, \dots, v_n\}.$

SchemaDB provides a parallel move for relational structure. Its graph representation uses typed nodes for tables, columns, and foreign keys; table nodes store table name and primary key(s), column nodes store name, datatype, optional length, and position, and foreign key nodes store the constraint name if explicit (Christopher et al., 2021). For coarser reasoning, it defines a “skeleton” graph whose nodes are tables and whose edges denote foreign-key dependencies. The skeleton is then topologically sorted to canonize schemas into a valid MySQL creation order.

Event-schema curation uses a richer knowledge representation. The released schema library contains 232 schemas, of which 82 were created manually from textual complex-event descriptions and 150 by fleshing out automatically induced skeleton schemas (Weber et al., 2021). Each schema encodes a scenario name and description, an event structure, participant roles, co-referring participants across events, coarse and fine-grained entity types, and relations between participants. The schema backbone is a temporally ordered event chain, but the representation also permits unordered subsequences and mutually exclusive branches.

Unknown-schema work on semi-structured and multi-model data similarly depends on explicit intermediate representations. SPARQL2XQuery transforms XML Schema into an OWL Schema ontology $O_s$ and a Backwards Compatibility ontology $O_{BC}$, then defines mappings from ontology constructs to XPath sets so that users query an ontology rather than raw XML structure (Bikakis et al., 2013). SkiQL presupposes a U-Schema model that preserves type names, variations, shared and non-shared properties, aggregation relationships, and references, precisely because a union-of-fields view is inadequate for heterogeneous schemaless stores (Candel et al., 2022).

Across these systems, the decisive move is the same: structure is exposed as a manipulable object. This suggests that unknown-schema robustness improves when schema knowledge is represented as input data rather than treated as hidden training regularity.

3. Retrieval, exploration, and interaction under partial observability

In text-to-SQL, unknown-schema work has shifted from prompt construction to active retrieval and verification. Schema-First Retrieval treats catalog selection as a first-class retrieval problem and indexes five typed catalog object types—tables, columns, metrics, relationships, and query history—using object-specific text templates (Agrawal et al., 23 Jun 2026). At query time it combines parallel vector search, lineage expansion, cross-encoder reranking, workload memory, and deterministic access-control gates before SQL generation. Its scoring fusion is written as

$$s(q, o) = \alpha s_{\text{embed}(q, o)} + \beta s_{\text{rerank}(q, o)} + \gamma s_{\text{history}(q, o)},$$

with history evidence defined by

$$s_{\text{history}(q, o)} = \max_{h \in \mathcal{H}(o)} \cos(q, h)\exp(-\lambda \Delta t_h).$$

On CRUSH4SQL, the full system reaches 96.4% table recall@20, and cross-encoder reranking adds +11.1 points at column recall@10; on SEDE, query history raises table recall@5 from 52.1% to 92.3%; on BIRD, schema-first context reduces SQL execution errors from 15.6% to 6.2%, a 2.5x reduction (Agrawal et al., 23 Jun 2026).

TRUST-SQL approaches the same condition as sequential decision-making under hidden state. It formulates unknown-schema text-to-SQL as a Partially Observable Markov Decision Process

$$(\mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \Omega, \mathcal{Z}, \gamma),$$

where the hidden state $s_t$ is the true database schema, observations $o_t$ come from tool or execution feedback, and the agent maintains a verified knowledge state

$K = \{\kappa_1, \kappa_2, \dots, \kappa_n\},$0

containing only schema elements confirmed through tools (Jian et al., 17 Mar 2026). The protocol restricts actions to Explore, Propose, Generate, and Confirm, exposed as explore_schema, propose_schema, generate_sql, and confirm_answer. The Propose phase is a structural checkpoint at which the agent must serialize the verified schema and is forbidden from hallucinating unverified tables or columns. Experimentally, hallucination falls from 26.4% of failures in EC to 2.8% in EPGC, a 9.4× reduction. With Dual-Track GRPO, the 4B model reaches 64.9/67.2 on BIRD-Dev and the 8B model reaches 65.8/67.7, despite operating without pre-loaded metadata (Jian et al., 17 Mar 2026).

Unknown-schema access in XML systems is mediated differently. SPARQL2XQuery assumes that users query only against an ontology and are not expected to know the underlying XML Schemas or even the existence of XML data (Bikakis et al., 2013). Automatic ontology generation from XML Schema and automatic mapping generation make the XML structure executable without exposing it. SkiQL addresses a related exploration problem after schema extraction: schema-query returns the type-variations sub-schema of a selected entity or relationship type, and relationship-query returns subgraphs defined by aggregation or reference relationships, including indirect paths via >> (Candel et al., 2022).

These systems differ in interface—retrieval layer, tool-using agent, ontology mapping, or schema query language—but they converge on the same architectural principle: under unknown schemas, systems first narrow, verify, or re-express structure, and only then perform downstream reasoning or generation.

4. Learning and transfer across schema variation

Unknown-schema settings stress not only retrieval but generalization. STAR was designed to tackle the “unknown schema” problem in task-oriented dialogue by making task logic explicit rather than forcing a model to memorize it implicitly from training dialogs (Mosig et al., 2020). It contains 127,833 utterances and KB queries across 5,820 task-oriented dialogs in 13 domains and 24 tasks, with 4,152 single-task dialogs and 1,668 multi-task dialogs. The three-party interaction

$K = \{\kappa_1, \kappa_2, \dots, \kappa_n\},$1

requires models to decide when to query, what to query, and how to verbalize KB results. Its schema-guided next-action model computes

$K = \{\kappa_1, \kappa_2, \dots, \kappa_n\},$2

and blends schema-guided and schema-free predictions through

$K = \{\kappa_1, \kappa_2, \dots, \kappa_n\},$3

In zero-shot next-action prediction, schema guidance improves or matches schema-free baselines on held-out tasks and domains: for task transfer, BERT gives 36.45 / 36.89 and BERT + schema gives 36.77 / 37.15; for domain transfer, BERT gives 34.84 / 35.63 and BERT + schema gives 37.20 / 35.71. In zero-shot response generation, GPT-2 + schema improves entity F-1 from 48.22 to 52.12 in task-transfer Happy and from 50.43 to 53.02 in domain-transfer Happy + Unhappy (Mosig et al., 2020).

Human event-schema curation addresses transfer by expanding the schema library itself. Skeleton schemas are induced from text using Association Rule Mining and a causal compatibility scorer,

$K = \{\kappa_1, \kappa_2, \dots, \kappa_n\},$4

then human annotators add participants, types, coreference, and relations in SchemaBlocks (Weber et al., 2021). Corpus coverage is evaluated as

$K = \{\kappa_1, \kappa_2, \dots, \kappa_n\},$5

with

$K = \{\kappa_1, \kappa_2, \dots, \kappa_n\},$6

Adding the 150 curated skeleton-based schemas improves coverage by about 20%, indicating that extending the schema inventory increases the diversity of represented scenarios (Weber et al., 2021).

Relational learning studies the same problem at the level of inductive bias. Schema Independent Relational Learning defines schema independence under a bijective transformation $K = \{\kappa_1, \kappa_2, \dots, \kappa_n\},$7 by requiring both hypothesis invariance and semantic equivalence of learned definitions:

$K = \{\kappa_1, \kappa_2, \dots, \kappa_n\},$8

The paper proves that FOIL is not hypothesis invariant, that bottom-clause construction is schema dependent, and that ProGolem is not schema independent, while Castor achieves schema independence under decomposition and composition by leveraging inclusion dependencies (Picado et al., 2015). Empirically, on UW-CSE Castor remains stable across schemas with precision 0.93 and recall 0.54, whereas FOIL-, Progol-, and ProGolem-style systems vary substantially across Original, 4NF, and denormalized schemas (Picado et al., 2015).

A common lesson emerges: if the same information can be expressed under multiple schemas, then learning performance depends on whether the algorithm internalizes only correlations tied to one representation or exploits explicit structural correspondences.

5. Discovery, reconstruction, and inference when schemas are implicit

Some unknown-schema scenarios begin not with an unseen explicit schema but with the absence of any declared schema at all. SchemaDB was created because real relational schemata are rarely shared publicly for privacy and security reasons, leaving the field with benchmark databases, textbook examples, and engine benchmark designs as poor proxies for “database structures in the wild” (Christopher et al., 2021). Its 2,500 samples, standardized to MySQL syntax and accompanied by graph forms, document high variability in number of tables, number of columns, foreign-key presence, and likely normalization level. The authors note that a “not insignificant” number of schemas with more than two tables contain no foreign keys at all, and exact or near-exact naming matches can recover implicit foreign keys in about 20% of schemas lacking recorded foreign keys, whereas beyond edit distance 3 false positives become too high (Christopher et al., 2021).

The NoSQL study reconstructs schema from application code rather than from the datastore. It mines GitHub repositories using imports for Objectify or Morphia, clones over 1.2K candidate repositories, and selects the top 5 Objectify projects and top 5 Morphia projects with the largest NoSQL schemas over project lifetime (Scherzinger et al., 2020). For each commit version it locates entity classes, removes schema-irrelevant members, inspects remaining attributes, and compares the result with the previous commit to derive a schema history. The study reports that every analyzed project had at least one denormalized entity class, that schema size generally grows over time, and that schema-relevant commits range from 2.8% to over 30%, with 4 projects exceeding 20% (Scherzinger et al., 2020).

SkiQL begins after such extraction. Because U-Schema preserves structural variations, SkiQL can query not only whether a property exists, but whether it is shared, optional, or specific to a particular variation (Candel et al., 2022). SPARQL2XQuery solves an analogous implicit-access problem for XML by translating XML Schema into OWL and mappings into XPath sets, so that the system rather than the end user bears the burden of schema knowledge (Bikakis et al., 2013).

These studies indicate that unknown-schema reasoning often starts with schema recovery. This suggests that “unknown” may mean unknown to the user, unknown to the model, or absent as an explicit artifact even though recoverable from data, metadata, or code.

6. Evolution, migration, and modelling when schemas are not fixed

Unknown-schema conditions also arise when structure is changing rather than merely hidden. Automatic Recommendations for Evolving Relational Databases Schema addresses what it calls a scenario of “unknown schema change sequence,” where the DBA knows the intended evolution but does not fully know the sequence of operations required to realize it safely (Etien et al., 2024). The method uses a 3-step iterative process: compute impact, recommend corrective actions, and generate a valid SQL patch. A meta-model with structural entities, behavioral entities, and reified references allows the tool to identify impacted entities, partition them into coherent subsets, and compile ordered patches in which removals come first and creations or modifications come later. On the AppSI PostgreSQL system—95 tables, 62 views, 20 triggers, 64 stored procedures, and 19 trigger functions—the tool reproduced a non-trivial migration in about 15 minutes versus about 1 hour for the DBA, a 75% time reduction (Etien et al., 2024).

The live/local-first challenge paper generalizes schema change beyond databases. Its challenge problems include Extract Entity, bidirectional transfer between divergent schema versions, collaborative document refactoring in the presence of concurrent edits, formula migration after structural change, runtime state migration for live state-machine DSLs, and coordinated updates of State, Event, update, and render in an Elm-like architecture (Edwards et al., 2023). The paper treats schema change as coordination between code change and data migration, not as a post hoc cleanup step.

A more architectural critique appears in work on the “schema turn.” That paper argues that conceptual modelling has often privileged a conceptual schema completely separated from the information base, but that modern tooling enables an inclusive schema-and-base approach in which the base remains present from the beginning and can be used to derive, test, refine, or expand the schema (Partridge et al., 1 Sep 2025). In unknown-schema situations—when the schema is implicit in legacy systems, distributed across data sources, or not yet stable—the proposed response is pipeline-based, empirical, and iterative rather than schema-first.

Taken together, these works shift unknown-schema research from static inference toward dynamic maintenance. The central problem becomes preserving validity, intent, and continuity while structure is discovered or altered in flight.

7. Recurrent difficulties and methodological implications

Several difficulties recur across the literature. First, naming is an unstable signal. SchemaDB shows that foreign-key inference from names is helpful only up to exact or near-exact matches and becomes unreliable beyond edit distance 3 (Christopher et al., 2021). Schema-First Retrieval likewise observes that enterprise warehouses contain abbreviated tables and columns, hidden join conventions, and business concepts whose meaning may live only in metric definitions or query history rather than in physical identifiers (Agrawal et al., 23 Jun 2026).

Second, overexposure to raw schema can be as harmful as underexposure. TRUST-SQL is motivated by databases with hundreds of tables and massive noisy metadata, and Schema-First Retrieval explicitly contrasts focused retrieval with wrong or over-broad schema context (Jian et al., 17 Mar 2026, Agrawal et al., 23 Jun 2026). This suggests that unknown-schema robustness is not equivalent to loading more metadata; it often depends on selecting a verified or governed subset.

Third, benchmark regularity can mask real difficulty. SchemaDB criticizes public examples that are too regular, too small, or too pedagogical (Christopher et al., 2021). The NoSQL study shows that schema evolution is continuous and high-churn even in systems marketed as schema-free (Scherzinger et al., 2020). The live/local-first challenge problems emphasize that hidden invariants, runtime state, formulas, and replica divergence complicate schema change beyond what static type or DDL updates capture (Edwards et al., 2023).

Finally, several papers converge on a common design pattern: externalize structure, condition on it directly, and maintain a boundary between verified and unverified knowledge. STAR does this with schema graphs and template mappings, TRUST-SQL with a Verified Schema Knowledge set and a Propose checkpoint, Schema-First Retrieval with typed catalog objects and deterministic access-control gates, and inclusive conceptual modelling with schema-and-base pipelines (Mosig et al., 2020, Jian et al., 17 Mar 2026, Agrawal et al., 23 Jun 2026, Partridge et al., 1 Sep 2025). A plausible implication is that unknown-schema scenarios are best treated as problems of structured mediation rather than as cases for unconstrained end-to-end inference.