SLiCE: Composite Metric for Schema Lineage

• The paper introduces SLiCE, a composite metric that rigorously evaluates schema-lineage extraction by enforcing strict output format and exact source schema recovery.
• It assesses four elements—source schemas, source tables, transformation logic, and aggregation operations—using binary checks, fuzzy matching, BLEU scores, and multilingual AST comparisons.
• SLiCE improves data governance and downstream applications like text-to-SQL by mitigating semantic drift and ensuring reliable pipeline documentation.

Searching arXiv for the specified paper to ground the article and citation. arxiv_search.query({"search_query":"id:(Yin et al., 10 Aug 2025)","start":0,"max_results":5}) Schema Lineage Composite Evaluation (SLiCE) is a composite metric introduced in "Schema Lineage Extraction at Scale: Multilingual Pipelines, Composite Evaluation, and Language-Model Benchmarks" for evaluating automated extraction of fine-grained schema lineage from multilingual enterprise pipeline scripts (Yin et al., 10 Aug 2025). It is defined for settings in which enterprise data pipelines produce a semantic disconnect between original metadata and downstream data, a condition described as "semantic drift," which compromises data reproducibility and governance and impairs the utility of services such as retrieval-augmented generation (RAG) and text-to-SQL systems. Within that framework, schema-lineage extraction requires recovery of source schemas, source tables, transformation logic, and aggregation operations in a standardized representation, and SLiCE assesses both structural correctness and semantic fidelity.

1. Conceptual role within schema-lineage extraction

SLiCE was proposed for a task characterized as inherently multi-faceted. The extraction system must produce a strictly formatted output, identify exactly which source columns and tables feed into a target column, reconstruct arbitrary transformation code that may appear in SQL, Python, or C#, and recover any aggregation logic. The metric is therefore designed not as a single lexical similarity score, but as a structured evaluation of all four elements.

The motivation for a composite metric is tied to limitations attributed to existing metrics such as pass@k and CodeBLEU. In the formulation presented for schema-lineage extraction, those metrics either assume full, executable code or a single language, and they do not penalize format errors nor propagate downstream mistakes. SLiCE addresses four stated challenges: strict format requirements for automated parsing in enterprise systems, the requirement that binary mistakes in source-column selection should cancel all further credit, partial credit for table-name variations such as schema.table versus table, and semantic matching of multilingual, non-executable code fragments.

This design places schema-lineage evaluation closer to structured information extraction than to conventional code generation benchmarking. A plausible implication is that the metric is intended for operational settings in which parseability and column-level correctness are prerequisites rather than optional quality attributes.

2. Task formulation and lineage components

The framework defines the prediction target for a pipeline script $\mathcal{S}$ and a target schema $\sigma$ as a lineage

$$\hat{L} = \{ \text{source\_schema}: \hat{C}, \text{source\_table}: \hat{T}, \text{transformation}: \hat{F}, \text{aggregation}: \hat{A} \}$$

to be compared against a gold lineage

$L^\star = \{C^\star, T^\star, F^\star, A^\star\}.$

The required recovered elements are the following.

Component	Required content	Aligned submetric
Source Schemas $(C)$	All contributing columns	$M_{\text{src}}$
Source Tables $(T)$	All tables or files, including paths	$M_{\text{tbl}}$
Transformation Logic $(F)$	Code fragments such as renames, casts, filters	$M_{\text{trf}}$
Aggregation Operations $\sigma$0	GROUP BY keys and aggregate functions	$\sigma$1

A separate format check, $\sigma$2, verifies that all four keys appear in the required output form. The paper makes the alignment explicit: $\sigma$3 checks that all four keys appear, $\sigma$4 checks $\sigma$5 exactly, $\sigma$6 checks $\sigma$7 with fuzzy allowances, $\sigma$8 enforces semantic match on $\sigma$9, and $$\hat{L} = \{ \text{source\_schema}: \hat{C}, \text{source\_table}: \hat{T}, \text{transformation}: \hat{F}, \text{aggregation}: \hat{A} \}$$0 does the same for $$\hat{L} = \{ \text{source\_schema}: \hat{C}, \text{source\_table}: \hat{T}, \text{transformation}: \hat{F}, \text{aggregation}: \hat{A} \}$$1.

This decomposition is important because the target object is not a monolithic program. It is instead a column-level lineage description whose fields mix discrete set membership, string normalization issues, and multilingual code semantics.

3. Formal definition of the metric

SLiCE is defined as a gated composite score (Yin et al., 10 Aug 2025). The first gate is format correctness:

$$\hat{L} = \{ \text{source\_schema}: \hat{C}, \text{source\_table}: \hat{T}, \text{transformation}: \hat{F}, \text{aggregation}: \hat{A} \}$$2

The second gate is exact source-schema recovery:

$$\hat{L} = \{ \text{source\_schema}: \hat{C}, \text{source\_table}: \hat{T}, \text{transformation}: \hat{F}, \text{aggregation}: \hat{A} \}$$3

Source tables are evaluated by a hybrid of exact-match $$\hat{L} = \{ \text{source\_schema}: \hat{C}, \text{source\_table}: \hat{T}, \text{transformation}: \hat{F}, \text{aggregation}: \hat{A} \}$$4 and fuzzy similarity:

$$\hat{L} = \{ \text{source\_schema}: \hat{C}, \text{source\_table}: \hat{T}, \text{transformation}: \hat{F}, \text{aggregation}: \hat{A} \}$$5

with $$\hat{L} = \{ \text{source\_schema}: \hat{C}, \text{source\_table}: \hat{T}, \text{transformation}: \hat{F}, \text{aggregation}: \hat{A} \}$$6, and in experiments $$\hat{L} = \{ \text{source\_schema}: \hat{C}, \text{source\_table}: \hat{T}, \text{transformation}: \hat{F}, \text{aggregation}: \hat{A} \}$$7 and $$\hat{L} = \{ \text{source\_schema}: \hat{C}, \text{source\_table}: \hat{T}, \text{transformation}: \hat{F}, \text{aggregation}: \hat{A} \}$$8. The fuzzy term is

$$\hat{L} = \{ \text{source\_schema}: \hat{C}, \text{source\_table}: \hat{T}, \text{transformation}: \hat{F}, \text{aggregation}: \hat{A} \}$$9

where FuzzyMatch is based on normalized Levenshtein similarity.

Transformation and aggregation are evaluated identically, but on different code snippets:

$L^\star = \{C^\star, T^\star, F^\star, A^\star\}.$0

$L^\star = \{C^\star, T^\star, F^\star, A^\star\}.$1

with $L^\star = \{C^\star, T^\star, F^\star, A^\star\}.$2 for each group and experimental weights $L^\star = \{C^\star, T^\star, F^\star, A^\star\}.$3, $L^\star = \{C^\star, T^\star, F^\star, A^\star\}.$4, $L^\star = \{C^\star, T^\star, F^\star, A^\star\}.$5.

The multilingual AST term is defined as

$L^\star = \{C^\star, T^\star, F^\star, A^\star\}.$6

where $L^\star = \{C^\star, T^\star, F^\star, A^\star\}.$7 and $L^\star = \{C^\star, T^\star, F^\star, A^\star\}.$8 is the proportion of $L^\star = \{C^\star, T^\star, F^\star, A^\star\}.$9-specific keywords in the snippet, serving as a soft language attribution.

The full composite score is

$(C)$0

with $(C)$1, and in all experiments $(C)$2, $(C)$3, and $(C)$4.

Because $(C)$5 and $(C)$6 are binary, any format or column-selection error zeroes out the entire score. This enforces the stated principle that foundational errors invalidate any downstream logical match.

4. Benchmark construction and annotation protocol

The benchmark associated with SLiCE comprises 1,700 manually annotated column-level lineages drawn from 50 real-world pipeline scripts in SQL, Python/PySpark, and C# (Yin et al., 10 Aug 2025). Script difficulty is partitioned into Easy, Medium, and Hard by a three-point scheme: one point if there are at least three distinct sources, one point if there are chained transformations, and one point if there is any aggregation.

The annotation guidelines specify exact column names, semicolon-separated tables, code snippets broken at logical breaks with <CODEEND>, and strict dictionary format. Human experts also wrote reasoning traces for each script, with one trace for Easy, two for Medium, and three for Hard, to support chain-of-thought prompting.

These choices define both the evaluation target and the supervision regime. The emphasis on strict formatting and manually written reasoning traces indicates that the benchmark was built not only for leaderboard-style comparison, but also for studying prompting strategies and structured output compliance under realistic enterprise conditions.

5. Empirical behavior across models and prompting regimes

Experiments were conducted with 12 LLMs, spanning 1.3B to 32B small LLMs and larger models such as GPT-4o and GPT-4.1 (Yin et al., 10 Aug 2025). The reported results demonstrate that schema-lineage extraction performance scales with model size and with the sophistication of prompting techniques.

An excerpted set of average SLiCE scores, reported with standard deviation over six seeds, is as follows:

Model	Base	One-Shot	CoT-1
GPT-4.1	$(C)$7	$(C)$8	$(C)$9
GPT-4o	$M_{\text{src}}$0	$M_{\text{src}}$1	$M_{\text{src}}$2
Qwen2.5-32B	$M_{\text{src}}$3	$M_{\text{src}}$4	$M_{\text{src}}$5

Several empirical regularities are stated. Base zero-shot prompting is uniformly poor. One-Shot prompting yields gains of +60–100%. CoT-1 adds approximately +10–15 percentage points for models with at least 3B parameters. Within model families, larger models do better. The paper also states that Qwen2.5-32B with CoT-1 matches GPT-4.1 and GPT-4o under one-shot prompting.

The scaling and prompting analysis further reports that performance declines as script difficulty increases from Easy to Hard, but that this decline is mitigated by CoT prompting. Increasing few-shot examples from 1 to 2 yields large gains, whereas increasing from CoT-1 to CoT-2 yields smaller gains, indicating diminishing returns on multiple reasoning traces.

The paper’s abstract additionally highlights one result with deployment significance: a 32B open-source model, using a single reasoning trace, can achieve performance comparable to the GPT series under standard prompting. This suggests a scalable and economical approach for deploying schema-aware agents in practical applications.

6. Interpretation, applications, and limitations

The practical implications identified for SLiCE-driven extraction are concentrated in three areas (Yin et al., 10 Aug 2025). First, in data governance and documentation, the extraction pipeline can auto-generate near-perfect schema documentation for RAG systems, with the stated objective that downstream AI systems see correct field definitions and transformations. Second, in text-to-SQL, lineage context can be fed into a text-to-SQL engine to improve query accuracy by clarifying column origins and transformations. Third, for RLHF and fine-tuning, the decomposable SLiCE score is described as a natural reward for reinforcement learning or supervised fine-tuning of lineage-specialized models.

The paper also identifies several limitations and future extensions. Human-crafted reasoning traces are still needed for best chain-of-thought performance, and automating trace generation remains open. The metric currently uses fixed weights in its submetrics, and learning adaptive weights per domain is proposed as a possible improvement. In addition, the current AST matching ignores data-flow, and reintroducing a partial data-flow term is suggested as a way to boost strict semantic guarantees.

A common misunderstanding would be to treat SLiCE as merely a code-similarity metric. The formal definition does not support that interpretation. It is instead a metric in which format validity and exact source-column recovery are hard prerequisites, table matching permits controlled fuzziness, and transformation and aggregation logic are evaluated through a BLEU- and AST-based multilingual hybrid. Another misunderstanding would be to assume that higher lexical overlap alone is sufficient; the inclusion of $M_{\text{src}}$6 and the binary gates on formatting and source schemas explicitly reject that assumption.

In summary, SLiCE is a rigorously defined scalar metric with diagnostically separable components. It enforces strict output format, zeroes out on foundational mistakes, provides partial credit for table naming variations, and evaluates transformation and aggregation logic with multilingual AST-and-BLEU hybrids. Within the broader schema-lineage extraction framework, it functions both as a benchmark score and as a structured objective aligned with enterprise parsing, governance, and downstream model integration.