TOPJoin: Context-Aware Join Search

• TOPJoin is a context-aware method that combines syntactic overlap, statistical estimates, and semantic similarity to identify joinable columns in enterprise data lakes.
• It employs a two-stage process: candidate identification using multiple indexes followed by a TOPSIS-based ranking to prioritize columns that yield meaningful join results.
• The term also spans generic top-k join operators in theory, linking practical data discovery with ranked enumeration in join processing.

Searching arXiv for recent and related papers on TOPJoin to ground the article in current literature. TOPJoin is a context-aware multi-criteria approach for joinable column search in enterprise data lakes, introduced to rank candidate join columns by combining syntactic overlap, join-cardinality estimates, statistical cues, and semantic similarity from values and metadata (Kokel et al., 15 Jul 2025). In a separate database-theoretic usage, the term also appears as a generic label for top-$k$ join operators and ranked-enumeration algorithms, rather than for one specific method (Tziavelis et al., 2020). The first usage addresses data discovery—finding tables that can meaningfully enrich an analysis—while the second concerns ordered production of join-query results under a ranking function.

1. Conceptual basis and formal problem statement

In the joinable-column-search literature, TOPJoin is defined against the “joinable column search problem”: given a collection of columns $C$, a query column $q^Q$ from a table $Q$, and a constant $k$, the task is to return the top-$k$ columns in $C$ that are joinable with $q^Q$ (Kokel et al., 15 Jul 2025). The key departure from earlier systems is that joinability is not reduced to column similarity alone.

The underlying motivation is enterprise heterogeneity. Data lakes may contain tens or hundreds of thousands of tables from DB2, NoSQL, SaaS, and open-data sources. In that setting, columns can share many values while denoting different concepts, or can appear close in embedding space while not referring to the same entities. The paper therefore defines context-aware joinable columns by two simultaneous conditions: the tables $Q$ and $T$ must possess a semantic relationship or belong to related domains, and the columns $C$0 and $C$1 must be semantically joinable in a way that produces meaningful results satisfying the query’s objectives (Kokel et al., 15 Jul 2025).

This definition is motivated by cases in which purely syntactic or embedding-based methods produce false positives or pathological join sizes. In the paper’s open-data example, a County column from Texas Child Population by County can be joined with County in CPS Purchased Services by County to enrich an analysis, but a join with Texas: All Cigarette and Tobacco Retailers may massively increase table size, and a join with Missouri Deaf and Hard of Hearing Demographics is not meaningful because the geography is unrelated. The same example also highlights the need for fuzzy matching, since values such as “Madison,” “madison,” “Madison County,” and “Madisonvil” may refer to the same real-world entity (Kokel et al., 15 Jul 2025).

Historically, join discovery had emphasized either syntactic similarity—set intersection, Jaccard, containment—or semantic similarity based on value embeddings, column embeddings, or semantically equivalent instances. TOPJoin is explicitly presented as a response to the insufficiency of those column-centric notions in enterprise settings, where the cost of false positive joins is high and systems must prioritize joins that are both statistically significant and contextually relevant (Kokel et al., 15 Jul 2025).

2. System architecture and retrieval workflow

TOPJoin is organized as a two-step framework: candidate identification followed by multi-criteria ranking (Kokel et al., 15 Jul 2025). The candidate-identification stage uses three strategies: syntactic joinability, metadata semantics, and value semantics. The ranking stage computes preference criteria over the merged candidate set and applies TOPSIS, the Technique for Order of Preference by Similarity to Ideal Solution.

The offline preprocessing stage constructs several representations for every column in the data lake. For each column, the system samples up to $C$2M values, builds a value-level inverted index or minhash index, computes minhash signatures, builds a metadata sentence and embeds it with Sentence-BERT, and constructs a value-semantics embedding from either all values or the most frequent values. These representations are stored in three indexes: an inverted index or minhash index for syntactic joinability, a nearest-neighbor index for metadata embeddings, and a nearest-neighbor index for value-semantics embeddings (Kokel et al., 15 Jul 2025).

At query time, the system retrieves candidates from each index independently. The query column is looked up in the inverted or minhash index to obtain syntactic candidates, its metadata sentence is embedded and passed to $C$3-nearest-neighbor search to obtain metadata candidates, and its values are embedded for value-semantics retrieval. The three candidate sets are then unioned; in the reported experiments, TOPJoin takes the top-100 results from each index before scoring the merged list (Kokel et al., 15 Jul 2025).

This architecture distributes “context” across multiple representational layers. Metadata contributes table names, descriptions, tags, organization identifiers, source information, and neighboring column names; values contribute lexical and semantic evidence; statistical summaries capture structural effects such as uniqueness and join cardinality. There is no explicit graph or probabilistic context model. Instead, context is encoded as feature vectors in embedding spaces together with scalar statistics, and these feed the TOPSIS ranking stage (Kokel et al., 15 Jul 2025).

3. Multi-criteria ranking and operational definition of context

For each candidate column $C$4, TOPJoin constructs a feature vector $C$5 and ranks candidates by their closeness to the TOPSIS ideal solution (Kokel et al., 15 Jul 2025). The reported criteria are uniqueness ratio, intersection size, join size, reverse join size, value semantics similarity, disjoint value semantics similarity, and metadata semantics similarity.

Criterion	Operational role	Signal type
Uniqueness ratio	Approximates primary-key likelihood	Statistical
Intersection size	Estimates value overlap via minhash	Syntactic
Join size / reverse join size	Penalizes cardinality blow-up	Structural
Value semantics	Compares frequent values semantically	Semantic
Disjoint value semantics	Tests non-overlapping values for conceptual similarity	Semantic
Metadata semantics	Compares table/column context	Contextual

The uniqueness criterion is defined as the number of distinct values divided by the number of rows in the column. Its intuition is relational: joins are often between a unique key and a non-unique foreign key, so higher uniqueness suggests stronger join potential. Intersection size is estimated rather than computed exactly. TOPJoin uses minhash with 100 permutations and estimates overlap or Jaccard similarity from the Hamming distance between signatures. This criterion receives the largest experimental weight: intersection size is set to $C$6, while all other criteria are set to $C$7, reflecting the design choice that exact overlaps are more trustworthy than embeddings (Kokel et al., 15 Jul 2025).

Join size and reverse join size are used to approximate the cardinality of LEFT JOIN Q ON q^Q = t^T and RIGHT JOIN Q ON q^Q = t^T. These estimates are intended to demote candidates that would produce many-to-many blow-ups or duplicate large parts of the input without adding meaningful information. The paper states that cardinality estimation follows Swami and Schiefer (Kokel et al., 15 Jul 2025).

The semantic criteria are deliberately separated. Value semantics compares embeddings derived from the most frequent values of two columns and is used only for string columns. Disjoint value semantics asks whether the values that do not overlap still denote similar entities; it compares the embeddings of $C$8 and $C$9, again only for string columns. Metadata semantics embeds a metadata sentence built from table and column names, descriptions, tags, organization IDs, and related metadata, and uses cosine similarity between those embeddings as a direct operationalization of contextual relatedness (Kokel et al., 15 Jul 2025).

TOPSIS then normalizes the decision matrix, applies the weight vector, defines an ideal solution by taking the best value for each criterion, computes each candidate’s distance to that ideal, and ranks by closeness. The computational cost of this ranking stage is described as straightforward $q^Q$0 per query once the candidate set has been fixed (Kokel et al., 15 Jul 2025).

4. Empirical evaluation, benchmarks, and observed behavior

TOPJoin is evaluated on two benchmarks: a real-world enterprise data lake called CIO and an academic benchmark derived from OpenData (Kokel et al., 15 Jul 2025). CIO is described as a replica of an Operational Data Store with 59 tables, data from multiple DB2 and NoSQL sources, and a mixture of string, integer, float, and date columns. Table sizes range from 0 to 90M rows, with an average of about 5.16M rows. Ground-truth joinable column pairs are derived from SQL query logs. OpenData is based on open government tables that had been used in prior join-search work, but the TOPJoin study introduces a new context-aware annotation protocol. Starting from 471 high-containment column pairs, 15 annotators reviewed table snippets, metadata, and example join results; only 42 of the 471 pairs were judged context-aware joinable under the paper’s threshold of at least 10% positive labels (Kokel et al., 15 Jul 2025).

The baselines are LSH Ensemble, an equality-join method based on set containment, and DeepJoin, a semantic join baseline based on column-value text embeddings. The study also evaluates TOPJoin-Minhash, an approximate version that replaces the full inverted index with a minhash-based syntactic index (Kokel et al., 15 Jul 2025).

The reported evaluation uses $q^Q$1 and the metrics MRR, MAP, and Recall@10.

Method	OpenData (MRR / MAP / Recall)	CIO (MRR / MAP / Recall)
LSH Ensemble	0.29 / 0.25 / 0.41	0.44 / 0.37 / 0.46
DeepJoin	0.20 / 0.16 / 0.29	0.44 / 0.36 / 0.47
TOPJoin	0.39 / 0.34 / 0.68	0.51 / 0.49 / 0.65
TOPJoin-Minhash	0.37 / 0.33 / 0.60	0.60 / 0.48 / 0.62

These results support the paper’s central empirical claim that TOPJoin performs better than the baselines on both benchmarks across all reported metrics (Kokel et al., 15 Jul 2025). On OpenData, Recall@10 rises from 0.41 for LSH Ensemble and 0.29 for DeepJoin to 0.68 for TOPJoin. On CIO, Recall@10 rises from 0.46–0.47 for the baselines to 0.65 for TOPJoin. The approximate TOPJoin-Minhash variant remains close to the full system in MRR and MAP, with some reduction in Recall.

The storage trade-off is particularly pronounced in the CIO setting. For that dataset, the paper reports a 2.6GB inverted index versus a 508KB minhash index. This suggests that the approximate variant is primarily motivated by enterprise-scale memory constraints rather than by a change in the scoring model itself (Kokel et al., 15 Jul 2025).

The reported limitations are also precise. Embedding-based methods can be misleading because strings representing different entities may still be close in embedding space; this is one reason intersection size is weighted more heavily. The authors also state that the current techniques used in TOPJoin for identifying semantically similar candidates are not finetuned for tabular data, and they identify tabular-data-specific fine-tuned models as a direction for future work (Kokel et al., 15 Jul 2025).

5. Relation to top-$q^Q$2 join processing and ranked enumeration

In database theory, the term “TOPJoin” is also used more generically for algorithms or operators that compute the top-$q^Q$3 results of a join under a ranking function, or support ranked enumeration of join-query results with theoretically justified optimality criteria (Tziavelis et al., 2020). This usage is conceptually distinct from the 2025 TOPJoin system. The data-lake system searches for joinable columns and tables; the theoretical literature assumes a fixed join query and asks how to enumerate output tuples in score order.

The tutorial “Optimal Join Algorithms Meet Top-k” emphasizes that classic top-$q^Q$4 algorithms such as Fagin’s algorithm, Threshold Algorithm, and Rank-Join were developed in a middleware access-cost model, whereas optimal join algorithms are analyzed in the RAM model with explicit attention to intermediate results, cyclicity, and worst-case output size (Tziavelis et al., 2020). Its argument is that top-$q^Q$5 joins and optimal joins should be studied together in a common model. In this literature, ranked enumeration is often framed as any-$q^Q$6 processing: the system should return answers in rank order with small delay, ideally without knowing $q^Q$7 in advance. For acyclic joins one target is $q^Q$8 or $q^Q$9; for cyclic queries with width parameter $Q$0, the target becomes $Q$1 with small per-output delay (Tziavelis et al., 2020).

A further extension appears in work on theta-joins. “Beyond Equi-joins: Ranking, Enumeration and Factorization” studies ranked enumeration for acyclic full join queries with inequality conditions, non-equalities, and band predicates, and proves that the top-$Q$2 answers can be returned in $Q$3 time under specific structural assumptions (Tziavelis et al., 2021). Its main technical device is a Tuple-Level Factorization Graph (TLFG), a compact DAG representation of binary theta-joins, which allows ranking over a factorized representation instead of over the flat join output.

The relation between these strands is methodological rather than terminological. The 2025 TOPJoin system uses TOPSIS over heterogeneous evidence to decide which columns are meaningful join targets in a data lake (Kokel et al., 15 Jul 2025). The top-$Q$4-join literature uses ranking functions over join tuples and studies how to enumerate answers with guarantees on preprocessing, delay, and dependence on structural parameters such as acyclicity, fractional edge cover, hypertree width, or submodular width (Tziavelis et al., 2020). Both lines are concerned with avoiding unhelpful combinatorial explosion, but they do so at different layers of the data-management stack.

6. Deployment contexts, limitations, and research directions

TOPJoin is explicitly motivated by deployment in data discovery systems and conversational interfaces, including settings such as the Semantic Automation Layer where a system must automatically propose join candidates in response to high-level questions (Kokel et al., 15 Jul 2025). In such environments, the practical requirement is not merely to find syntactically joinable columns, but to recommend joins that are domain-consistent, useful for enrichment, and unlikely to create pathological result sizes.

The implementation profile reflects that aim. Index construction is incremental: whenever a new table arrives in the data lake, its representations and indexes are precomputed. Query latency is then dominated by nearest-neighbor queries over embedding indexes, Hamming-distance or lookup operations over the syntactic index, and preference computation for a few hundred candidates. For large columns, the paper reports sampling up to 1M rows for semantic representations and 10K rows for the inverted-index construction used in experiments (Kokel et al., 15 Jul 2025).

Several limitations are explicit. The system relies on general-purpose pretrained LLMs for metadata and value semantics rather than models finetuned for tabular data. Embedding similarity can mislead when lexically or semantically nearby strings refer to different entities. The full inverted index can be heavy in large data lakes, motivating the approximate minhash variant. The system also assumes the availability of usable metadata, and its ranking procedure depends on offline preprocessing and index maintenance (Kokel et al., 15 Jul 2025).

The immediate research direction identified by the paper is the use of tabular-data-specific fine-tuned models within the TOPJoin framework (Kokel et al., 15 Jul 2025). A plausible implication is that future systems may tighten the interaction between candidate generation and context modeling, especially where metadata is sparse or entity representations are noisy. In the broader top-$Q$5-join literature, open problems include richer ranking functions, decomposition strategies tailored to ranked enumeration, integration with worst-case-optimal and factorized engines, and dynamic or incremental maintenance of ranked-enumeration guarantees under updates (Tziavelis et al., 2020). Together, these lines position TOPJoin both as a concrete data-lake join-discovery system and as part of a larger research program on ranking-aware join processing.