BDIViz in Action: Interactive Curation and Benchmarking for Schema Matching Methods

Abstract: Schema matching remains fundamental to data integration, yet evaluating and comparing matching methods is hindered by limited benchmark diversity and lack of interactive validation frameworks. BDIViz, recently published at IEEE VIS 2025, is an interactive visualization system for schema matching with LLM-assisted validation. Given source and target datasets, BDIViz applies automatic matching methods and visualizes candidates in an interactive heatmap with hierarchical navigation, zoom, and filtering. Users validate matches directly in the heatmap and inspect ambiguous cases using coordinated views that show attribute descriptions, example values, and distributions. An LLM assistant generates structured explanations for selected candidates to support decision-making. This demonstration showcases a new extension to BDIViz that addresses a critical need in data integration research: human-in-the-loop benchmarking and iterative matcher development. New matchers can be integrated through a standardized interface, while user validations become evolving ground truth for real-time performance evaluation. This enables benchmarking new algorithms, constructing high-quality ground-truth datasets through expert validation, and comparing matcher behavior across diverse schemas and domains. We demonstrate two complementary scenarios: (i) data harmonization, where users map a large tabular dataset to a target schema with value-level inspection and LLM-generated explanations; and (ii) developer-in-the-loop benchmarking, where developers integrate custom matchers, observe performance metrics, and refine their algorithms.

• The paper introduces BDIViz, a framework that integrates human-in-the-loop validation with live benchmarking for schema matching methods.
• It provides an extensible architecture supporting in-browser matcher integration, LLM-generated explanations, and dynamic ground truth updates.
• The system addresses schema alignment challenges in semantically complex domains, ensuring reproducibility and continuous algorithm improvement.

BDIViz in Action: Interactive Curation and Benchmarking for Schema Matching Methods

Introduction

BDIViz presents a comprehensive framework for interactive schema matching, addressing the persistent bottleneck of aligning heterogeneous data schemas, especially in high-stakes, semantically ambiguous domains. While automated methods (rule-based, embedding-driven, LLM-assisted) have individually advanced schema matching, real deployments continue to require substantial expert validation. Furthermore, development and comparative evaluation of matching algorithms are hampered by the scarcity of curated ground truth benchmarks and the lack of continuous in-situ feedback mechanisms. BDIViz addresses these challenges by providing an architecture that enables human-in-the-loop validation, benchmarking, and iterative matcher development, tightly integrated through a visualization-centric interface.

System Architecture and Workflow

The BDIViz architecture (Figure 1) comprises an extensible backend pipeline, an agentic LLM-based validation assistant, and a multi-panel frontend workspace.

Figure 1: BDIViz system overview, illustrating the coordinated flow of data, matcher outputs, user validation, and benchmarking integration.

Matching Pipeline and Dynamic Ground Truth

After raw schema uploads, the backend performs ontology inference (attribute semantics, groupings); matcher-agnostic candidate generation (ensemble execution of heuristic, embedding, and LLM-based matchers); and value profiling (distribution summaries, value-domain mapping proposals). User feedback (accept/reject/edit) is appended as session-level provenance and incrementally materializes as evolving ground truth, supporting immediate benchmarking and regression diagnostics without explicit re-labeling.

Extensible Matcher Integration and Benchmarking

Developers integrate new matchers via a standardized in-browser Python interface, registering arbitrary algorithms without modifying the backend. All matcher outputs are unified into the candidate artifact store for comparative visualization and live evaluation against the dynamic ground truth.

Agentic LLM Validation and Provenance

The Harmonization Assistant—implemented as a multi-agent supervisor architecture with vector-store memory (ChromaDB)—generates structured candidate explanations using heterogeneous signals: syntactic similarity, semantic description analysis, value-level compatibility, distributional evidence, and historical mapping context. The agent operates asynchronously and can perform tool-based workflow modifications, maintaining coherence between UI-driven and agent-driven state changes.

Scenario 1: Domain-Expert Data Harmonization

The biomedical data harmonization scenario demonstrates large-scale, expert-driven schema matching for a high-cardinality, high-semantic-complexity source-target pair (e.g., CPTAC dataset to GDC schema).

Task Setup and Matrix Navigation

Users initiate a session, upload source/target schemas, and interact with an ensemble-driven candidate heatmap. The interactive matrix encodes matcher consensus/confidence at cell level, supporting high-throughput triage and local drill-down. High-confidence matches are auto-accepted, minimizing manual load, while ambiguous regions are prioritized for review.

Ambiguity Resolution and LLM-aided Grounding

Ambiguous correspondences are diagnosed through coordinated panels: schema/attribute metadata, comparative value distributions, and LLM-generated explanations. The assistant synthesizes multi-factorial evidence to provide structured acceptance/rejection justifications, recommends value-domain transformations, and facilitates disambiguation in complex cases.

Figure 2: BDIViz interactive workspace for data harmonization, showcasing candidate triage, LLM-explanation panels, and value-level drill-downs.

Value Mapping and Provenance

Subsets of the matrix can be hierarchically filtered (ontology-aware axes) to limit scope to specific semantic regions, streamlining candidate review. The Value Comparison and Value Wrangler views enable alignment checking and edge-case validation for categorical/numeric attribute domains. The full action provenance and export functions guarantee reproducibility of integration decisions and downstream reusability.

Scenario 2: Developer-in-the-Loop Matcher Benchmarking

BDIViz operationalizes schema matcher evaluation as a live, iterative process, unifying curation and benchmarking.

Matcher Integration and In-situ Evaluation

Developers can write or paste custom matchers in-browser, leveraging arbitrary backends (e.g., HuggingFace models), exposing only a standardized interface. As curation proceeds and the live ground truth is populated, matcher quality metrics—Precision, F1, MRR—are recalculated in real-time.

Diagnostics and Failure Mode Analysis

Live analytics dashboards highlight ranking issues (e.g., BERT-based matchers producing low MRR despite adequate F1, suggesting poor top-$k$ utility in interactive review), matcher consensus/divergence patterns, and per-source attribute breakdowns. Consensus views (UpSet-style visualizations) reveal systematic method agreement/disagreement across evolving validation sets.

Figure 3: Developer benchmarking panel, supporting matcher upload, real-time metric updates, and consensus/failure analyses.

Empirical Insights

In the WikiData benchmark use case, lexical/syntactic matchers converge to near-optimal performance, with suggestions that more complex (embedding-based) methods offer negligible improvements on tasks dominated by attribute name similarity. This speaks to the necessity of in-context, task-specific, ground-truth-driven benchmarking for meaningful algorithmic progress.

Practical and Theoretical Implications

BDIViz reifies human-in-the-loop schema matching and benchmarking into a single, agent-augmented, provenance-aware workspace. Practically, this allows domain experts to iteratively curate high-quality ground truth datasets, propagate integration specifications, and enable reproducibility in data-centric science. Developers benefit from online feedback loops, enabling rapid matcher prototyping, robust failure mode diagnostics, and evidence-driven improvements.

Theoretically, the framework exposes the limitations of current benchmarks and underscores the importance of evolving, domain-native ground truth in the evaluation of schema matching algorithms. The coupling of user feedback, LLM-augmented assistance, and live benchmarking contributes to a more granular and context-aware understanding of algorithmic efficacy.

Conclusion

BDIViz provides an integrated environment for interactive schema curation, LLM-assisted ambiguity resolution, and in-situ schema matcher benchmarking—bridging the gap between automation and expert knowledge. By capturing user validation as first-class ground truth and operationalizing continuous evaluation, BDIViz sets a precedent for extensible, transparent, and repeatable data integration workflows. Future research directions include scaling agentic support for more complex entity sets, adapting matching workflows for multi-table or graph-structured data, and leveraging large-scale, longitudinal provenance for generalized matcher diagnostics and transfer learning.

(2604.10763)