OLaLa: Ontology Matching with LLMs

• The paper demonstrates that OLaLa systems combine retrieval, prompt engineering, and LLM inference to outperform traditional ontology matching with notable F₁ gains.
• OLaLa frameworks implement modular pipelines featuring dense retrieval, in-context scoring, and strict filtering to efficiently produce high-quality semantic mappings.
• The approach highlights challenges such as computational cost, prompt sensitivity, and domain adaptation, setting a foundation for future hybrid methodologies.

Ontology Matching with LLMs (OLaLa) refers to a class of methodologies and frameworks that leverage LLMs to establish semantic correspondences—such as equivalence, subsumption, or relatedness—between entities in heterogeneous ontologies or knowledge graphs. This area addresses the core challenge of semantic interoperability in data integration, biomedical informatics, digital libraries, and knowledge-based systems, where concept drift, lexical variation, and structural heterogeneity hinder automatic mapping using traditional or purely lexical methods. OLaLa systems seek to combine retrieval-augmented pipelines, prompt engineering, and embedding-based candidate selection with the generative and contextual strengths of modern LLMs to approach or even surpass the effectiveness of supervised ontology matching (OM) systems, without fine-tuning on extensive ground-truth alignments.

1. Problem Definition and Motivating Challenges

Ontology matching is the task of discovering semantic mappings $M \subseteq C \times D$ between concept sets $C$ and $D$ of two ontologies (or taxonomies), possibly labeling each mapping with relation types such as equivalence $(\equiv)$, subsumption, or domain-specific custom relations. The complexity of the OM task is exacerbated by context-dependent mappings, polysemy, granularity mismatches, and incomplete attribute information.

Key challenges motivating the OLaLa approach include:

• Scale and Subjectivity: Manual expert-driven OM does not scale, with low inter-annotator agreement in complex domains (e.g., only 22% unanimous decisions in biomedical essentiality mapping) (Itoku et al., 10 Jun 2025).
• Expressiveness: Need to generate not only simple equivalences but also expressive, aggregate, or relation-discovery correspondences, especially when structural heterogeneity is pronounced (Sousa et al., 19 Feb 2025).
• Insufficiency of Lexical Matching: Many correspondences require semantic interpretation beyond string similarity—demanding robust context modeling and paraphrase detection (Hertling et al., 2023).

2. Pipeline Architectures in OLaLa Systems

OLaLa frameworks instantiate a modular pipeline combining traditional candidate generation with LLM-based “in-context” inference, filtering, and matching. Architectures vary in aspects such as prompting, embedding models, seed alignment, and postprocessing, but share the following structure:

Stage	Method/Functionality	OLaLa Example
Candidate Generation	Dense retrieval (e.g., sBERT, Ada); high recall	multi-qa-mpnet-base-dot-v1 or sBERT (Hertling et al., 2023, Giglou et al., 2024)
Concept Verbalization	Label extraction, RDF verbalization, Turtle desc.	TextExtractorOnlyLabels, DescriptionInRDF (Hertling et al., 2023)
LLM Prompt/Scoring	Zero/few/many-shot; yes/no or multiple-choice	Prompt #7 with explicit examples
Seed Alignment	High-precision exact matches (label normalizing)	Trivial matcher unioned in
Filtering/Postprocess	Cardinality constraints, confidence thresholds	1:1 max-weight, $c \ge 0.5$ (Hertling et al., 2023)

OLaLa enables a separation of concern between “recall-oriented” retrieval and “precision-oriented” LLM filtering, reducing $O(n^2)$ scoring complexity to $O(n \cdot k)$ via top-$k$ candidate restriction (Giglou et al., 2024).

3. Prompt Engineering and Reasoning Modes

Prompt construction determines the LLM’s ability to correctly infer fine-grained equivalence or alignment relationships. OLaLa systems explore both zero-shot and few-shot prompting for binary and multi-class OM queries:

• Binary Prompt (Yes/No): “Classify if two descriptions refer to the same real-world entity (ontology matching). [...] Answer (yes or no):” (Hertling et al., 2023)
• Few-Shot Ensembles: Three positive and three negative task-specific examples are included in the prompt context, consistently reusing anatomy track data for all tracks (Hertling et al., 2023).
• Hierarchy-Aware Templates: Concept, concept-plus-parent, and concept-plus-children representations allow LLMs to leverage local hierarchy for disambiguation and refine decision making (Giglou et al., 2024).

Multi-stage prompt optimization, including hand-crafted, automated (MIPRO), and chain-of-thought (CoT) rationales, can enhance precision and calibration, with best results achieved by large in-context exemplars (e.g., F$_1=0.97$ for 200-shot, outstripping expert consensus) (Itoku et al., 10 Jun 2025).

4. Embeddings and Retrieval-Augmented Generation

State-of-the-art OLaLa pipelines use dense vector retrievers (SBERT, OpenAI Ada) to prefilter candidates. Embeddings are computed for each entity or verbalized fragment, and cosine similarity is used for efficient top-$k$ selection (Hertling et al., 2023, Giglou et al., 2024). For complex OM tasks (e.g., ABox-level relation discovery), LLM-derived embeddings are used at multiple integration points:

1. Instance Matching: Linking source and target instances via maximum cosine similarity (Sousa et al., 19 Feb 2025).
2. Label Similarity: Scoring all label pairs or aggregated representations between query and candidate subgraphs.
3. Subgraph and Query Aggregation: Aggregating hidden states of tokens, labels, or subgraphs to enable expressiveness beyond atomic equivalences.

Empirical studies demonstrate that LLM-derived embeddings yield substantial improvements in query-oriented F-measure (e.g., 0.68 vs 0.47 for Levenshtein, a ~45% increase) (Sousa et al., 19 Feb 2025).

5. Postprocessing, Filtering, and Alignment Construction

After LLM scoring, OLaLa systems typically apply:

• Threshold Filtering: Retain only correspondences $C$0; tunable parameter depending on application and confidence calibration (Hertling et al., 2023).
• Maximum-Weight Bipartite Matching: Enforce one-to-one or many-to-one alignment constraints to optimize for F$C$1 (Hertling et al., 2023).
• Seed Alignment Union: All exact-label matches are preserved to guarantee high precision without additional LLM queries.
• Human Validation: For low-confidence pairs, OLaLa frameworks include a human-in-the-loop review stage, using LLM-generated rationales to support decision-making. This optimization reduces expert workload to the ambiguous 2–3% of pairs when using a threshold $C$2 (Itoku et al., 10 Jun 2025).

6. Performance Benchmarks and Empirical Results

OLaLa has been evaluated on standard OAEI datasets (Anatomy, CommonKG, Knowledge Graph, Biodiversity, Bio-ML, MSE) and shown to deliver state-of-the-art or near-state-of-the-art performance on multiple semantic tracks:

Track / Task	OLaLa (F₁)	Best Comparator (F₁)	Model Baseline
Anatomy (mouse–human)	0.902	Matcha (0.941)	upstage/Llama-2-70b
CommonKG (NELL–DBpedia)	0.960	KGMatcher+ (0.950)	SBERT+LLM
Marvel Cinematic Universe	1.000	— (ties top)
Biodiv (ENVO–SWEET)	0.510	LogMap (0.713)
Bio-ML (best case)	0.721	BERTMap (0.628)	Flan-T5-XXL

Additional findings across multiple works:

• Many-shot CoT rationales and calibration yield F₁ up to 0.97, surpassing human agreement benchmarks (F₁ ≈ 0.68) (Itoku et al., 10 Jun 2025).
• LLM-based OM frameworks can outperform top OAEI systems in nine out of twenty benchmark tasks, especially where hierarchy context or lexical structure is discriminative (Giglou et al., 2024).
• For complex correspondences (e.g., structural/expressive OM), LLM-based embedding models deliver ∼45% relative F-measure gain over classic string-based baselines (Sousa et al., 19 Feb 2025).

7. Limitations, Open Challenges, and Future Directions

While OLaLa frameworks demonstrate strong empirical results and adaptability, several open challenges remain:

• Computation Cost: Inference over large candidate spaces (e.g., 20K+ pairs) incurs high latency, even with powerful GPUs (e.g., hours for ENVO–SWEET) (Hertling et al., 2023).
• Prompt and Parameter Sensitivity: No universally optimal prompt/model/extractor configuration has emerged; per-task tuning and meta-learning are ongoing research foci (Hertling et al., 2023, Giglou et al., 2024).
• Scalability: Scaling to very large ontologies or instance-level alignment remains computationally impractical without further optimizations (Hertling et al., 2023).
• Reliance on Labeled Data: Best performance requires carefully-curated few/many-shot exemplars or expert rationales for in-context learning (Itoku et al., 10 Jun 2025).
• Explanation and Justification: While chain-of-thought rationales can be generated, their production for every mapping is not yet feasible at scale (Itoku et al., 10 Jun 2025).
• Domain Adaptation: Performance can drop on specialized domains (e.g., biomedical, taxonomic) with heavy jargon or sparse labels, warranting domain-specific LLMs or fine-tuning.

Ongoing research explores chain-of-thought for structure-aware OM, retrieval-augmented pipelines for scalable candidate restriction, meta-learning for automated prompt selection, and hybrid symbolic/neural approaches for integrating reasoning and LLM inference (Sousa et al., 19 Feb 2025, Itoku et al., 10 Jun 2025, Giglou et al., 2024).

• OLaLa: Ontology Matching with LLMs (Hertling et al., 2023)
• Transforming Expert Knowledge into Scalable Ontology via LLMs (Itoku et al., 10 Jun 2025)
• LLMs4OM: Matching Ontologies with LLMs (Giglou et al., 2024)
• Complex Ontology Matching with LLM Embeddings (Sousa et al., 19 Feb 2025)
• Exploring LLMs for Ontology Alignment (He et al., 2023)
• Agent-OM: Leveraging LLM Agents for Ontology Matching (Qiang et al., 2023)