OntoLearner: A Modular Python Library for Ontology Learning with Large Language Models

Abstract: Ontology learning (OL) aims to automatically construct structured knowledge models from text, yet progress remains fragmented across methods, domains, and evaluation practices. Despite decades of research, OL lacks a shared infrastructure for systematic evaluation and ontology access. This absence has hindered progress and fragmented research, leaving the central challenges of OL largely unaddressed. We introduce OntoLearner, a modular, cross-domain, and first-of-its-kind framework that unifies ontology access, LLM-driven learning pipelines, and standardized benchmarking. OntoLearner releases 180 machine-readable ontologies spanning 22 domains and provides pipeline-ready datasets with train/dev/test splits for three core OL tasks: term typing, taxonomy discovery, and non-taxonomic relation extraction. Using this infrastructure, we conduct a large-scale empirical study of OL, evaluating 22 retrieval models and 12 LLMs across domains and tasks. The results converge on a finding that reframes the central challenge of OL: failure modes scale with ontological complexity rather than model size or architectural sophistication. The primary bottleneck is not model capability, but a structural mismatch between how models encode knowledge and how ontologies organize it. These findings establish that effective OL is reachable through the cross-domain, multi-task benchmarking enabled by OntoLearner. OntoLearner is open-source (MIT license) at https://github.com/sciknoworg/OntoLearner/.

• The paper presents OntoLearner’s main contribution: a unified, modular Python library standardizing ontology learning and benchmarking for LLMs.
• The methodology integrates 180 machine-readable ontologies and diverse OL tasks with LLM-driven pipelines, enabling reproducible evaluation across 22 domains.
• Empirical analysis reveals that bridging the structural gap between ontological logic and neural representations is crucial for advancing ontology learning performance.

OntoLearner: Infrastructure and Large-Scale Benchmarking for Ontology Learning with LLMs

Introduction and Motivation

The paper presents OntoLearner, addressing the chronic lack of unified infrastructure for ontology learning (OL) evaluation and integration with LLMs. The authors argue that progress in OL has been stymied not by limited model capabilities, but by the absence of systematic, cross-domain benchmarking, standardized datasets, and reusable pipelines—gaps that have fragmented advances across symbolic, statistical, and neural approaches.

Ontologies play a central role in neuro-symbolic AI as structured, interpretable conceptual frameworks. However, constructing and maintaining high-quality ontologies remains a substantial bottleneck, decelerating practical deployment of robust AI reasoning systems. Prior OL efforts have either focused on scalable but weakly-structured extraction or on infrastructure for hosting and discovery, without facilitating systematic automation or evaluation of ontology construction. Recent LLM-driven OL pipelines further compound this fragmentation: lacking shared benchmarks and evaluation settings, methodological progress is difficult to accumulate and compare.

OntoLearner fills this infrastructure void with a modular, domain-agnostic Python library that unifies access to 180 machine-readable ontologies from 22 domains, LLM-driven learning workflows, and comprehensive benchmarking with standard train/dev/test splits across term typing, taxonomy discovery, and relation extraction.

Architectural Overview

OntoLearner's architecture is grounded in modularity and separation-of-concerns, supporting extensibility, reusable benchmarking, and robust integration with evolving LLM and retrieval paradigms.

Figure 1: The conceptual and functional architecture of the OntoLearner library, illustrating its modular design for ontology access and learning within LLM workflows.

Key architectural components comprise:

• Ontologizer: A core module that abstracts ontology ingestion, parsing, and modularization, supporting inheritance-based extensibility. It enforces standard metadata schemas and enables direct Pythonic import, versioning, and FAIR-compliant documentation.
• Learning Tasks: Abstracted representations for core OL tasks. Each task—term typing, taxonomy discovery, and non-taxonomic relation extraction—comprises standardized input/output schemas and leakage-aware split generation, ensuring robust, reproducible experimental setups.
• Learner Models: Pluggable orchestration of model families via interface-segregated modules: AutoRetriever (embedding-based or lexical retrieval), AutoLLM (prompt-driven or finetuned LLMs), and AutoLearner (pipeline composition across RAG, hybrid, or ensemble strategies).
• Synthetic Corpus Generation: Text2Onto module for automatic creation of gold-standard, labeled corpora via controlled LLM verbalization, ensuring rigorous benchmarking in the absence of real-world annotation.
• Ontology Complexity Scorer: Pointwise metric combining logarithmic normalization, interpretable weighting, and sigmoid squashing, delivering a reproducible structural complexity score for each ontology. This supports sampling and evaluation stratified by ontological intricacy.

Figure 2: Architecture and module schema of the OntoLearner Python library, illustrating extensibility and future points of integration.

Ontology Benchmark Collection and Diversity

The OntoLearner benchmark consists of 180 ontologies, spanning materials science, biology, agriculture, medicine, general knowledge, and additional domains. This collection presents substantial structural diversity—class counts ranging from tens to over 200,000, and varying in depth, breadth, and relational complexity.

Figure 3: Distribution of 180 benchmark ontologies across 22 domains, reflecting OntoLearner's emphasis on cross-domain coverage and heterogeneity.

Comprehensive metric analysis and the complexity scorer reveal a heavy-tailed distribution of expressivity, with a small subset of highly complex, deeply hierarchical ontologies (e.g., SUMO, ChEBI, GO) and a majority of smaller, lightweight vocabularies. Importantly, benchmarking is stratified to assess method generalization and robustness beyond the high-resource biomedical and taxonomic domains.

Empirical Analysis of Retrieval and LLM Pipelines

The experimental protocol systematically characterizes failure modes and performance ceilings across 22 baseline retrieval and 12 LLM models. Tasks are formulated within a RAG framework, evaluating both canonical bi-encoders and LLM-augmented retrieval.

Figure 4: Recall performance of various retrieval models using top-k 15, by task; substantial variance is evident between term typing, taxonomy discovery, and non-taxonomic relation extraction.

Key findings:

• Retrieval Model Limitations: Embedding-based models exhibit high recall for term typing (>90%), but taxonomy discovery recall remains low (<40%) for all model sizes and families, including Qwen3-8B, Nomic, and dedicated sentence transformers. Lexical and traditional embeddings are sharply outperformed by modern retrievers on all but the most trivial tasks. Domain-specific embedding models (e.g., BiomedBERT, MatSciBERT) fail to generalize outside their focus domains and suffer from poor cross-domain robustness.
• LLM-Augmented Retrieval: Incorporating a generative candidate proposal stage via LLMs leads to consistent recall improvements (average +14–16%), with LLM-augmented Qwen3-8B recall increasing from 33.8% to 47.7% for taxonomy discovery. However, this augmentation does not close the semantic gap; retrieval limitations are fundamentally bounded by embedding anisotropy and poor separation of entailment from generic semantic relatedness.
• Numerical Highlights: Term typing F1-scores reach up to 83% (Finance, Qwen3-Next-80B), while taxonomy discovery in large, complex ontologies remains an open challenge, rarely exceeding 20–30% F1 in realistic search spaces.

Error Analysis and Ontology-Determined Failure Modes

Detailed analysis on ontologies distinguished by complexity (Gene Ontology, MDSOnto) exposes that failure modes scale with ontological structure, not with model capacity or family.

Figure 5: Gene Ontology topology illustrating dense taxonomic and non-taxonomic relations. Embedding representations collapse compositional distinctions, saturating vector space and undermining effective retrieval.

Figure 6: Recall@15 for retrieval models across taxonomy discovery and non-taxonomic RE tasks, with only modest gains from LLM-augmented retrieval, especially as complexity increases.

Figure 7: F1 scores of LLMs on taxonomy discovery and non-taxonomic RE across GO and MDSOnto, highlighting divergent model behavior as a function of ontology structure rather than parameter count.

Key observations:

• Embedding Collapse: Vector space occupation is thin and anisotropic for compositional, densely-related class labels, rendering is-a and entailment relations difficult to discriminate with cosine similarity.
• Calibration Pathologies: Models display high-confidence errors that track with ontology, not learning algorithm; over-prediction and hallucination rates correlate with hierarchical density, while MDSOnto triggers under-prediction.
• Output Discipline: Instruction-tuned LLM variants substantially outperform “thinking” or chain-of-thought models. Structured output, not unconstrained reasoning, is critical for OL extraction.
• Domain Ceiling: High-resource domains (e.g., OM for units, GoodRelations for finance) yield strong F1 and low perplexity due to likely pretraining exposure, while real-world complexity in specialized ontologies consistently limits performance.

Methodological and Theoretical Implications

A central empirical claim—repeatedly supported across benchmarks, retrievers, and LLMs—is that the dominant bottleneck in OL is the representational and structural mismatch between learned models and the deep organizational logic of ontologies, not merely lack of model capacity.

Contradicting a recurrent theme in neural-symbolic AI, scaling and architecture improvements (from BERT and GloVe to Qwen3-80B, Mistral-24B, or Gemma-27B) deliver at best modest and inconsistent gains for hard OL tasks rooted in hierarchical reasoning, logical entailment, and complex scheme compliance.

The infrastructure and findings of OntoLearner press for several immediate directions:

• Explicit contrastive learning and hyperbolic embedding spaces for improved separation of entailment/relatedness.
• Hybrid symbolic–neural approaches, integrating logical signals and symbolic features into neural architectures to overcome geometric and calibration pathologies.
• Expanded evaluation criteria: calibration error, contamination analysis, hallucination diagnostics, and error-biased measures—beyond F1—are necessary to expose true model behavior and failure typology.

Future Prospects and Community Impact

Practically, OntoLearner provides robust, maintainable, and extensible infrastructure for the development, benchmarking, and deployment of OL workflows, establishing the conditions for community-driven advancement akin to those that catalyzed progress in general NLP and information retrieval.

Theoretically, OntoLearner's findings reinforce that the symbiosis between symbolic and neural approaches is more than a matter of system integration; it demands infrastructure that forces models to address—rather than bypass—structural and logical constraints. Future AI developments should track precisely the measurable failure modes revealed here, particularly for applications in scientific discovery, explainable AI, and semantic web automation.

Conclusion

OntoLearner establishes the first systematic, modular infrastructure for ontology learning with LLMs, offering a comprehensive ontology benchmark, unified train/dev/test splits for core OL tasks, and robust, extensible pipelines for evaluation across classical, neural, and hybrid models (2607.01977). The empirical analyses demonstrate that effective OL depends less on scaling model size and more on closing the structural gap between neural model representations and ontological organization. This work provides the foundation and reproducibility necessary for accelerating method development, infrastructure standardization, and theoretical advances in hybrid neuro-symbolic AI.