LLMs4OL 2025 Challenge Overview

• LLMs4OL 2025 Challenge is a benchmark that redefines ontology construction by integrating document-based extraction, term typing, and taxonomy discovery into a cohesive pipeline.
• The challenge employs a mix of retrieval-augmented prompting, embedding-based classification, and lightweight cross-attention to address both few-shot and zero-shot scenarios.
• Empirical results highlight that task-specific system designs outperform unified architectures, emphasizing the importance of domain adaptation and scalable, knowledge-aware methods.

Searching arXiv for the main LLMs4OL challenge and related system papers to ground the article in current sources. The LLMs4OL 2025 Challenge is a shared-task benchmark in LLMs for Ontology Learning (LLMs4OL) that frames ontology construction as a set of distinct but connected problems spanning extraction, typing, and hierarchy induction. Within the challenge literature, 2025 is presented as covering the “full ontology construction pipeline” through Task A (“Text2Onto”), Task B (“Term Typing”), and Task C (“Taxonomy Discovery”), with task-specific systems rather than a single unified architecture proving effective in official results (Beliaeva et al., 26 Aug 2025). The challenge should be understood against the background established by the original LLMs4OL formulation—term typing, taxonomy discovery, and extraction of non-taxonomic relations as core ontology-learning tasks (Giglou et al., 2023)—and by the first challenge edition in 2024, which showed strong progress on term typing but much weaker performance on harder structured prediction tasks (Giglou et al., 2024). For researchers preparing for or interpreting LLMs4OL 2025, the central technical theme is the tension between general-purpose LLM prompting, domain adaptation, retrieval-augmented methods, and lightweight learned structure induction.

1. Origins and challenge lineage

The LLMs4OL line begins with the proposal to evaluate LLMs on ontology learning through three foundational tasks: term typing, taxonomy discovery, and extraction of non-taxonomic relations (Giglou et al., 2023). In that formulation, ontology learning is cast in terms of ontology primitives such as lexical entries $L$, conceptual types $T$, taxonomy $H_T$, non-taxonomic relations $R$, relation heterarchy $H_R$, and axioms $A$, with prompting used as the main interface for zero-shot evaluation (Giglou et al., 2023). The original study emphasized that out-of-the-box zero-shot LLM performance was uneven, with taxonomy discovery generally more reliable than fine-grained term typing in specialized domains, and with fine-tuning often necessary for practical ontology construction support (Giglou et al., 2023).

The first challenge edition, LLMs4OL 2024, institutionalized this agenda as a community benchmark collocated with ISWC 2024 (Giglou et al., 2024). It organized ontology learning into Task A — Term Typing, Task B — Taxonomy Discovery, and Task C — Non-Taxonomic Relation Extraction, with both few-shot and zero-shot evaluation phases across resources including WordNet, GeoNames, UMLS, Gene Ontology, DBpedia Ontology, Schema.org, and FoodOn (Giglou et al., 2024). The resulting performance profile was sharply stratified: term typing could be near-ceiling in some settings, taxonomy discovery was substantially harder, and non-taxonomic relation extraction remained largely unsolved in the 2024 challenge setting (Giglou et al., 2024).

The 2025 challenge, as described in the 2025 system paper, extends this trajectory by reframing the benchmark as a broader ontology-construction pipeline rather than a narrow collection of pairwise classification subtasks (Beliaeva et al., 26 Aug 2025). The 2025 paper explicitly defines Task A (“Text2Onto”) as joint extraction of terms and their types from raw domain documents, Task B (“Term Typing”) as assigning ontology types to given lexical terms in few-shot and zero-shot regimes, and Task C (“Taxonomy Discovery”) as inferring hierarchical is-a relations among a flat list of types (Beliaeva et al., 26 Aug 2025). This marks a substantive shift from the 2024 setup: extraction from documents is promoted to a first-class challenge component, and taxonomy induction is treated as graph inference rather than only pairwise truth-value prediction.

A plausible implication is that LLMs4OL 2025 represents a partial convergence between the original LLMs4OL task decomposition and more operational ontology-engineering workflows. That interpretation is consistent with the broader literature on prompt-based ontology generation in complex domains, where staged workflows commonly move from requirements and concept discovery to hierarchy formation and formalization (Fathallah et al., 2024).

2. Task structure in the 2025 edition

The 2025 challenge defines three tasks with different input modalities and prediction targets (Beliaeva et al., 26 Aug 2025). Task A (“Text2Onto”) is a document-conditioned extraction problem over raw domain texts. It is split into A1 term extraction and A2 type extraction, but the cited 2025 system treats both as jointly solvable in one generation pass from a document (Beliaeva et al., 26 Aug 2025). The task domains are stated as Ecology, Scholarly, and Engineering (Beliaeva et al., 26 Aug 2025). This differs from the older 2024 Task A, which was a term-typing problem over already extracted lexical items rather than raw documents (Giglou et al., 2024).

Task B (“Term Typing”) in 2025 is the direct assignment of ontology types to given lexical terms, but it is evaluated in two settings: few-shot on known domains and zero-shot on unseen domains (Beliaeva et al., 26 Aug 2025). The few-shot domains are OBI, MatOnto, and SWEET, while the zero-shot setting corresponds to unseen domains in B4–B6 (Beliaeva et al., 26 Aug 2025). This two-regime setup makes transferability explicit and formalizes a distinction already central to the 2024 challenge, where few-shot and zero-shot phases were also separated to test generalization across unseen ontologies (Giglou et al., 2024).

Task C (“Taxonomy Discovery”) is defined as reconstructing a taxonomy from type names alone by predicting directed is-a relations among the types (Beliaeva et al., 26 Aug 2025). The 2025 paper states the Task C domains as OBI, MatOnto, SWEET, DOID, Schema.org, PROCO, FoodOn, and PO (Beliaeva et al., 26 Aug 2025). Unlike the 2024 formulation, which described taxonomy discovery as identifying valid superclass–subclass pairs (Giglou et al., 2024), the 2025 system paper models it directly as adjacency-matrix prediction over type embeddings (Beliaeva et al., 26 Aug 2025).

This task partition reflects a methodological claim made explicitly in the 2025 paper: extraction over documents, classification over isolated terms, and graph prediction over type sets place different demands on the system and therefore “benefit from task-specific methods rather than one unified architecture” (Beliaeva et al., 26 Aug 2025). This claim is also broadly compatible with evidence from earlier challenge and pre-challenge work: performance varies strongly by task formulation, domain, and prompt structure, and no single strategy dominates all ontology-learning subtasks (Giglou et al., 2023).

3. Data representations and preprocessing regimes

A notable feature of the 2025 challenge literature is the amount of task-specific data repair and reformulation required before modeling (Beliaeva et al., 26 Aug 2025). For Task A, the released files are reported to include documents.jsonl, terms.txt, types.txt, terms2docs.json, and terms2types.json, but the 2025 system paper states that terms2docs.json was mislabeled and in fact mapped types to document IDs (Beliaeva et al., 26 Aug 2025). To reconstruct usable supervision, the authors “rescanned documents for exact matches from terms.txt and rebuilt a term-document index,” then merged this with terms2types.json to produce supervision tuples of the form

$\langle doc, \{term, type\}\rangle.$

This transformed the training data into a document-to-terms-and-types correspondence suitable for few-shot document-conditioned generation (Beliaeva et al., 26 Aug 2025).

Each Task A training document is then converted into an instruction pair whose input concatenates the title, full text, and the top-20 TF–IDF keywords, while the output is a structured JSON-style object: $H_R$0 (Beliaeva et al., 26 Aug 2025). The paper explicitly reports that adding TF–IDF keywords improved recall by about 1.3 percentage points, and this lexical augmentation was retained in the final system (Beliaeva et al., 26 Aug 2025). No stemming, lemmatization, or broader normalization beyond exact-match rescanning and TF–IDF enrichment is described.

In Task B few-shot, the representation becomes term-centered rather than document-centered. All training terms are embedded, and for each test term the top-$k$ nearest labeled examples are retrieved and inserted into the prompt as demonstrations (Beliaeva et al., 26 Aug 2025). In the Task B zero-shot setting, the supervision representation changes again: both test terms and candidate type labels are embedded, and label assignment is performed by term–label similarity rather than by in-context analogy over same-domain training examples (Beliaeva et al., 26 Aug 2025).

For Task C, the data representation is purely type-level. Taxonomy discovery is cast as adjacency prediction for a matrix

$A \in \mathrm{Mat}(N,N),$

where

$a_{ij} \in [0,1]$

represents the likelihood that $T$0 is a subclass of $T$1 (Beliaeva et al., 26 Aug 2025). The 2025 paper states that node representations are “embeddings of type labels,” produced by a Qwen-based encoder (Beliaeva et al., 26 Aug 2025). Training/validation splits are performed by types rather than by edges, retaining only is-a edges whose endpoints lie in the same split; this is intended to prevent leakage across node partitions (Beliaeva et al., 26 Aug 2025).

The broader implication is that LLMs4OL 2025 is as much about representation design as model choice. That pattern is also visible in domain-specialized ontology-learning workflows outside the official challenge. In life-science ontology generation, for example, decomposition into categories, competency-question intermediates, and carefully curated ontology fragments was necessary to cope with token limits and structural complexity (Fathallah et al., 2024).

4. Methodological paradigms used in 2025 systems

The 2025 system paper is explicit in rejecting a single architecture for all subtasks (Beliaeva et al., 26 Aug 2025). Instead, it implements a deliberately heterogeneous design: retrieval-augmented prompting for extraction and few-shot typing, embedding-based zero-shot classification for unseen domains, and a lightweight cross-attention layer for taxonomy induction (Beliaeva et al., 26 Aug 2025).

For Task A, the dominant method is retrieval-augmented generation (RAG) without finetuning (Beliaeva et al., 26 Aug 2025). All training and test documents are embedded using Qwen3-Embedding-4B, selected because it supports up to 32k-token inputs and offers “strong multilingual semantics” (Beliaeva et al., 26 Aug 2025). For each test document, the top-$T$2 nearest training examples are retrieved by cosine similarity with $T$3, and these examples are prepended to the prompt as few-shot demonstrations (Beliaeva et al., 26 Aug 2025). Output generation is constrained into a JSON-like format using the Outline library, after which predictions are concatenated, deduplicated, and written to terms.txt and types.txt (Beliaeva et al., 26 Aug 2025).

For Task B few-shot, the same retrieval-augmented prompting recipe is reused over terms rather than documents (Beliaeva et al., 26 Aug 2025). The paper contrasts this with embedding-only baselines—a Random Forest classifier on Qwen3 embeddings, and the same classifier enhanced with graph-derived features from a term co-occurrence graph—stating that these baselines are explicitly weaker than few-shot RAG (Beliaeva et al., 26 Aug 2025). This provides a direct within-paper argument that in-context analogy over retrieved exemplars is stronger than direct embedding classification when same-domain supervision is available.

For Task B zero-shot, the method becomes a nearest-label assignment in embedding space (Beliaeva et al., 26 Aug 2025). The paper describes two single-model setups: MPNet (all-mpnet-base-v2, 768 dimensions, mean pooling) and Qwen3-Embedding-4B (2560 dimensions, last-token pooling), with embeddings L2-normalized and cosine similarity used to score term–type compatibility (Beliaeva et al., 26 Aug 2025). A three-model ensemble over MPNet, Qwen3, and BGE (bge-large-en-v1.5) is then formed through dynamically weighted score combination. The confidence rule is given as

$T$4

and the unnormalized weight as

$T$5

followed by normalization so that

$T$6

(Beliaeva et al., 26 Aug 2025). The paper states that final prediction is obtained by summing similarity scores across models weighted by these dynamic weights.

The same paper also evaluates a DistMult-inspired zero-shot Task B variant, with score

$T$7

for a term-type pair $T$8, using mean-pooled Qwen3-Embedding-4B representations and adaptive z-score thresholding (Beliaeva et al., 26 Aug 2025). The authors note that unlike cosine similarity, this score can support multi-type predictions and may capture latent compatibility beyond lexical overlap.

For Task C, taxonomy discovery is modeled as graph inference via a single lightweight cross-attention layer (Beliaeva et al., 26 Aug 2025). Two identical sequences,

$T$9

are fed through the encoder, projected into query and key spaces for child and parent candidates, and used to compute a predicted soft adjacency matrix

$H_T$0

(Beliaeva et al., 26 Aug 2025). Training uses binary cross-entropy against the incidence matrix of the true taxonomy, averaged over all valid $H_T$1 pairs and counting missing edges as negatives (Beliaeva et al., 26 Aug 2025). Two encoder-attention configurations are compared: 4B-Frozen, where Qwen3-Embedding-4B is frozen and only the cross-attention layer is trained, and 0.6B+LoRA, where a smaller Qwen-0.6B encoder is augmented with rank-8 LoRA adapters with $H_T$2 and trained jointly with the same head (Beliaeva et al., 26 Aug 2025).

These 2025 methods can be contrasted with earlier challenge baselines. In 2024 Task A, for instance, a small fine-tuned flan-t5-small baseline was already highly competitive on WordNet, whereas a semantic-tower retrieval augmentation often reduced aggregate F1 despite improving semantic grounding in some cases (Akl, 2024). Taken together, these papers show that the main methodological fault lines in LLMs4OL are not simply “LLM versus non-LLM,” but rather fine-tuning versus prompting, intrinsic versus extrinsic knowledge, and pairwise classification versus structured inference.

5. Empirical performance and leaderboard patterns

The 2025 system paper reports official leaderboard placements across all three tasks and therefore provides one of the clearest views of what was competitive in the challenge (Beliaeva et al., 26 Aug 2025). In Task A, results differ by domain and by prompting configuration. On A1 term extraction, Method 2 is better on Scholarly while Method 1 is better on Engineering; reported $H_T$3 scores are 0.6471 versus 0.4884 on Scholarly, and 0.4418 versus 0.3277 on Engineering, depending on the method pairing (Beliaeva et al., 26 Aug 2025). On A2 type extraction, Method 2 dominates across all domains: Ecology $H_T$4, Scholarly $H_T$5, and Engineering $H_T$6, versus much weaker Method 1 scores (Beliaeva et al., 26 Aug 2025). Official placements include 2nd on A1.2 Scholarly Terms, 2nd on A2.1 Ecology Types, 2nd on A2.2 Scholarly Types, and 4th on A2.3 Engineering Types (Beliaeva et al., 26 Aug 2025).

In Task B few-shot, retrieval-augmented prompting strongly outperforms embedding-only baselines on the reported MatOnto comparison: Few-shot RAG achieves $H_T$7, compared with 0.1188 for Embeddings only and 0.1772 for Embeddings + Graph (Beliaeva et al., 26 Aug 2025). Official rankings are 6th on B1 OBI with $H_T$8, 2nd on B2 MatOnto with $H_T$9, and 2nd on B3 SWEET with $R$0 (Beliaeva et al., 26 Aug 2025).

In Task B zero-shot, the paper emphasizes that the best method depends on the subtask rather than ensembling being uniformly dominant (Beliaeva et al., 26 Aug 2025). On B4, Qwen3 + simple QA prompt reaches $R$1, outperforming MPNet + simple QA prompt at 0.5652 and the Ensemble at 0.4783 (Beliaeva et al., 26 Aug 2025). On B5, MPNet + simple QA prompt is best with $R$2, while on B6 the Ensemble is strongest with $R$3 (Beliaeva et al., 26 Aug 2025). Official placements are 2nd on B4, 2nd on B5, and 1st on B6 (Beliaeva et al., 26 Aug 2025). This is a useful corrective to any simplified narrative that dynamic multi-embedding ensembling is always superior.

In Task C, the clearest ablation is between frozen and adapted encoders (Beliaeva et al., 26 Aug 2025). The 0.6B+LoRA system outperforms the 4B-Frozen configuration across all eight evaluated ontologies: for example, MatOnto improves from 0.4426 to 0.5590, PROCO from 0.0589 to 0.3865, and PO from 0.1086 to 0.4817 (Beliaeva et al., 26 Aug 2025). Official rankings include 1st on DOID, 1st on PROCO, 1st on FoodOn, 1st on PO, 2nd on MatOnto, 2nd on SWEET, and 3rd on OBI (Beliaeva et al., 26 Aug 2025). The same paper also shows that threshold selection matters substantially: Sparsity-Matched thresholding often beats Validation-$R$4, especially on PO, OBI, DOID, and FoodOn (Beliaeva et al., 26 Aug 2025).

To place these results in historical context, the 2024 overview already showed a pronounced difficulty gradient: WordNet term typing could reach 0.9938 F1, while many taxonomy subtasks stayed in the 0.6 range or below and Task C remained near-zero in official submissions (Giglou et al., 2024). The 2025 evidence suggests that moving Task C to a learned graph-inference formulation substantially improves competitiveness, though scores remain far from saturation (Beliaeva et al., 26 Aug 2025).

6. Recurring technical themes and open problems

One of the most persistent themes across the LLMs4OL literature is the distinction between intrinsic and extrinsic knowledge. The 2024 DSTI system paper formalizes this distinction through a comparison between fine-tuned flan-t5-small models and the same models augmented with semantic towers, a vector-store representation of type-level semantic primitives derived from Wikidata (Akl, 2024). A semantic tower is defined as

$R$5

where $R$6 is “a domain semantic primitive pointing to a semantic property for a given domain” and $R$7 is “the minimal number of primitives needed to define the domain” (Akl, 2024). Inference retrieves a top-1 candidate type by cosine similarity and injects it into the prompt as search result $R$8 (Akl, 2024). Empirically, however, this naive retrieval augmentation degrades F1 on both WordNet and GeoNames, even though it can improve semantic grounding on selected examples such as “into the bargain” or plural label forms in GeoNames (Akl, 2024). The authors’ central claim is therefore a trade-off between performance and semantic grounding (Akl, 2024).

A second recurring theme is that fine-tuning remains highly competitive, especially on simpler label spaces. In the 2024 DSTI Task A study, a lightly fine-tuned flan-t5-small reaches 0.9820 F1 on the WordNet unofficial test set and 0.9716 F1 on the official A.1 leaderboard, placing near the top of the reported table (Akl, 2024). This aligns with the broader 2024 challenge overview, which repeatedly associates top performance with fine-tuning and hybridization rather than pure prompting (Giglou et al., 2024). The 2025 system paper partly departs from this pattern by showing that RAG-style prompting can be highly competitive for Task A and Task B few-shot without encoder finetuning, but it still uses trainable adaptation for Task C and observes that the best zero-shot strategy varies by subtask (Beliaeva et al., 26 Aug 2025).

A third theme is scalability. The 2024 DSTI paper explicitly states that the authors did not submit GeoNames A.2 officially because “the length of the official test set” was too challenging on available resources (Akl, 2024). Their training subset for GeoNames was reduced to 2041 terms representing all possible categories via aggressive category-wise sampling (Akl, 2024). This highlights a challenge-specific constraint that is easy to overlook: ontology-learning benchmarks can involve large label spaces and very large instance counts, so throughput and resource efficiency are often decisive.

A fourth theme is task-specific decomposition rather than monolithic modeling. The 2025 paper argues this directly for extraction, classification, and graph prediction (Beliaeva et al., 26 Aug 2025), while life-science ontology-generation work outside the official challenge reaches a similar conclusion through a different route. In LLMs4Life, ontology construction in the AquaDiva domain is decomposed into requirement specification, competency-question generation, entity and property extraction, conceptual triple construction, Turtle serialization, ontology reuse, and iterative correction (Fathallah et al., 2024). That workflow uses prompt engineering, reuse examples from ENVO, repeated structural constraints such as

$R$9

and validation with RDFLib, HermiT, and Pallet (Fathallah et al., 2024). This suggests that as ontology-learning tasks become more structurally rich, staged and modular pipelines become not just convenient but necessary.

A fifth theme is evaluation design. Although not a challenge paper, MathArena is relevant as a methodological reference because it treats benchmark design as a controlled experimental problem, emphasizing temporally fresh tasks, contamination annotations, mixed-format assessment, and uncertainty quantification (Balunović et al., 29 May 2025). This suggests a plausible direction for future LLMs4OL challenge design: greater attention to benchmark freshness, explanation quality, and robustness diagnostics rather than only static end-state scores.

7. Significance and outlook

The LLMs4OL 2025 Challenge occupies an important position in the evolution of ontology-learning benchmarks because it reframes the field from isolated classification subtasks toward a more complete ontology-construction pipeline (Beliaeva et al., 26 Aug 2025). In contrast to the original LLMs4OL paper, which was primarily a zero-shot evaluation of prompted LLMs on term typing, taxonomy discovery, and non-taxonomic relation extraction (Giglou et al., 2023), and to the 2024 challenge overview, which revealed a sharp degradation in performance as tasks became more structured (Giglou et al., 2024), the 2025 system literature shows that competitive results can be obtained through deliberate heterogeneity: RAG where local analogical structure matters, embedding-based label matching where zero-shot transfer is required, and lightweight learned adjacency prediction where graph structure must be induced (Beliaeva et al., 26 Aug 2025).

At the same time, the challenge literature does not support a simplistic claim that larger or more general LLMs alone are sufficient. The 2024 and 2025 papers repeatedly show that performance depends on data reformulation, lexical augmentation, retrieval quality, threshold calibration, and domain-specific adaptation (Akl, 2024, Beliaeva et al., 26 Aug 2025). A plausible implication is that LLMs4OL is evolving into a benchmark not just for LLMs, but for knowledge-aware system design under ontology constraints.

The challenge is also significant as a bridge between benchmark-centric ontology learning and more realistic ontology-engineering applications. The move from term typing over fixed labels to document-to-ontology extraction in Task A, and from pairwise taxonomy judgments to adjacency-matrix prediction in Task C, makes the benchmark more structurally aligned with practical ontology construction workflows (Beliaeva et al., 26 Aug 2025). This development resonates with domain-specialized work in life sciences, where ontology generation requires ontology reuse, staged conceptualization, and iterative formal verification rather than direct one-shot prompting (Fathallah et al., 2024).

The most conservative synthesis is therefore this: LLMs4OL 2025 demonstrates that ontology learning is not a single LLM capability but a family of related tasks with distinct inductive biases and system requirements. Retrieval-augmented prompting is effective for extraction and few-shot typing when semantically similar examples can be surfaced; embedding-based matching is useful for zero-shot label transfer but sensitive to prompt style and label-set characteristics; and taxonomy induction benefits from explicit learned structure rather than relying solely on prompted judgments (Beliaeva et al., 26 Aug 2025). The remaining open questions—robust cross-domain transfer, semantic grounding without F1 degradation, scalable inference on large taxonomies, and stronger structure-aware evaluation—are already visible in the 2024 and 2025 papers and are likely to remain central to future editions (Akl, 2024, Giglou et al., 2024).