---
title: LLMs4OL 2025 Challenge Overview
url: https://www.emergentmind.com/topics/llms4ol-2025-challenge
type: topic
---

# LLMs4OL 2025 Challenge Overview

Searching arXiv for the main LLMs4OL challenge and related system papers to ground the article in current sources.
The **LLMs4OL 2025 Challenge** is a shared-task benchmark in **Large Language Models for Ontology Learning (LLMs4OL)** that frames ontology construction as a set of distinct but connected problems spanning extraction, typing, and hierarchy induction. Within the challenge literature, 2025 is presented as covering the “full ontology construction pipeline” through **Task A (“Text2Onto”)**, **Task B (“Term Typing”)**, and **Task C (“Taxonomy Discovery”)**, with task-specific systems rather than a single unified architecture proving effective in official results [2508.19428]. The challenge should be understood against the background established by the original LLMs4OL formulation—term typing, taxonomy discovery, and extraction of non-taxonomic relations as core ontology-learning tasks [2307.16648]—and by the first challenge edition in 2024, which showed strong progress on term typing but much weaker performance on harder structured prediction tasks [2409.10146]. For researchers preparing for or interpreting LLMs4OL 2025, the central technical theme is the tension between general-purpose LLM prompting, domain adaptation, retrieval-augmented methods, and lightweight learned structure induction.

## 1. Origins and challenge lineage

The LLMs4OL line begins with the proposal to evaluate large language models on ontology learning through three foundational tasks: **term typing**, **taxonomy discovery**, and **extraction of non-taxonomic relations** [2307.16648]. In that formulation, ontology learning is cast in terms of ontology primitives such as lexical entries \(L\), conceptual types \(T\), taxonomy \(H_T\), non-taxonomic relations \(R\), relation heterarchy \(H_R\), and axioms \(A\), with prompting used as the main interface for zero-shot evaluation [2307.16648]. The original study emphasized that out-of-the-box zero-shot LLM performance was uneven, with taxonomy discovery generally more reliable than fine-grained term typing in specialized domains, and with fine-tuning often necessary for practical ontology construction support [2307.16648].

The first challenge edition, **LLMs4OL 2024**, institutionalized this agenda as a community benchmark collocated with ISWC 2024 [2409.10146]. It organized ontology learning into **Task A — Term Typing**, **Task B — Taxonomy Discovery**, and **Task C — Non-Taxonomic Relation Extraction**, with both few-shot and zero-shot evaluation phases across resources including WordNet, GeoNames, UMLS, Gene Ontology, DBpedia Ontology, Schema.org, and FoodOn [2409.10146]. The resulting performance profile was sharply stratified: term typing could be near-ceiling in some settings, taxonomy discovery was substantially harder, and non-taxonomic relation extraction remained largely unsolved in the 2024 challenge setting [2409.10146].

The 2025 challenge, as described in the 2025 system paper, extends this trajectory by reframing the benchmark as a broader ontology-construction pipeline rather than a narrow collection of pairwise classification subtasks [2508.19428]. The 2025 paper explicitly defines **Task A (“Text2Onto”)** as joint extraction of terms and their types from raw domain documents, **Task B (“Term Typing”)** as assigning ontology types to given lexical terms in few-shot and zero-shot regimes, and **Task C (“Taxonomy Discovery”)** as inferring hierarchical is-a relations among a flat list of types [2508.19428]. This marks a substantive shift from the 2024 setup: extraction from documents is promoted to a first-class challenge component, and taxonomy induction is treated as graph inference rather than only pairwise truth-value prediction.

A plausible implication is that LLMs4OL 2025 represents a partial convergence between the original LLMs4OL task decomposition and more operational ontology-engineering workflows. That interpretation is consistent with the broader literature on prompt-based ontology generation in complex domains, where staged workflows commonly move from requirements and concept discovery to hierarchy formation and formalization [2412.02035].

## 2. Task structure in the 2025 edition

The 2025 challenge defines three tasks with different input modalities and prediction targets [2508.19428]. **Task A (“Text2Onto”)** is a document-conditioned extraction problem over raw domain texts. It is split into **A1 term extraction** and **A2 type extraction**, but the cited 2025 system treats both as jointly solvable in one generation pass from a document [2508.19428]. The task domains are stated as **Ecology, Scholarly, and Engineering** [2508.19428]. This differs from the older 2024 Task A, which was a term-typing problem over already extracted lexical items rather than raw documents [2409.10146].

**Task B (“Term Typing”)** in 2025 is the direct assignment of ontology types to given lexical terms, but it is evaluated in two settings: **few-shot** on known domains and **zero-shot** on unseen domains [2508.19428]. The few-shot domains are **OBI, MatOnto, and SWEET**, while the zero-shot setting corresponds to unseen domains in **B4–B6** [2508.19428]. This two-regime setup makes transferability explicit and formalizes a distinction already central to the 2024 challenge, where few-shot and zero-shot phases were also separated to test generalization across unseen ontologies [2409.10146].

**Task C (“Taxonomy Discovery”)** is defined as reconstructing a taxonomy from type names alone by predicting directed **is-a** relations among the types [2508.19428]. The 2025 paper states the Task C domains as **OBI, MatOnto, SWEET, DOID, Schema.org, PROCO, FoodOn, and PO** [2508.19428]. Unlike the 2024 formulation, which described taxonomy discovery as identifying valid superclass–subclass pairs [2409.10146], the 2025 system paper models it directly as adjacency-matrix prediction over type embeddings [2508.19428].

This task partition reflects a methodological claim made explicitly in the 2025 paper: extraction over documents, classification over isolated terms, and graph prediction over type sets place different demands on the system and therefore “benefit from task-specific methods rather than one unified architecture” [2508.19428]. This claim is also broadly compatible with evidence from earlier challenge and pre-challenge work: performance varies strongly by task formulation, domain, and prompt structure, and no single strategy dominates all ontology-learning subtasks [2307.16648].

## 3. Data representations and preprocessing regimes

A notable feature of the 2025 challenge literature is the amount of task-specific data repair and reformulation required before modeling [2508.19428]. For **Task A**, the released files are reported to include `documents.jsonl`, `terms.txt`, `types.txt`, `terms2docs.json`, and `terms2types.json`, but the 2025 system paper states that `terms2docs.json` was mislabeled and in fact mapped types to document IDs [2508.19428]. To reconstruct usable supervision, the authors “rescanned documents for exact matches from `terms.txt` and rebuilt a term-document index,” then merged this with `terms2types.json` to produce supervision tuples of the form
\[
\langle doc, \{term, type\}\rangle.
\]
This transformed the training data into a document-to-terms-and-types correspondence suitable for few-shot document-conditioned generation [2508.19428].

Each Task A training document is then converted into an instruction pair whose input concatenates the **title, full text, and the top-20 TF–IDF keywords**, while the output is a structured JSON-style object:
```json
{"terms":[...], "types":[...]}
```
[2508.19428]. The paper explicitly reports that adding **TF–IDF keywords improved recall by about 1.3 percentage points**, and this lexical augmentation was retained in the final system [2508.19428]. No stemming, lemmatization, or broader normalization beyond exact-match rescanning and TF–IDF enrichment is described.

In **Task B few-shot**, the representation becomes term-centered rather than document-centered. All training terms are embedded, and for each test term the top-\(k\) nearest labeled examples are retrieved and inserted into the prompt as demonstrations [2508.19428]. In the **Task B zero-shot** setting, the supervision representation changes again: both test terms and candidate type labels are embedded, and label assignment is performed by term–label similarity rather than by in-context analogy over same-domain training examples [2508.19428].

For **Task C**, the data representation is purely type-level. Taxonomy discovery is cast as adjacency prediction for a matrix
\[
A \in \mathrm{Mat}(N,N),
\]
where
\[
a_{ij} \in [0,1]
\]
represents the likelihood that \(type_i\) is a subclass of \(type_j\) [2508.19428]. The 2025 paper states that node representations are “embeddings of type labels,” produced by a Qwen-based encoder [2508.19428]. Training/validation splits are performed **by types rather than by edges**, retaining only is-a edges whose endpoints lie in the same split; this is intended to prevent leakage across node partitions [2508.19428].

The broader implication is that LLMs4OL 2025 is as much about representation design as model choice. That pattern is also visible in domain-specialized ontology-learning workflows outside the official challenge. In life-science ontology generation, for example, decomposition into categories, competency-question intermediates, and carefully curated ontology fragments was necessary to cope with token limits and structural complexity [2412.02035].

## 4. Methodological paradigms used in 2025 systems

The 2025 system paper is explicit in rejecting a single architecture for all subtasks [2508.19428]. Instead, it implements a deliberately heterogeneous design: **retrieval-augmented prompting** for extraction and few-shot typing, **embedding-based zero-shot classification** for unseen domains, and a **lightweight cross-attention layer** for taxonomy induction [2508.19428].

For **Task A**, the dominant method is **retrieval-augmented generation (RAG)** without finetuning [2508.19428]. All training and test documents are embedded using **Qwen3-Embedding-4B**, selected because it supports up to 32k-token inputs and offers “strong multilingual semantics” [2508.19428]. For each test document, the top-\(k\) nearest training examples are retrieved by cosine similarity with \(k=3\), and these examples are prepended to the prompt as few-shot demonstrations [2508.19428]. Output generation is constrained into a JSON-like format using the **Outline library**, after which predictions are concatenated, deduplicated, and written to `terms.txt` and `types.txt` [2508.19428].

For **Task B few-shot**, the same retrieval-augmented prompting recipe is reused over terms rather than documents [2508.19428]. The paper contrasts this with embedding-only baselines—a **Random Forest classifier on Qwen3 embeddings**, and the same classifier enhanced with graph-derived features from a term co-occurrence graph—stating that these baselines are explicitly weaker than few-shot RAG [2508.19428]. This provides a direct within-paper argument that in-context analogy over retrieved exemplars is stronger than direct embedding classification when same-domain supervision is available.

For **Task B zero-shot**, the method becomes a nearest-label assignment in embedding space [2508.19428]. The paper describes two single-model setups: **MPNet** (`all-mpnet-base-v2`, 768 dimensions, mean pooling) and **Qwen3-Embedding-4B** (2560 dimensions, last-token pooling), with embeddings **L2-normalized** and **cosine similarity** used to score term–type compatibility [2508.19428]. A three-model ensemble over **MPNet**, **Qwen3**, and **BGE** (`bge-large-en-v1.5`) is then formed through dynamically weighted score combination. The confidence rule is given as
\[
confidence = p_{\max}(1 - H_{\text{norm}})
\]
and the unnormalized weight as
\[
weight = 0.7\,confidence + 0.3(1 - H_{\text{norm}})
\]
followed by normalization so that
\[
\sum w = 1
\]
[2508.19428]. The paper states that final prediction is obtained by summing similarity scores across models weighted by these dynamic weights.

The same paper also evaluates a **DistMult-inspired** zero-shot Task B variant, with score
\[
\text{score}(t, y) = \mathbf{t}^\top \mathbf{y}
\]
for a term-type pair \((t,y)\), using mean-pooled Qwen3-Embedding-4B representations and adaptive z-score thresholding [2508.19428]. The authors note that unlike cosine similarity, this score can support multi-type predictions and may capture latent compatibility beyond lexical overlap.

For **Task C**, taxonomy discovery is modeled as graph inference via a single **lightweight cross-attention layer** [2508.19428]. Two identical sequences,
\[
\mathrm{TERMS}_1 = \mathrm{TERMS}_2,
\]
are fed through the encoder, projected into query and key spaces for child and parent candidates, and used to compute a predicted soft adjacency matrix
\[
\hat A \in [0,1]^{N \times N}
\]
[2508.19428]. Training uses **binary cross-entropy** against the incidence matrix of the true taxonomy, averaged over all valid \((i,j)\) pairs and counting missing edges as negatives [2508.19428]. Two encoder-attention configurations are compared: **4B-Frozen**, where Qwen3-Embedding-4B is frozen and only the cross-attention layer is trained, and **0.6B+LoRA**, where a smaller Qwen-0.6B encoder is augmented with **rank-8 LoRA adapters** with \(\alpha = 16\) and trained jointly with the same head [2508.19428].

These 2025 methods can be contrasted with earlier challenge baselines. In 2024 Task A, for instance, a small fine-tuned `flan-t5-small` baseline was already highly competitive on WordNet, whereas a semantic-tower retrieval augmentation often reduced aggregate F1 despite improving semantic grounding in some cases [2408.14236]. Taken together, these papers show that the main methodological fault lines in LLMs4OL are not simply “LLM versus non-LLM,” but rather **fine-tuning versus prompting**, **intrinsic versus extrinsic knowledge**, and **pairwise classification versus structured inference**.

## 5. Empirical performance and leaderboard patterns

The 2025 system paper reports official leaderboard placements across all three tasks and therefore provides one of the clearest views of what was competitive in the challenge [2508.19428]. In **Task A**, results differ by domain and by prompting configuration. On **A1 term extraction**, **Method 2** is better on Scholarly while **Method 1** is better on Engineering; reported \(F_1\) scores are **0.6471** versus **0.4884** on Scholarly, and **0.4418** versus **0.3277** on Engineering, depending on the method pairing [2508.19428]. On **A2 type extraction**, **Method 2 dominates across all domains**: **Ecology** \(F_1=0.5745\), **Scholarly** \(F_1=0.7586\), and **Engineering** \(F_1=0.4688\), versus much weaker Method 1 scores [2508.19428]. Official placements include **2nd** on **A1.2 Scholarly Terms**, **2nd** on **A2.1 Ecology Types**, **2nd** on **A2.2 Scholarly Types**, and **4th** on **A2.3 Engineering Types** [2508.19428].

In **Task B few-shot**, retrieval-augmented prompting strongly outperforms embedding-only baselines on the reported MatOnto comparison: **Few-shot RAG** achieves \(F_1=0.6053\), compared with **0.1188** for **Embeddings only** and **0.1772** for **Embeddings + Graph** [2508.19428]. Official rankings are **6th** on **B1 OBI** with \(F_1=0.7709\), **2nd** on **B2 MatOnto** with \(F_1=0.6053\), and **2nd** on **B3 SWEET** with \(F_1=0.6557\) [2508.19428].

In **Task B zero-shot**, the paper emphasizes that the best method depends on the subtask rather than ensembling being uniformly dominant [2508.19428]. On **B4**, **Qwen3 + simple QA prompt** reaches \(F_1=0.6560\), outperforming **MPNet + simple QA prompt** at **0.5652** and the **Ensemble** at **0.4783** [2508.19428]. On **B5**, **MPNet + simple QA prompt** is best with \(F_1=0.4722\), while on **B6** the **Ensemble** is strongest with \(F_1=0.1715\) [2508.19428]. Official placements are **2nd** on **B4**, **2nd** on **B5**, and **1st** on **B6** [2508.19428]. This is a useful corrective to any simplified narrative that dynamic multi-embedding ensembling is always superior.

In **Task C**, the clearest ablation is between frozen and adapted encoders [2508.19428]. The **0.6B+LoRA** system outperforms the **4B-Frozen** configuration across all eight evaluated ontologies: for example, **MatOnto** improves from **0.4426** to **0.5590**, **PROCO** from **0.0589** to **0.3865**, and **PO** from **0.1086** to **0.4817** [2508.19428]. Official rankings include **1st** on **DOID**, **1st** on **PROCO**, **1st** on **FoodOn**, **1st** on **PO**, **2nd** on **MatOnto**, **2nd** on **SWEET**, and **3rd** on **OBI** [2508.19428]. The same paper also shows that threshold selection matters substantially: **Sparsity-Matched** thresholding often beats **Validation-\(F_1\)**, especially on **PO**, **OBI**, **DOID**, and **FoodOn** [2508.19428].

To place these results in historical context, the 2024 overview already showed a pronounced difficulty gradient: **WordNet term typing** could reach **0.9938 F1**, while many taxonomy subtasks stayed in the **0.6** range or below and **Task C** remained near-zero in official submissions [2409.10146]. The 2025 evidence suggests that moving Task C to a learned graph-inference formulation substantially improves competitiveness, though scores remain far from saturation [2508.19428].

## 6. Recurring technical themes and open problems

One of the most persistent themes across the LLMs4OL literature is the distinction between **intrinsic** and **extrinsic** knowledge. The 2024 DSTI system paper formalizes this distinction through a comparison between fine-tuned `flan-t5-small` models and the same models augmented with **semantic towers**, a vector-store representation of type-level semantic primitives derived from Wikidata [2408.14236]. A semantic tower is defined as
\[
ST = \{s_1,s_2,..,s_n\},
\]
where \(s\) is “a domain semantic primitive pointing to a semantic property for a given domain” and \(n\) is “the minimal number of primitives needed to define the domain” [2408.14236]. Inference retrieves a top-1 candidate type by cosine similarity and injects it into the prompt as search result \(Z\) [2408.14236]. Empirically, however, this naive retrieval augmentation degrades F1 on both WordNet and GeoNames, even though it can improve semantic grounding on selected examples such as “into the bargain” or plural label forms in GeoNames [2408.14236]. The authors’ central claim is therefore a **trade-off between performance and semantic grounding** [2408.14236].

A second recurring theme is that **fine-tuning remains highly competitive**, especially on simpler label spaces. In the 2024 DSTI Task A study, a lightly fine-tuned `flan-t5-small` reaches **0.9820 F1** on the WordNet unofficial test set and **0.9716 F1** on the official A.1 leaderboard, placing near the top of the reported table [2408.14236]. This aligns with the broader 2024 challenge overview, which repeatedly associates top performance with fine-tuning and hybridization rather than pure prompting [2409.10146]. The 2025 system paper partly departs from this pattern by showing that RAG-style prompting can be highly competitive for Task A and Task B few-shot without encoder finetuning, but it still uses trainable adaptation for Task C and observes that the best zero-shot strategy varies by subtask [2508.19428].

A third theme is **scalability**. The 2024 DSTI paper explicitly states that the authors did not submit **GeoNames A.2** officially because “the length of the official test set” was too challenging on available resources [2408.14236]. Their training subset for GeoNames was reduced to **2041 terms representing all possible categories** via aggressive category-wise sampling [2408.14236]. This highlights a challenge-specific constraint that is easy to overlook: ontology-learning benchmarks can involve large label spaces and very large instance counts, so throughput and resource efficiency are often decisive.

A fourth theme is **task-specific decomposition rather than monolithic modeling**. The 2025 paper argues this directly for extraction, classification, and graph prediction [2508.19428], while life-science ontology-generation work outside the official challenge reaches a similar conclusion through a different route. In **LLMs4Life**, ontology construction in the AquaDiva domain is decomposed into requirement specification, competency-question generation, entity and property extraction, conceptual triple construction, Turtle serialization, ontology reuse, and iterative correction [2412.02035]. That workflow uses prompt engineering, reuse examples from ENVO, repeated structural constraints such as
\[
\text{SubClassOf count} \geq n - 1,
\]
and validation with RDFLib, HermiT, and Pallet [2412.02035]. This suggests that as ontology-learning tasks become more structurally rich, staged and modular pipelines become not just convenient but necessary.

A fifth theme is evaluation design. Although not a challenge paper, **MathArena** is relevant as a methodological reference because it treats benchmark design as a controlled experimental problem, emphasizing temporally fresh tasks, contamination annotations, mixed-format assessment, and uncertainty quantification [2505.23281]. This suggests a plausible direction for future LLMs4OL challenge design: greater attention to benchmark freshness, explanation quality, and robustness diagnostics rather than only static end-state scores.

## 7. Significance and outlook

The LLMs4OL 2025 Challenge occupies an important position in the evolution of ontology-learning benchmarks because it reframes the field from isolated classification subtasks toward a more complete ontology-construction pipeline [2508.19428]. In contrast to the original LLMs4OL paper, which was primarily a zero-shot evaluation of prompted LLMs on term typing, taxonomy discovery, and non-taxonomic relation extraction [2307.16648], and to the 2024 challenge overview, which revealed a sharp degradation in performance as tasks became more structured [2409.10146], the 2025 system literature shows that competitive results can be obtained through deliberate heterogeneity: RAG where local analogical structure matters, embedding-based label matching where zero-shot transfer is required, and lightweight learned adjacency prediction where graph structure must be induced [2508.19428].

At the same time, the challenge literature does not support a simplistic claim that larger or more general LLMs alone are sufficient. The 2024 and 2025 papers repeatedly show that performance depends on data reformulation, lexical augmentation, retrieval quality, threshold calibration, and domain-specific adaptation [2408.14236][2508.19428]. A plausible implication is that LLMs4OL is evolving into a benchmark not just for language models, but for **knowledge-aware system design** under ontology constraints.

The challenge is also significant as a bridge between benchmark-centric ontology learning and more realistic ontology-engineering applications. The move from term typing over fixed labels to document-to-ontology extraction in Task A, and from pairwise taxonomy judgments to adjacency-matrix prediction in Task C, makes the benchmark more structurally aligned with practical ontology construction workflows [2508.19428]. This development resonates with domain-specialized work in life sciences, where ontology generation requires ontology reuse, staged conceptualization, and iterative formal verification rather than direct one-shot prompting [2412.02035].

The most conservative synthesis is therefore this: **LLMs4OL 2025 demonstrates that ontology learning is not a single LLM capability but a family of related tasks with distinct inductive biases and system requirements**. Retrieval-augmented prompting is effective for extraction and few-shot typing when semantically similar examples can be surfaced; embedding-based matching is useful for zero-shot label transfer but sensitive to prompt style and label-set characteristics; and taxonomy induction benefits from explicit learned structure rather than relying solely on prompted judgments [2508.19428]. The remaining open questions—robust cross-domain transfer, semantic grounding without F1 degradation, scalable inference on large taxonomies, and stronger structure-aware evaluation—are already visible in the 2024 and 2025 papers and are likely to remain central to future editions [2408.14236][2409.10146].

Source: https://www.emergentmind.com/topics/llms4ol-2025-challenge