ESCO-PrereqSkill: Zero-Shot Prerequisite Benchmark

• ESCO-PrereqSkill is a benchmark that defines prerequisite skills as ordered dependencies using expert-curated ESCO data with names and descriptions.
• It uses a directed skill graph of 3,196 skills and evaluates models via semantic similarity metrics like Sentence-BERT and BERTScore.
• The dataset supports zero-shot generative inference, pairwise classification, and candidate ranking to enhance personalized learning and curriculum planning.

ESCO-PrereqSkill is a zero-shot benchmark for prerequisite skill prediction derived from ESCO, the European Skills, Competences, Qualifications and Occupations taxonomy. It is designed to test whether LLMs can infer foundational, “learn-before” relationships among skills using only names and natural language descriptions, without task-specific fine-tuning. The benchmark was introduced in “How Well Do LLMs Predict Prerequisite Skills? Zero-Shot Comparison to Expert-Defined Concepts” and comprises 3,196 target skills with expert-defined prerequisite links, positioning prerequisite inference as a semantic generation problem grounded in an expert-maintained occupational taxonomy (Le et al., 24 Jul 2025).

1. Provenance and conceptual scope

ESCO-PrereqSkill is curated from the ESCO “skills/competences” pillar, which is expert-maintained and multilingual. The benchmark contains 3,196 target skills; each has one or more expert-defined prerequisite skills. The ground-truth prerequisite lists are extracted by traversing ESCO relations that encode foundational requirements, specifically via properties “such as the inverse of isEssentialSkillFor,” and then filtering for skills with a clear name, a descriptive text, and at least one linked prerequisite in ESCO (Le et al., 24 Jul 2025).

Within the benchmark, a prerequisite is defined as a foundational concept that should be learned before a target skill. The directionality is explicit: if $p$ is a prerequisite of $t$, then $p \to t$. This directional interpretation distinguishes prerequisite structure from looser notions of semantic relatedness or co-occurrence. The benchmark therefore operationalizes prerequisite knowledge as an ordered dependency relation rather than as simple taxonomic similarity.

The paper frames prerequisite skills as important for supporting effective learning, assessment, and skill-gap analysis. Traditionally, such relationships are curated by domain experts, but this is characterized as costly to maintain and difficult to scale. ESCO-PrereqSkill addresses that scaling problem by providing an expert-grounded evaluation set for zero-shot inference, thereby isolating model reasoning about pedagogical structure from supervised adaptation (Le et al., 24 Jul 2025).

2. Data schema and graph representation

Each benchmark record corresponds to a target skill together with its expert prerequisites. The instance schema includes four fields: a stable ESCO identifier, a human-readable skill name, a natural language description from ESCO, and a list of ground-truth prerequisite skill names. ESCO also links skills to occupations and skill groups, but the benchmark instance schema includes only those four fields. Occupation-level context therefore requires cross-referencing skill identifiers with ESCO proper (Le et al., 24 Jul 2025).

An example given in the paper is the target skill “Supervise correctional procedures,” with skill ID 00064735-8fad-454b-90c7-ed858cc993f2 and the description: “Supervise the operations of a correctional facility or other correctional procedures, ensuring that they are compliant with legal regulations, and ensure that the staff complies with regulations, and aim to improve the facility's efficiency and safety.” Its ground-truth prerequisites are “Correctional procedures,” “Legal regulations in corrections,” “Staff management,” and “Safety protocols” (Le et al., 24 Jul 2025).

The benchmark is also formalized as a directed skill graph. Let $G = (V, E)$ denote the skill graph, where $V$ is the set of skill nodes and $E$ is the set of directed edges encoding prerequisite relations $p \to t$. In this release, $|V| = 3{,}196$. The adjacency matrix $A \in \{0,1\}^{|V| \times |V|}$ satisfies $A_{ij} = 1$ if skill $t$0 is a prerequisite of skill $t$1, and $t$2 otherwise. Graph density is defined as

$t$3

The paper reports $t$4 but does not enumerate $t$5 or derived statistics such as average degree, density, cycles, or strongly connected components, and it does not claim that the graph is a DAG (Le et al., 24 Jul 2025).

Because ESCO is built for occupational skills rather than course syllabi, cycles may exist. This suggests that ESCO-PrereqSkill is best treated as a directed dependency graph whose partial-order properties must be empirically checked rather than assumed. The paper accordingly notes that users interested in topological ordering should first test acyclicity or work on a per-component partial order, for example within SCCs (Le et al., 24 Jul 2025).

3. Construction procedure and supported tasks

The construction methodology centers on source extraction from ESCO. The selected skills are those that have a clear name, include a textual description, and are linked to prerequisites through ESCO relations such as the inverse of isEssentialSkillFor. The resulting prerequisite lists are treated as expert ground truth. Additional external expert validation beyond ESCO is not reported (Le et al., 24 Jul 2025).

For evaluation, model outputs are normalized by lowercasing, whitespace trimming, and punctuation removal. The paper does not describe additional de-duplication or language filtering beyond ESCO provenance. It also does not introduce negative sampling such as random non-prerequisites. Consequently, the benchmark evaluates set-level alignment between predicted and curated prerequisite lists rather than binary discrimination between positive and negative edges (Le et al., 24 Jul 2025).

The dataset is intended as a zero-shot test set. No train/dev/test splits are provided or used in the reported experiments. In the reported study, the evaluated task is zero-shot generation: given a target skill’s name and description, the model generates a list of prerequisite concepts, which is then compared to the expert list using semantic metrics (Le et al., 24 Jul 2025).

The paper also identifies additional tasks that the dataset supports but does not evaluate. These include pairwise directionality classification, in which a model predicts whether $t$6 is a prerequisite edge for a given pair $t$7, and candidate ranking, in which a model ranks candidate prerequisites for a target skill $t$8. A plausible implication is that ESCO-PrereqSkill can serve as a common substrate for both generative and graph-prediction paradigms, although only the zero-shot generative setting is reported in the benchmark study.

4. Prompting protocol, evaluated models, and metrics

The prompting strategy is standardized and zero-shot. The template has four components: context (“act as an education expert”), instruction (“list essential prerequisites”), input text consisting of the skill name and description, and a strict formatting constraint. For “Machine Learning,” the example prompt is: “List the essential prerequisite concepts or foundational knowledge areas needed to begin learning the skill: ‘Machine Learning’. Skill Description: ‘The ability to design algorithms that learn patterns from data and make predictions’. Respond only with a comma-separated list.” Models are instructed to return only a comma-separated list and no explanations; no tool use is employed (Le et al., 24 Jul 2025).

Thirteen models were evaluated via official APIs or endpoints, without fine-tuning. The model families listed are OpenAI (gpt-4.5-preview, gpt-4o, o1-mini), Anthropic (claude-3.5-haiku, claude-3.7-sonnet, claude-3-opus), Google (gemini-2.0-flash, gemini-2.5-pro-preview, Mixture-of-Experts family), Meta (llama4-maverick, llama4-scout), DeepSeek (deepseek-v3, deepseek-r1), and Alibaba (Qwen2-72B, 128K context). Temperatures, top-$t$9, and max tokens are not reported; models were called under provider default settings (Le et al., 24 Jul 2025).

Two semantic metrics assess alignment between predicted and expert prerequisite sets. Let $p \to t$0 be the ground-truth prerequisites for a target skill, and let $p \to t$1 be the model’s predictions after normalization. The first metric is semantic similarity via Sentence-BERT embeddings. The predicted and reference sets are concatenated into strings, embedded with Sentence-BERT $p \to t$2, and compared using cosine similarity:

$p \to t$3

with

$p \to t$4

This is intended to capture overall conceptual alignment, including paraphrasing and near-synonymy (Le et al., 24 Jul 2025).

The second metric is BERTScore, applied at the phrase level by matching each skill phrase in one set to its best counterpart in the other and averaging. Using the notation aligned to the paper:

$p \to t$5

$p \to t$6

$p \to t$7

No thresholds, classification metrics such as accuracy or F1, or ranking metrics such as MAP, MRR, or NDCG are used. Metrics are averaged over all skills (Le et al., 24 Jul 2025).

Latency is measured via API calls, and the paper reports minimum, mean, and maximum per-request latency in seconds on a log scale. Hardware and network conditions are not specified. This makes latency comparisons informative at the endpoint level, but not a controlled benchmark of deployment infrastructure.

5. Empirical findings and observed failure modes

The reported results show that several models generate prerequisite lists closely aligned with ESCO’s expert links. The strongest $p \to t$8 is achieved by llama4-maverick at 0.8347, followed by qwen2-72b at 0.8262, llama4-scout at 0.8256, and gemini-2.0-flash at 0.8238. Representative rows reported in the paper include llama4-maverick with $p \to t$9 and $G = (V, E)$0, claude-3.7-sonnet with $G = (V, E)$1 and $G = (V, E)$2, and gpt-4o with $G = (V, E)$3 and $G = (V, E)$4 (Le et al., 24 Jul 2025).

The paper also reports a precision–recall balance for top-performing systems. For example, llama4-maverick has $G = (V, E)$5 and $G = (V, E)$6. Some models achieve high $G = (V, E)$7 but lower semantic similarity. gemini-2.0-flash, for instance, attains high $G = (V, E)$8 with lower $G = (V, E)$9, which the paper interprets as a tendency toward lexical overlap and simple paraphrases rather than deeper semantic restructuring (Le et al., 24 Jul 2025).

Qualitatively, the study identifies two broad behavioral modes. One is surface-level lexical matching, where models generate synonyms and close paraphrases of ESCO labels. The other is conceptual decomposition, where models produce prerequisite structures that better reflect pedagogical organization. claude-3.7-sonnet and llama4-scout are described as exhibiting a balanced profile, gemini-2.0-flash as emphasizing lexical matching, and llama4-maverick as leading on $V$0 while maintaining competitive semantic similarity (Le et al., 24 Jul 2025).

Several failure modes are observed. These include lexical overlap bias, confusions between co-requisites and prerequisites, over-generalization such as predicting broad topics in place of more specific skills, and list hygiene problems including duplication or minor variants. The paper does not report prompt-sensitivity studies, ablations that remove descriptions, or alternative similarity functions. Robustness to prompt variation and domain shifts is instead listed as future work (Le et al., 24 Jul 2025).

Latency results indicate a trade-off between responsiveness and output quality. The paper highlights that gemini-2.0-flash, gpt-4o, and llama4-maverick are under approximately 2 seconds on average, whereas deepseek-r1 is reported at approximately 16.09 seconds average and gpt-4.5-preview at approximately 7.04 seconds average. This suggests that some high-performing models are compatible with interactive settings, while others may be more suitable for offline analysis (Le et al., 24 Jul 2025).

6. Applications, reproducibility, limitations, and future directions

The paper identifies several application domains for ESCO-PrereqSkill. In personalized learning and intelligent tutoring, inferred prerequisite gaps can be used to backfill missing fundamentals for a learner’s target skill. In skill-gap analysis and workforce development, prerequisite structures can be compared against a worker’s profile to produce upskilling paths. In curriculum planning and content sequencing, modules or resources can be reordered to respect inferred prerequisite structure. In skill-based recommendation, next-step skills can be recommended when their prerequisites are satisfied (Le et al., 24 Jul 2025).

An example pipeline consists of five stages: input of a target skill name and ESCO description; zero-shot LLM generation of prerequisites as a comma-separated list; normalization through lowercasing, trimming, and punctuation stripping, with optional mapping to ESCO skill inventory via fuzzy matching or embeddings; comparison against ESCO-PrereqSkill ground truth using Sentence-BERT semantic similarity and BERTScore; and action, such as planning learning paths or detecting gaps. The paper also provides evaluation pseudocode reflecting this sequence, including API invocation, normalization, embedding-based comparison, phrase-level BERTScore computation, and aggregation of scores and latency (Le et al., 24 Jul 2025).

Reproducibility is partially specified. The dataset repository is listed as https://github.com/lengocluyen/ESCO-PrereqSkill. The paper treats the dataset as a test set for zero-shot evaluation. Reproduction requires downloading the dataset, implementing the standardized prompt and normalization, querying the evaluated models via official APIs under default settings, computing Sentence-BERT cosine similarity and BERTScore at set or phrase level, averaging metrics across the 3,196 skills, and logging per-request latencies. License details are not specified in the paper, and scripts for preprocessing and metric computation are not explicitly linked (Le et al., 24 Jul 2025).

The benchmark’s limitations are directly tied to ESCO’s scope and editorial choices. Coverage may be biased toward European occupational contexts; multilingual aspects exist in ESCO, but the benchmark instance fields are presented in English. Domain drift may affect how models interpret skills whose meanings evolve. Because outputs may influence learning pathways and hiring, the paper recommends safeguards for fairness and transparency and retaining the human in the loop for high-stakes decisions (Le et al., 24 Jul 2025).

Future work is organized along data, models, methods, and causality. Proposed data extensions include multilingual prerequisites, quantification of graph structure such as $V$1, density, and SCCs, publication of negative samples for pairwise classification, and alignment with other taxonomies such as O*NET and ACM CCS. Proposed model directions include few-shot prompting, supervised fine-tuning on ESCO subsets, and hybrid neuro-symbolic approaches combining ontology constraints with generative reasoning. Proposed methodological extensions include ranking and classification tasks with MAP, MRR, and NDCG, robustness studies over prompt variants and description length, and alternative semantic scorers such as cross-encoders and instruction-tuned embedding models. The paper also proposes investigating causal inference on skill graphs and the temporal dynamics of prerequisite change as occupations evolve (Le et al., 24 Jul 2025).

Taken together, ESCO-PrereqSkill functions as an expert-grounded test bed for zero-shot prerequisite inference. Its principal significance lies in anchoring LLM outputs to curated occupational-skill structure while using semantic evaluation metrics designed for non-exact lexical overlap. The reported findings suggest that contemporary LLMs already encode substantial pedagogical structure, but also that practical deployment requires ontology-aware postprocessing, explicit attention to directionality, and human oversight in consequential settings (Le et al., 24 Jul 2025).