TalentCLEF 2025 Benchmark

• TalentCLEF 2025 is a benchmark campaign for Human Capital Management that evaluates multilingual job title matching and skill prediction.
• The campaign employs real de-identified job application data combined with manual annotations to provide a transparent and realistic evaluation framework.
• It rigorously applies metrics like Mean Average Precision and Rank-Biased Overlap across diverse language and fairness scenarios to inform future HCM research.

Searching arXiv for TalentCLEF 2025 overview and representative system papers. TalentCLEF 2025 was the first evaluation campaign focused on skill and job title intelligence for Human Capital Management (HCM). It introduced two ranking tasks—Task A, Multilingual Job Title Matching, and Task B, Job Title-Based Skill Prediction—and was designed to address the lack of public, standardized, realistic benchmarks for labor-market NLP. Its benchmark design combined real de-identified job application data, manually annotated development and test sets, multilingual and cross-lingual evaluation, and a dedicated analysis of gender-marked expressions in Spanish and German (Gasco et al., 17 Jul 2025).

1. Historical position and research rationale

TalentCLEF 2025 emerged from a setting in which language technologies were increasingly being applied to talent acquisition, onboarding and training, workforce planning, job-title normalization, skill extraction, title matching, and job-skill relation modeling. The benchmark paper identifies a structural obstacle in this area: most prior work relied on private data, and even when public resources existed, they often lacked transparent labeling criteria or reusable evaluation scripts. TalentCLEF 2025 was therefore conceived as an open benchmark infrastructure that was realistic, multilingual, fairness-aware, transfer-oriented, and reusable through public datasets and Codabench leaderboards (Gasco et al., 17 Jul 2025).

Within CLEF, the benchmark defined job-title matching and job-title-to-skill retrieval as information retrieval problems rather than closed-set classification. This framing matters because both tasks are intrinsically many-to-many: a query title can have multiple relevant titles or skills, and ranking quality is more informative than a single predicted label. A later overview of TalentCLEF 2026 explicitly describes that edition as the second edition of the challenge, which confirms the inaugural status of the 2025 campaign and shows that the 2025 benchmark became the reference point for subsequent task design in the series (Gasco et al., 30 Jun 2026).

2. Task definitions and corpus construction

Task A addressed multilingual job title matching. The input was a query job title derived from a job offer, the corpus consisted of candidate job titles extracted as the most recent titles from applicants’ resumes, and the output was a ranked list of corpus titles ordered by relevance. Task A covered English, Spanish, German, and Chinese. Its official settings included monolingual evaluation for en-en, es-es, and de-de, cross-lingual evaluation for en-es and en-de, a voluntary zh-zh Chinese track, and a gender-bias evaluation for Spanish and German. No training data were provided for Chinese, so the zh-zh track operated as a transfer setting (Gasco et al., 17 Jul 2025).

Task B focused on job title-based skill prediction. The input was an English job title from a job offer, the corpus was a set of skills, and the output was a ranked list of skills by relevance. Its training set was automatically generated from ESCO job-title to skill relevance relations, whereas its development and test sets were manually annotated using real application data. The task was English-only (Gasco et al., 17 Jul 2025).

The development and test corpora were derived from a large database of job descriptions and associated candidate applications. To maximize diversity, job offers were clustered using K-means and sampled from different clusters. For each selected job offer, a fixed number of relevant candidate profiles was retrieved, with relevance operationalized by the fact that the candidates had actually applied to the job. From candidate profiles, the benchmark creators extracted the most recent job title and a set of skills using a high-recall skill extraction system. The data were carefully anonymized or de-identified; for example, "CLOSED - Clinical Trial Manager – Remote" was transformed into "Clinical Trial Manager" (Gasco et al., 17 Jul 2025).

Task A annotation proceeded in two stages: an initial relevance judgment phase and an extensive review phase. Annotators were instructed to flag titles that were incomplete, ambiguous, or not genuine job titles, while the benchmark deliberately preserved natural variation, typographical errors, incomplete entries, and other noise characteristic of real recruiting data. After English relevance annotation, the dataset was manually translated into Spanish, German, and Chinese by professional linguists. For Spanish and German, masculine, feminine, and neutral forms were generated when applicable. The review phase was substantial: 28.4% of the initial matches were edited during review (Gasco et al., 17 Jul 2025).

Task B skill annotation used skills derived from candidate CVs. A total of 7,493 skills appearing at least 90 times in the dataset were selected and manually mapped to ESCO taxonomy v1.2.0. Four annotators labeled which ESCO skills were relevant to each job title, followed by an initial quality-control stage and a final complete expert review. The test set contained 2,400 annotations, and about a third of them were edited during review (Gasco et al., 17 Jul 2025).

The resulting benchmark sizes were substantial for a manually curated HCM resource. For Task A development, the benchmark contained 105 English queries with 2,619 corpus elements and 23.0 average relevant items per query; 185 Spanish queries with 4,661 corpus elements and 41.0 average relevant items per query; 203 German queries with 4,729 corpus elements and 41.5 average relevant items per query; and 103 Chinese queries with 2,513 corpus elements and 22.5 average relevant items per query. For Task A test, it contained 116 English queries with 769 corpus elements and 32.9 average relevant items per query; 191 Spanish queries with 1,231 corpus elements and 57.9 average relevant items per query; 226 German queries with 1,509 corpus elements and 65.1 average relevant items per query; and 116 Chinese queries with 769 corpus elements and 32.9 average relevant items per query. For Task B, the development set had 304 queries, 1,439 corpus elements, and 85.2 average relevant items per query, while the test set had 125 queries, 1,986 corpus elements, and 101.8 average relevant items per query (Gasco et al., 17 Jul 2025).

3. Evaluation protocol, metrics, and benchmark scenarios

Both tasks were evaluated on Codabench, with separate competitions for Task A and Task B. For both tasks, the primary official metric was Mean Average Precision (MAP), reflecting the benchmark’s retrieval-oriented design. For Task A, the benchmark additionally used Rank-Biased Overlap (RBO) in the gender-bias analysis, where rankings were compared under masculine and feminine title variants in Spanish and German. In this setting, high RBO indicated that rankings remained stable across gendered forms, whereas low RBO indicated stronger sensitivity to gender wording (Gasco et al., 17 Jul 2025).

Task A defined four official scenarios. The main leaderboard averaged monolingual MAP across English, Spanish, and German. A separate cross-lingual evaluation averaged en-es and en-de. The Chinese zh-zh track was voluntary because no Chinese training data were released. The gender-based evaluation reported average MAP across Spanish and German and average RBO between masculine and feminine query variants. Task B had a single official English-only scenario for job-title-to-skill ranking (Gasco et al., 17 Jul 2025).

The published baseline scores show that TalentCLEF 2025 was nontrivial even before considering multilingual transfer or fairness. For Task A, the baseline achieved average monolingual MAP 0.360, with 0.408 for en-en, 0.348 for es-es, and 0.324 for de-de. Its cross-lingual average was 0.340, with 0.335 for en-es and 0.345 for en-de. In the bias table, the baseline had average MAP(es,de) 0.336 and average RBO 0.915, with RBO(es) 0.893 and RBO(de) 0.937. For Task B, the baseline MAP was 0.196 (Gasco et al., 17 Jul 2025).

4. Participation and methodological landscape

TalentCLEF 2025 attracted 76 registered teams and more than 280 submissions overall. Task A had 66 registered teams, 16 teams that submitted at least one run in evaluation, 12 teams included in the benchmark, and 196 submissions across development and test. Task B had 68 registered teams, 10 teams that submitted at least one run, 8 teams included in the final benchmark, and 84 Codabench submissions (Gasco et al., 17 Jul 2025).

The methodological pattern across both tasks was notably consistent. Most systems relied on information retrieval techniques built with multilingual encoder-based models, often fine-tuned with contrastive learning. Several teams incorporated LLMs for translation, synthetic data augmentation, definition generation, or reranking. Common embedding families included bge-m3, multilingual-e5, and the GTE family; decoder-based embedding models such as gte-Qwen2-7B-instruct and Linq-Embed-Mistral also appeared. LLMs mentioned in participant pipelines included Gemma 2, Claude Sonnet 3.7, Qwen2.5, Llama 3.1, and gpt-4.1-nano (Gasco et al., 17 Jul 2025).

A representative system report is NLPnorth’s comparison of three method families: fine-tuned discriminative classification, fine-tuned contrastive learning, and prompt-based retrieval. For Task A, NLPnorth found that a prompting approach performed best, with an average of 0.492 MAP on test data over English, Spanish, and German. For Task B, its best result was 0.290 MAP with a fine-tuned classification-based approach. The team also augmented supervision by pulling language-specific titles and corresponding descriptions from ESCO for each job and skill, using those resources especially in the contrastive setup (Zhang et al., 23 Jun 2025).

The overview paper emphasizes that encoder-based approaches predominated and that participants generally did not rely on sheer model scale alone. This is important for interpreting the benchmark: TalentCLEF 2025 became not merely a ranking of architectures, but an empirical comparison of training strategy, multilingual transfer, data augmentation, reranking, and bias robustness under a shared evaluation protocol (Gasco et al., 17 Jul 2025).

5. Official results and representative systems

On the main Task A monolingual leaderboard, AlexU-NLP ranked first with average MAP 0.534, followed by TechWolf with 0.517 and pjmathematician with 0.515. AlexU-NLP obtained 0.559 on en-en, 0.527 on es-es, and 0.516 on de-de; TechWolf obtained 0.533, 0.519, and 0.500; pjmathematician obtained 0.563, 0.507, and 0.476. The next systems were NLPnorth at 0.492, NT at 0.464, and SCaLAR at 0.446 (Gasco et al., 17 Jul 2025).

In the cross-lingual Task A evaluation, pjmathematician and AlexU-NLP were tied at average 0.514, with pjmathematician obtaining 0.525 on en-es and 0.504 on en-de, and AlexU-NLP obtaining 0.516 on en-es and 0.512 on en-de. TechWolf followed at 0.504. The overview explicitly notes that cross-lingual MAP was generally lower than monolingual MAP, but still relatively high, which it interprets as evidence that multilingual semantic matching had improved substantially even though cross-lingual transfer remained challenging (Gasco et al., 17 Jul 2025).

The voluntary Chinese track provided an additional zero-shot transfer perspective. In the four-language ranking, AlexU-NLP obtained average 0.527, while pjmathematician and TechWolf both obtained 0.515. The best zh-zh score was 0.516 from pjmathematician, followed by 0.510 from TechWolf and 0.508 from AlexU-NLP (Gasco et al., 17 Jul 2025).

The Task A fairness evaluation showed a different ranking pattern. NLPnorth achieved average MAP(es,de) 0.469 together with average RBO 1.000, and Ixa achieved average MAP(es,de) 0.171 with average RBO 1.000. Among the highest-performing retrieval systems, AlexU-NLP had the best average MAP(es,de) at 0.521 with average RBO 0.976, while TechWolf had 0.510 with average RBO 0.971 and pjmathematician had 0.492 with average RBO 0.947. The benchmark paper emphasizes two findings: some systems produced perfectly invariant rankings under gendered lexical changes, and the strongest retrieval systems also exhibited very high fairness robustness (Gasco et al., 17 Jul 2025).

Task B exhibited a clearer separation between leading systems. The final leaderboard placed pjmathematician first with MAP 0.360, moali second with 0.345, and NLPnorth third with 0.290. They were followed by iagox at 0.278, Techwolf at 0.265, SkillSeekers at 0.224, COTECMAR-UTB at 0.215, the baseline at 0.196, and HULAT-UC3M at 0.141 (Gasco et al., 17 Jul 2025).

A later TechWolf system paper provides a detailed account of one of the benchmark’s strongest architectural lines. It introduces JobBERT-V3, a multilingual extension of JobBERT-V2 for cross-lingual job title matching, built from scratch on the multilingual backbone paraphrase-multilingual-mpnet-base-v2, which produces 768-dimensional embeddings, and augmented with an asymmetric linear projection layer that maps job-title embeddings from 768 to 1024 dimensions. The model was trained with InfoNCE-based contrastive learning on a multilingual corpus of 21,123,868 job titles across English, German, Spanish, and Chinese, constructed via synthetic translations using gpt-4.1-nano. On the official blind Task A test set, it reported MAP 0.533 for en-en, 0.519 for es-es, 0.500 for de-de, 0.510 for zh-zh, 0.510 for en-es, and 0.498 for en-de, closely matching the official TechWolf leaderboard profile in the overview paper (Decorte et al., 29 Jul 2025).

6. Analytical conclusions, limitations, and legacy

One of the benchmark’s strongest conclusions is that training strategy had a larger effect than model size alone. In Task A, the benchmark paper reports a positive average trend between model size and performance, but also states that the best MAP scores were achieved by models around 500M parameters and that these often outperformed much larger 7B decoder-based embedding models. In Task B, the paper states that LLM-based data augmentation plus contrastive fine-tuning could yield up to 8 MAP points improvement among similarly sized models, while larger decoder-based embeddings showed only a slight advantage when properly fine-tuned (Gasco et al., 17 Jul 2025).

At the same time, the benchmark had clear limitations. Chinese lacked training data, making the zh-zh track transfer-based. Task B was English-only. The overview does not provide inter-annotator agreement statistics, and it does not describe the baseline architecture in detail. Fairness evaluation centered on grammatical gender in Spanish and German rather than broader protected attributes. Training data for both tasks were partly generated automatically from ESCO, which means that benchmark learning was shaped in part by taxonomy-defined relations. The task scope was also deliberately narrow: the benchmark focused on job titles and skills rather than richer contextual candidate-job matching (Gasco et al., 17 Jul 2025).

The benchmark’s legacy is visible in the design of TalentCLEF 2026. The 2026 overview states explicitly that the previous edition’s Task A focused exclusively on job title matching and that Task B focused on retrieving the skills most relevant to a given job title. In the second edition, Task A was expanded to contextualized job-person matching over job vacancies and resumes in English and Spanish, while Task B was extended to distinguish between core and contextual skills. The 2026 paper also states that the Task B development and test sets refined and expanded the dataset used in the previous year and that the job-title queries were the same as in the previous edition. This continuity suggests that TalentCLEF 2025 established the foundational datasets, task formulations, and evaluation culture for a continuing CLEF benchmark line in Human Capital Management (Gasco et al., 30 Jun 2026).