Annif: Automated Subject Indexing Toolkit

• Annif is a multilingual, open-source automated subject indexing toolkit developed by the National Library of Finland to assign controlled-vocabulary concepts to documents.
• Its project-based architecture integrates multiple backends like Bonsai, MLLM, and XTransformer with ensemble methods to tackle extreme multi-label text classification over large vocabularies.
• The system leverages targeted LLM modules for translation and synthetic data generation, enhancing multilingual normalization while relying on robust traditional NLP techniques.

Annif is a multilingual, open-source automated subject indexing toolkit developed by the National Library of Finland for assigning controlled-vocabulary concepts to documents automatically. In the cited 2025 shared-task systems, it is presented as a project-based framework for extreme multi-label text classification (XMTC) over large thesauri such as the GND subject vocabulary, with support for multiple backends, ensembles, hyperparameter optimization, and multilingual workflows. A central methodological point in both task papers is that Annif is not itself a LLM: its strongest recent systems remain fundamentally based on traditional NLP, machine learning, and XMTC components, while LLMs are used selectively for translation, synthetic data generation, and, in the GermEval system, candidate reranking (Suominen et al., 28 Apr 2025, Suominen et al., 21 Aug 2025).

1. Subject indexing problem and task settings

Annif is designed for the automatic assignment of controlled-vocabulary subject labels to bibliographic metadata records. In the 2025 LLMs4Subjects tasks, the input record consists of a title and abstract from TIBKAT, the bilingual bibliographic database of TIB, and the output is a ranked set of GND subjects. This is treated as a classic XMTC problem because the label space is extremely large. In SemEval-2025 Task 5, the organizers provided two GND variants: all-subjects, containing 200,035 GND subjects, and tib-core-subjects, containing 78,741 subjects. The corresponding train/development/test splits were 81,937 / 13,666 / 27,986 for all-subjects and 41,923 / 7,001 / 6,119 for tib-core-subjects. In GermEval-2025 LLMs4Subjects Subtask 2, the task used 200,035 GND subjects with TIBKAT splits of 90,452 train, 19,949 development, and 27,998 test records (Suominen et al., 28 Apr 2025, Suominen et al., 21 Aug 2025).

The task framing also clarifies what Annif is not. Although both shared tasks emphasized LLM-based subject indexing, the Annif systems deliberately did not use an LLM for the core task of subject selection in SemEval, and in GermEval the LLM was introduced only after classical candidate generation. This distinguishes controlled-vocabulary recommendation from free-form text generation and makes explicit that Annif operates as a framework for supervised and lexical subject recommendation rather than as a generative labeling system (Suominen et al., 28 Apr 2025, Suominen et al., 21 Aug 2025).

2. Project architecture and indexing backends

Annif’s architecture is project-based: a project specifies a vocabulary and backend configuration, and Annif trains and applies a subject indexing model for that setup. In the task systems, Annif functioned as the orchestration layer for multiple base indexing backends and ensemble models, allowing the same controlled vocabulary to be modeled through complementary algorithmic families (Suominen et al., 28 Apr 2025, Suominen et al., 21 Aug 2025).

Backend	Description	Role in the systems
Omikuji (Bonsai-style configuration)	Efficient multilabel tree-based methods including Parabel and Bonsai	Strongest supervised XMTC backend
MLLM	Maui-like Lexical Matching with heuristic features plus a small machine learning model	Lexical recall and coverage, especially for rare labels
XTransformer	Transformer-based XMTC and ranking method from PECOS using FacebookAI/xlm-roberta-base	Neural label ranking

The two papers present a stable ordering among these backends. In SemEval development experiments, the ordering was consistently Bonsai > XTransformer > MLLM across both all-subjects and tib-core-subjects. GermEval development results show the same pattern in both German and English. The interpretation offered in the papers is structural: Bonsai and XTransformer are similar in being supervised subject recognizers that struggle with low-frequency labels, whereas MLLM can suggest any GND concept if the preferred or alternate label appears lexically in the text (Suominen et al., 28 Apr 2025, Suominen et al., 21 Aug 2025).

The preprocessing and feature settings reported for these projects are intentionally conventional. The SemEval paper states that the base projects tested Snowball stemming and Simplemma lemmatization. For Bonsai, the authors enabled bigram features via ngram=2 and rare feature filtering with min_df between 2 and 5; the GermEval paper again reports ngram=2 and min_df=5. XTransformer was integrated experimentally into Annif in the SemEval work and refined during the task, while the GermEval system reused model-specific hyperparameters from earlier work, with final project configurations stored outside the paper text (Suominen et al., 28 Apr 2025, Suominen et al., 21 Aug 2025).

3. Multilingual normalization and LLM-based data construction

A defining feature of the recent Annif systems is that they are not directly multilingual at prediction time in one unified model. Instead, the authors create parallel monolingual pipelines, one for German and one for English. In SemEval, this yielded 12 base projects: 2 GND variants × 2 languages × 3 backends. Each original record was rewritten into a German-only and an English-only version, and language-specific models were trained on the corresponding monolingual data. The authors explicitly state that they did not model document type or language splits separately in SemEval because records often mixed languages and the types looked structurally similar (Suominen et al., 28 Apr 2025, Suominen et al., 21 Aug 2025).

The LLM integration in SemEval occurs in three places. First, Llama-3.1-8B-Instruct was used for record translation with vLLM on a single NVIDIA A100 GPU with 80GB VRAM on the University of Helsinki HPC cluster Turso, at about 4 records/second. The translation prompt instructed the model to ensure that the title and description were in the target language and, if already in that language, not to change or summarize them. Second, because MLLM requires record text and vocabulary terms to be in the same language, GND preferred labels were translated from German into English using GPT-4o-mini on Azure OpenAI Service, in batches of 100 terms, and bilingual GND SKOS files were created. Third, the authors generated synthetic bibliographic records with Llama-3.1-8B-Instruct by conditioning on an existing training record, preserving its manually assigned subjects, and adding one additional randomly chosen GND preferred term; synthesis ran at approximately 8 records/second. This was repeated three times per language and GND variant, i.e. $3 \times 2 \times 2$ rounds, yielding a final training mix of 1 part original and 3 parts synthetic. In SemEval, synthetic data were used only for Bonsai training; the first synthetic portion increased Bonsai’s nDCG@10 by approximately 0.02, and the total gain reached about 0.03 points (Suominen et al., 28 Apr 2025).

The GermEval system retains the same translation-assisted monolingual strategy but reformulates it around efficiency. The paper evaluates many small and efficient LLMs for translation and selects models according to

$score = nDCG@20 + \alpha \cdot throughput$

with $\alpha = 0.003$. The chosen translation models were G4 (google/gemma-3-4b-it) for English and A8 (CohereLabs/aya-expanse-8b) for German. For synthetic data generation, the authors used the four top-ranking LLMs from the translation experiments for each language, producing 4 synthetic datasets per language, each of size 90,452. The best GermEval setting for Bonsai used two copies of original data together with all four synthetic datasets; XTransformer used original data plus two synthetic datasets; MLLM again used no synthetic data. Reported development nDCG values show clear synthetic-data gains for Bonsai, including German orig: 0.5122 versus German orig+A8: 0.5513, and English orig: 0.5595 versus English orig+Q4: 0.5896 (Suominen et al., 21 Aug 2025).

4. Ensembles, score fusion, and candidate reranking

Annif supports ensemble construction, and both papers treat ensemble design as central. In SemEval, the authors use simple ensembles, which merge suggestions from multiple backends by averaging their numeric scores, and neural ensembles, which also average backend scores but then train a neural network to adjust subject scores, including suppressing frequent false positives. Three ensemble families were built: BM simple = Bonsai + MLLM, BM neural = Bonsai + MLLM with neural adjustment, and BMX simple = Bonsai + MLLM + XTransformer. Ensemble weights were tuned with annif hyperopt to maximize nDCG on the development set, with 100 tries for BM ensembles and 200 tries for BMX ensembles. The optimized weights consistently show Bonsai as the dominant contributor, XTransformer as a substantial secondary signal in BMX, and MLLM as a smaller but complementary signal (Suominen et al., 28 Apr 2025).

The GermEval paper extends this machinery in two directions. First, it introduces exponent-aware score fusion:

$f = \sum_{k=1}^{n} w_k \cdot x_k^{p_k}$

where $x_k$ is the prediction score from the $k$-th project, $w_k$ is its weight, and $p_k$ is its power exponent. The authors tuned 400 different combinations of weights and exponents for each project family using annif hyperopt. Second, they implemented a new LLM ranking ensemble in which merged candidate labels are reranked by an LLM, and the reranker’s score is blended with the classical score as

$f = w \cdot r^p + (1-w)x$

where $r$ is the LLM relevance score rescaled to $score = nDCG@20 + \alpha \cdot throughput$0. This LLM does not generate labels from scratch. Instead, the ensemble first produces candidate subjects, keeps the top $score = nDCG@20 + \alpha \cdot throughput$1 candidates with $score = nDCG@20 + \alpha \cdot throughput$2, and then asks the LLM to assign a 0–100 relevance score to each candidate keyword in JSON format. Two rerankers were tested: Q4 (Qwen/Qwen3-4B) and M24 (mistralai/Mistral-Small-3.1-24B-Instruct-2503). The tuned LLM-reranking parameters were weight = 0.1592, exponent = 7.898 for English and weight = 0.0921, exponent = 9.328 for German. The papers note that the LLM contribution is numerically small at score level but still improves ranking quality (Suominen et al., 21 Aug 2025).

A further ensemble layer operates across languages. In SemEval, the German and English monolingual pipelines were combined by summing subject scores and choosing the top 50 subjects by score; in GermEval, the pipeline generated 100 predictions from German and English project variants before merging and then retained the top 20 for submission. In both papers, this bilingual merge is treated as a major source of performance gains (Suominen et al., 28 Apr 2025, Suominen et al., 21 Aug 2025).

5. Evaluation methodology and empirical performance

The evaluation regimes differ across the two tasks but are reported in comparable ranking terms. During SemEval development, the authors used F1@5 and nDCG@10; official test evaluation used precision, recall, and F1 at thresholds from 5 to 50, with ranking by average recall across thresholds. In GermEval, development used F1@5 and nDCG@20, and official ranking also used nDCG@20, with Codabench additionally reporting precision, recall, F1, and R-precision (Suominen et al., 28 Apr 2025, Suominen et al., 21 Aug 2025).

The SemEval base-project results establish the strength of the traditional backends. For all-subjects, development scores were Bonsai de 0.3003 / 0.5144, Bonsai en 0.3234 / 0.5532, MLLM de 0.2428 / 0.4245, MLLM en 0.2281 / 0.4005, XTrans de 0.2928 / 0.5056, and XTrans en 0.3091 / 0.5326 for F1@5 / nDCG@10. For tib-core-subjects, the corresponding values were 0.2625 / 0.5233, 0.2934 / 0.5807, 0.2087 / 0.4157, 0.2160 / 0.4291, 0.2552 / 0.5052, and 0.2761 / 0.5419. The authors report that English models outperformed German models in all but one case (Suominen et al., 28 Apr 2025).

The best SemEval submission was Run #9, a BMX simple system with de+en combined predictions. In all-subjects, it achieved test F1@5 = 0.3432, Avg recall = 0.6295, and Rank 1. In tib-core-subjects, the best Annif run again used Run #9, obtaining test F1@5 = 0.3136, Avg recall = 0.5899, and Rank 2. The multilingual merge was operationally decisive: the authors state that without it, their best all-subjects result would have ranked 2nd rather than 1st, and their best tib-core result 3rd rather than 2nd. In qualitative evaluation, the selected tib-core run #9 ranked 4th in both scoring cases, with 0.5024 when Y and I were counted as correct and 0.4484 when only Y counted as correct (Suominen et al., 28 Apr 2025).

The GermEval system further improves on this pattern by adding LLM reranking. Development results show progressive gains from BM simple to BMX simple to BMX LLM. For German, BM simple reached F1@5 0.3344, nDCG@20 0.5923, BMX simple 0.3386, 0.6012, BMX LLM Q4 0.3507, 0.6259, and BMX LLM M24 0.3587, 0.6375. For English, the corresponding values were 0.3345, 0.6101, 0.3479, 0.6162, 0.3547, 0.6329, and 0.3600, 0.6412. On the official test set, the winning system was the BMX LLM M24 bilingual ensemble, which achieved F1@5 = 0.3288, nDCG@20 = 0.5697, and Rank 1. In qualitative evaluation, Annif ranked 1st with C1 = 0.8787 and C2 = 0.7699 (Suominen et al., 21 Aug 2025).

6. Limitations, interpretive issues, and broader significance

The papers offer a mixed but coherent interpretation of these results. Their strongest shared conclusion is that traditional supervised XMTC remains highly competitive when combined with carefully targeted LLM support. Bonsai provides the strongest standalone performance, XTransformer adds neural semantic ranking, MLLM contributes vocabulary coverage and low-frequency lexical recall, multilingual merging improves recall, and LLMs are used where they have high leverage: translation, synthetic data generation, and, in GermEval, candidate reranking. This suggests that Annif’s broader role is that of a hybrid subject indexing platform rather than an LLM replacement (Suominen et al., 28 Apr 2025, Suominen et al., 21 Aug 2025).

The limitations are equally explicit. In SemEval, qualitative performance lagged behind quantitative performance. The authors attribute this to the system being strongly shaped by the existing TIBKAT training annotations and therefore reproducing their style and biases, whereas other systems may have produced labels judged more useful by librarians even when they diverged from the dataset. They also report that the BM neural ensemble overfit the development set because it was trained on the same development data used for evaluation. Translation quality is a downstream determinant of indexing quality in the monolingual-normalization pipeline, and rare-label challenges remain because Bonsai and XTransformer struggle with very low training frequency while MLLM can only compensate when lexical overlap exists. The SemEval paper further notes that no system exceeded F1@5 = 0.35, which the authors suggest may indicate inconsistency in TIBKAT subject metadata, and that the mismatch between development metrics and official evaluation complicated system tuning, with nDCG@50 suggested as a better development metric than nDCG@10 for that task (Suominen et al., 28 Apr 2025).

The GermEval paper reframes some of these issues as quality-efficiency trade-offs. It repeatedly emphasizes small local open-weight models, explicit throughput-aware model selection, and the existence of several operating points: simple ensembles without LLM reranking for lower cost, a Q4 reranker as an efficient compromise, and M24 for best quality. At the same time, it states that the paper does not provide explicit latency numbers, token throughput values, memory measurements beyond the use of a single NVIDIA A100 GPU with 80GB VRAM, or energy and monetary cost estimates. It also notes the absence of a deeper error analysis of specific failure cases. Reproducibility is addressed operationally: the paper states that code, configuration files, and customized datasets are available on GitHub, and that models are available on Hugging Face Hub (Suominen et al., 21 Aug 2025).

Taken together, these papers define Annif as a controlled-vocabulary recommendation framework in which symbolic, lexical, statistical, neural, and LLM-based components can be combined without collapsing subject indexing into unrestricted generation. Its recent shared-task systems advocate a division of labor: use LLMs for multilingual normalization, data enrichment, and selective reranking, while retaining Annif’s traditional indexing models for the final extreme-label prediction task.