---
title: 'AINL-Eval 2025: AI Abstracts Detection'
url: https://www.emergentmind.com/topics/ainl-eval-2025-shared-task
type: topic
---

# AINL-Eval 2025: AI Abstracts Detection

The AINL-Eval 2025 Shared Task is a benchmark and competitive evaluation initiative for the detection of AI-generated scientific abstracts in Russian. Motivated by the ascendancy of large language models (LLMs) in generating publishable-quality text and by the acute shortage of robust detectors for non-English (specifically Russian) scholarly output, the task aims to challenge and advance the state of the art in multilingual, multi-domain AI-generated text detection. Incorporating a carefully constructed, large-scale dataset and a two-phase evaluation framework, the shared task foregrounds generalization to both new scientific domains and previously unseen text generators, thereby establishing a challenging, extensible benchmark for academic integrity in Russian-language publishing [2508.09622].

## 1. Motivation and Objectives

The proliferation of LLMs has rendered machine-generated scientific text—particularly abstracts—indistinguishable from human-written content at scale. This trend, coupled with limited availability of non-English detection resources, endangers peer review processes and undermines public trust in research outputs when undetected synthetic abstracts permeate the scientific record. The AINL-Eval 2025 Shared Task was thus established with two principal objectives:

1. To assemble a benchmark Russian-language dataset spanning multiple scientific domains and LLMs.
2. To foster models capable of generalizing beyond both seen scientific domains (cross-domain detection) and known AI text generators (cross-model detection) [2508.09622].

The challenge targets systemic vulnerabilities arising from the adoption of LLMs in non-English academic environments, with Russian scientific publishing selected as the representative testbed.

## 2. Dataset Construction and Characteristics

The AINL-Eval 2025 dataset comprises 52,305 Russian scientific abstracts, evenly split between human-written and LLM-generated texts. Human abstracts were collected from twelve established Russian journals representing diverse domains, including Mathematics, Philology, Law, and Oil & Gas. For each human abstract, five LLMs (GPT-4-Turbo, Gemma 2-27B, Llama 3.3-70B, Deepseek-V3, and GigaChat-Lite) were used to generate corresponding synthetic abstracts in a controlled setting.

Generation procedure standardized prompts as follows:
> “Сгенерируй краткое содержание научной статьи по заголовку и ключевым словам. Напиши только текст аннотации. Не начинай текст аннотации с фразы ‘В данной статье’.”  
> Заголовок: {title}.  
> Ключевые слова: {keywords}.

Lightweight post-processing removed model-specific artifacts while preserving content. The dataset is partitioned into a 35,158-sample training set, a 10,978-sample development set, and a 6,169-sample blind test set. Critically, the test set incorporates two previously unseen scientific domains (Economics, Biology) and introduces outputs from Deepseek-V3, which was undisclosed to participants prior to testing, thereby explicitly encoding generalization benchmarks.

| Subset    | # Samples | Domains (Seen/Unseen) | LLMs Included           |
|-----------|-----------|----------------------|-------------------------|
| Train     | 35,158    | 12 (seen)            | 4 (seen)                |
| Dev       | 10,978    | 12 (seen)            | 4 (seen)                |
| Test      | 6,169     | 2 (unseen; Econ, Bio)| 1 unseen (Deepseek-V3)  |

## 3. Evaluation Protocol and Metrics

The shared task was structured around two phases:

- **Development Phase:** Participants accessed train/dev splits for ~2 months, submitting unlimited system runs on CodaLab. Ground-truth labels were hidden for the development set. The leaderboard reflected accuracy on the dev split.
- **Test Phase:** A one-week blind evaluation on the 6,169-sample private test set, strictly limited to five submissions per team to minimize leaderboard overfitting.

The official metric was accuracy, defined as  
$$
\text{Accuracy} = \frac{TP + TN}{\text{Total samples}}
$$
mirroring prior shared tasks (e.g., RuATD 2022). While related work frequently reports metrics such as precision, recall, F1  
$$
F_{1} = \frac{2 \cdot (\mathrm{precision} \cdot \mathrm{recall})}{\mathrm{precision} + \mathrm{recall}}
$$
and ROC AUC, these did not impact leaderboard rankings. For reference, the false positive rate (FPR) and false negative rate (FNR) are defined as  
$$
\mathrm{FPR} = \frac{\mathrm{FP}}{\mathrm{FP} + \mathrm{TN}}, \quad \mathrm{FNR} = \frac{\mathrm{FN}}{\mathrm{FN} + \mathrm{TP}}
$$

## 4. Baselines, System Designs, and Results

On the development set, baseline detectors included logistic regression over TF-IDF features (80.81% accuracy) and a BERT-based classifier (79.03%). Two systems outperformed all others:

- **sastsy (GigaCheck):** Mistral-7B backbone with a dual-head LoRA-fine-tuned architecture, achieving 91.22% accuracy.
- **adugeen:** Fine-tuned YandexGPT 8B, integrating neural features with bag-of-words statistics, at 86.96% accuracy.

In the blind test phase with unseen economics and biology domains and Deepseek-V3 data, sastsy again led (86.35%), followed by adugeen (84.62%). Both systems demonstrated that lightweight LLM fine-tuning with a multiclass (multi-task) head—first separating human/AI, then identifying LLM provenance—yielded robust cross-domain and cross-model generalization. Augmenting neural architectures with statistical text features (average sentence length, digit frequency ratio) was shown to further enhance detection on novel data.

## 5. Generalization Challenges: Cross-Domain and Cross-Model

A distinguishing feature of the shared task was the explicit evaluation of generalization. Unseen scientific domains in the test set (Economics and Biology) and outputs from Deepseek-V3 (previously unannounced to participants) compelled system designs that transcend domain- or model-specific overfitting. Multi-task heads were shown to increase sensitivity to nuanced style differences across LLM outputs, while feature engineering (especially rudimentary statistics) improved robustness.

*This suggests that hybrid neural-statistical models provide a practical pathway to mitigated overfitting in LLM-generated text detection across both domain and generator boundaries.*

## 6. Continuous Benchmarking Platform and Availability

The organizers have established a continuous shared task platform, preserving the private test partition for ongoing blind evaluation. This "living benchmark" permits researchers to submit system predictions on held-out data post-competition, supporting continuous progress measurement as detection methods evolve. All data, code (including pre-/post-processing scripts), prompts, generation scripts, and the CodaLab competition configuration are released under an open license at [https://github.com/iis-research-team/AINL-Eval-2025](https://github.com/iis-research-team/AINL-Eval-2025).

Tabular resources at this repository include full dataset downloads and example submission notebooks, enabling rapid prototyping and reproducibility for new entrants and reviewers alike.

## 7. Significance and Prospects

The AINL-Eval 2025 Shared Task provides an unprecedented, rigorously constructed benchmark for AI-generated scientific text detection in Russian, explicitly addressing both cross-domain and cross-model transfer. The initiative's impact is evident in the demonstrated generalization of system architectures, the baseline-defining public dataset, and the platform's open, extensible design. A plausible implication is that analogous multi-stage and hybrid approaches could benefit detection efforts in other under-resourced languages or novel academic genres, especially as LLMs become ubiquitous authorship agents in scholarly communication [2508.09622].

Source: https://www.emergentmind.com/topics/ainl-eval-2025-shared-task