AINL-Eval 2025: AI Abstracts Detection

• AINL-Eval 2025 is a benchmark for detecting AI-generated scientific abstracts in Russian across diverse domains and language models.
• The task employs a large-scale dataset with clear train, dev, and blind test splits, including unseen domains and a novel LLM for robust evaluation.
• Dual-phase evaluation and hybrid neural-statistical models demonstrate significant improvements in cross-domain and cross-model detection performance.

The AINL-Eval 2025 Shared Task is a benchmark and competitive evaluation initiative for the detection of AI-generated scientific abstracts in Russian. Motivated by the ascendancy of LLMs in generating publishable-quality text and by the acute shortage of robust detectors for non-English (specifically Russian) scholarly output, the task aims to challenge and advance the state of the art in multilingual, multi-domain AI-generated text detection. Incorporating a carefully constructed, large-scale dataset and a two-phase evaluation framework, the shared task foregrounds generalization to both new scientific domains and previously unseen text generators, thereby establishing a challenging, extensible benchmark for academic integrity in Russian-language publishing (Batura et al., 13 Aug 2025).

1. Motivation and Objectives

The proliferation of LLMs has rendered machine-generated scientific text—particularly abstracts—indistinguishable from human-written content at scale. This trend, coupled with limited availability of non-English detection resources, endangers peer review processes and undermines public trust in research outputs when undetected synthetic abstracts permeate the scientific record. The AINL-Eval 2025 Shared Task was thus established with two principal objectives:

1. To assemble a benchmark Russian-language dataset spanning multiple scientific domains and LLMs.
2. To foster models capable of generalizing beyond both seen scientific domains (cross-domain detection) and known AI text generators (cross-model detection) (Batura et al., 13 Aug 2025).

The challenge targets systemic vulnerabilities arising from the adoption of LLMs in non-English academic environments, with Russian scientific publishing selected as the representative testbed.

2. Dataset Construction and Characteristics

The AINL-Eval 2025 dataset comprises 52,305 Russian scientific abstracts, evenly split between human-written and LLM-generated texts. Human abstracts were collected from twelve established Russian journals representing diverse domains, including Mathematics, Philology, Law, and Oil & Gas. For each human abstract, five LLMs (GPT-4-Turbo, Gemma 2-27B, Llama 3.3-70B, Deepseek-V3, and GigaChat-Lite) were used to generate corresponding synthetic abstracts in a controlled setting.

Generation procedure standardized prompts as follows:

“Сгенерируй краткое содержание научной статьи по заголовку и ключевым словам. Напиши только текст аннотации. Не начинай текст аннотации с фразы ‘В данной статье’.” Заголовок: {title}. Ключевые слова: {keywords}.

Lightweight post-processing removed model-specific artifacts while preserving content. The dataset is partitioned into a 35,158-sample training set, a 10,978-sample development set, and a 6,169-sample blind test set. Critically, the test set incorporates two previously unseen scientific domains (Economics, Biology) and introduces outputs from Deepseek-V3, which was undisclosed to participants prior to testing, thereby explicitly encoding generalization benchmarks.

Subset	# Samples	Domains (Seen/Unseen)	LLMs Included
Train	35,158	12 (seen)	4 (seen)
Dev	10,978	12 (seen)	4 (seen)
Test	6,169	2 (unseen; Econ, Bio)	1 unseen (Deepseek-V3)

3. Evaluation Protocol and Metrics

The shared task was structured around two phases:

• Development Phase: Participants accessed train/dev splits for ~2 months, submitting unlimited system runs on CodaLab. Ground-truth labels were hidden for the development set. The leaderboard reflected accuracy on the dev split.
• Test Phase: A one-week blind evaluation on the 6,169-sample private test set, strictly limited to five submissions per team to minimize leaderboard overfitting.

The official metric was accuracy, defined as

$$\text{Accuracy} = \frac{TP + TN}{\text{Total samples}}$$

mirroring prior shared tasks (e.g., RuATD 2022). While related work frequently reports metrics such as precision, recall, F1

$$F_{1} = \frac{2 \cdot (\mathrm{precision} \cdot \mathrm{recall})}{\mathrm{precision} + \mathrm{recall}}$$

and ROC AUC, these did not impact leaderboard rankings. For reference, the false positive rate (FPR) and false negative rate (FNR) are defined as

$$\mathrm{FPR} = \frac{\mathrm{FP}}{\mathrm{FP} + \mathrm{TN}}, \quad \mathrm{FNR} = \frac{\mathrm{FN}}{\mathrm{FN} + \mathrm{TP}}$$

4. Baselines, System Designs, and Results

On the development set, baseline detectors included logistic regression over TF-IDF features (80.81% accuracy) and a BERT-based classifier (79.03%). Two systems outperformed all others:

• sastsy (GigaCheck): Mistral-7B backbone with a dual-head LoRA-fine-tuned architecture, achieving 91.22% accuracy.
• adugeen: Fine-tuned YandexGPT 8B, integrating neural features with bag-of-words statistics, at 86.96% accuracy.

In the blind test phase with unseen economics and biology domains and Deepseek-V3 data, sastsy again led (86.35%), followed by adugeen (84.62%). Both systems demonstrated that lightweight LLM fine-tuning with a multiclass (multi-task) head—first separating human/AI, then identifying LLM provenance—yielded robust cross-domain and cross-model generalization. Augmenting neural architectures with statistical text features (average sentence length, digit frequency ratio) was shown to further enhance detection on novel data.

5. Generalization Challenges: Cross-Domain and Cross-Model

A distinguishing feature of the shared task was the explicit evaluation of generalization. Unseen scientific domains in the test set (Economics and Biology) and outputs from Deepseek-V3 (previously unannounced to participants) compelled system designs that transcend domain- or model-specific overfitting. Multi-task heads were shown to increase sensitivity to nuanced style differences across LLM outputs, while feature engineering (especially rudimentary statistics) improved robustness.

This suggests that hybrid neural-statistical models provide a practical pathway to mitigated overfitting in LLM-generated text detection across both domain and generator boundaries.

6. Continuous Benchmarking Platform and Availability

The organizers have established a continuous shared task platform, preserving the private test partition for ongoing blind evaluation. This "living benchmark" permits researchers to submit system predictions on held-out data post-competition, supporting continuous progress measurement as detection methods evolve. All data, code (including pre-/post-processing scripts), prompts, generation scripts, and the CodaLab competition configuration are released under an open license at https://github.com/iis-research-team/AINL-Eval-2025.

Tabular resources at this repository include full dataset downloads and example submission notebooks, enabling rapid prototyping and reproducibility for new entrants and reviewers alike.

7. Significance and Prospects

The AINL-Eval 2025 Shared Task provides an unprecedented, rigorously constructed benchmark for AI-generated scientific text detection in Russian, explicitly addressing both cross-domain and cross-model transfer. The initiative's impact is evident in the demonstrated generalization of system architectures, the baseline-defining public dataset, and the platform's open, extensible design. A plausible implication is that analogous multi-stage and hybrid approaches could benefit detection efforts in other under-resourced languages or novel academic genres, especially as LLMs become ubiquitous authorship agents in scholarly communication (Batura et al., 13 Aug 2025).