AI Polish (APT-Eval) Dataset Overview

• AI Polish (APT-Eval) Dataset is a multi-task benchmark for evaluating classification and extraction models on Polish text across diverse domains.
• It aggregates seven distinct tasks from established Polish NLP datasets while preserving native label sets and minimal preprocessing.
• The benchmark supports robust few-shot and in-context learning with standardized prompt templates and detailed performance metrics.

The AI Polish (APT-Eval) Dataset is a curated, multi-task benchmark for evaluating classification and extraction models operating on Polish-language text. Developed for the assessment of both few-shot and in-context learning approaches with a focus on robust generalization, APT-Eval aggregates standardized tasks and domain-specific resources, including sentiment triplet extraction. The benchmark addresses linguistic challenges specific to Polish and serves as a reproducible testbed for language modeling, transfer learning, and evaluation of cross-lingual capabilities.

1. Composition and Covered Tasks

APT-Eval consists of aggregated resources from Polish NLP, representing seven distinct classification tasks selected from public benchmarks. These tasks are:

Code	Domain	Labels / Classes
PAC	Legal contract fairness (consumer law)	uczciwa, nieuczciwa
Polemo2	Online product reviews (sentiment)	pozytywny, negatywny, neutralny, ambiwalentny
CBD	Social-media cyberbullying detection	Tak, Nie
NKJP-NER	Named Entity Recognition (sentence-level)	geografia, organizacja, imię, miejsce, czas, brak
DYK	Wikipedia QA entailment	Tak, Nie
CDSC-E	Image-caption entailment	zgodne, sprzeczne, neutralne
CST-Wikinews	Wikinews sentence pair relations	12 discourse-oriented classes

Each of the original datasets is sourced from established Polish NLP resources (Lepiszcze, KLEJ, CLARIN-PL), and no further annotation was performed during curation. The sizes of the respective test splits range from 384 examples (CST-Wikinews) to 3,453 examples (PAC). The benchmark thus provides broad coverage for automated evaluation across domains such as law, sentiment analysis, social media moderation, entity recognition, QA, entailment, and discourse structure (Hadeliya et al., 2024).

2. Construction, Preprocessing, and Structure

APT-Eval was constructed by aggregation and minimal standardization, inheriting quality control and annotation guidelines from each source. Instead of unifying label semantics or annotation style, the collection maintains each task's native format and label set. Preprocessing is intentionally minimal: tasks requiring sentence pairs (e.g., entailment) employ simple concatenation, while spaCy tokenization and FNV hashing are used for XGBoost experiments. Outputs from LLMs are only lowercased and stripped of diacritics in post-processing. No new annotation was introduced, and task splits follow those of the base datasets (Hadeliya et al., 2024).

Each task is accompanied by:

• YAML-formatted prompt templates for in-context learning, consisting of a system directive, instruction, demonstration strings with placeholders, and label-to-word mappings; 71 such templates are provided.
• A standardized folder structure—each task has subfolders for data (.json splits), prompts (.yaml), and configurations (.json).
• Public access via HuggingFace Datasets (for data) and GitHub (for code, prompts, and evaluation scripts).

License inheritance ensures each base corpus retains its original license; the aggregation, templates, and scripts are released under Apache 2.0.

3. In-Context Learning Protocol and Templates

APT-Eval features a set of handcrafted, Polish-language templates for robust few-shot and in-context prompting. Each template block consists of:

• A classification-specific instruction (e.g., "Oceń czy następujący zapis jest uczciwy lub nieuczciwy.")
• Demonstration fields leveraging placeholders such as {text}, {sentence_1}, {sentence_2}, and {labels}
• Label mapping, explicitly encoding class IDs as Polish words

Every prompt is structured by concatenating a Polish meta-instruction with a selected instruction, followed by $n$ demonstrations, terminating with the test example. This enables systematic experimentation for prompt sensitivity and LLM adaptation in a few-shot context. The repository provides the complete set of YAML templates for reproducibility and custom experimentation (Hadeliya et al., 2024).

4. Evaluation Metrics and Performance Benchmarks

APT-Eval is designed for rigorous few-shot evaluation, with experiments reported using 16-shot settings and multiple random seeds. Key metrics include accuracy for multiclass and F1 for binary and NER tasks:

• In-context learning (ICL) using GPT-3.5 achieves mean accuracy $\bar{A}_{\text{ICL}} \approx 59.5\%$.
• SetFit scores $\bar{A}_{\text{SetFit}} \approx 47.1\%$; linear probing $\bar{A}_{\text{LP}} \approx 43.4\%$; and fine-tuning with nonlinear heads performed worst at $\bar{A}_{\text{FT}} \approx 31.2\%$.
• Full-data fine-tuned HerBERT-large sets the upper bound with $\bar{A}_{\text{full}} = 79.9\%$.
• Zero-shot GPT-4 yields $\bar{A}_{0\text{-shot}} = 65.9\%$.

The characteristic performance gap for Polish few-shot classification is quantified as $$\Delta = A_{\text{full}} - A_{0\text{-shot}} = 14.0\%$$, and between full-data HerBERT and 16-shot GPT-3.5: $\Delta_{\text{few-full}} = 20.4\%$. This highlights substantial headroom for few-shot generalization in Polish compared to full-resource fine-tuning (Hadeliya et al., 2024).

5. Domain-Specific Extensions: Polish-ASTE for Aspect–Sentiment Triplet Extraction

APT-Eval explicitly includes the Polish-ASTE datasets for domain-adapted sentiment extraction evaluation (Lango et al., 27 Feb 2025). These consist of:

• Polish-ASTE-hotels: 590 sentences, 1,197 annotated triplets from hotel reviews
• Polish-ASTE-products: 511 sentences, 851 annotated triplets from product reviews

Triplets comprise contiguous aspect spans, corresponding opinion spans (accounting for sentiment-bearing tokens and intensity modifiers), and sentiment polarity ({positive, negative, neutral}). Annotation follows strict guidelines—e.g., excluding opinion phrases not tied to explicit aspects, and preferring specific aspects over general item categories. Quality control involved expert review with less than 5% manual edits and an inter-annotator exact-match agreement of 78% for hotel samples. Data is distributed in JSON Lines, fully aligned with standard English ASTE corpora schema.

Benchmarking results demonstrate higher difficulty for Polish:

• The best F1 score (EPISA+TrelBERT) on products: 46.07%; hotels: 44.72%
• GTS+HerBERT lags at 35.65% (products), 30.02% (hotels)
• Polish results are 23 percentage points below English ASTE baselines, attributed to the complexity of Polish (multi-word opinions, balanced sentiment distribution)

These datasets are licensed under CC BY-SA 4.0 and intended both as standalone ASTE testbeds and as APT-Eval subcomponents for cross-domain and cross-lingual research (Lango et al., 27 Feb 2025).

6. Access, Licensing, and Extensibility

All data splits, templates, and evaluation scripts are openly accessible:

• Base datasets: HuggingFace Datasets platform, with explicit corpus names mapped to each task
• In-context prompt templates and code: GitHub repository at https://github.com/dkajtoch/LEPISZCZE
• Polish-ASTE: https://anonymous.4open.science/r/Polish-ASTE-Datasets-anonymous/

Licensing is unified: component datasets retain their public licenses (primarily CC-BY or MIT), while the APT-Eval aggregation, templates, and benchmark code are under Apache 2.0. The ASTE extension is CC BY-SA 4.0. No additional constraints apply.

The structure encourages extension to other domains (medicine, academic lectures) and additional Slavic languages, leveraging the same annotation and JSONL schema. It is further recommended to use the ASTE component for both single and multi-domain training, as well as for evaluating weak-supervision and transfer learning strategies to address the Polish–English performance gap (Lango et al., 27 Feb 2025).

7. Applications and Future Directions

APT-Eval provides a standardized, multi-task evaluation bed for Polish NLP research, supporting:

• Few-shot and in-context learning experiments for pre-trained LLMs, including robust prompt engineering
• Direct comparison of commercial (e.g., GPT-3.5, GPT-4) and open-source models (HerBERT, TrelBERT, Mistral-7B, Llama-2-13B, Bielik-7B, Trurl-13B)
• Assessment of continual pre-training benefits and transfer learning potential in under-resourced languages
• Domain and task transfer, particularly for sentiment triplet extraction and complex discourse analysis
• Cross-lingual experiments, via data merging with English corpora of matching schema

The benchmark highlights persistent gaps in Polish few-shot performance and motivates future research on model adaptation, weak supervision, and annotation protocol harmonization, especially for syntactically and morphologically rich languages (Hadeliya et al., 2024, Lango et al., 27 Feb 2025).