SeLeRoSa: Romanian Satire Dataset

• SeLeRoSa is a dataset for sentence-level satire detection in Romanian news, enabling fine-grained identification of satirical content within factual texts.
• It comprises 13,873 manually annotated sentences across various domains, labeled by native speakers to capture nuances of irony, sarcasm, and satire.
• Baseline evaluations show that transformer models and large language models perform significantly better when fine-tuned, highlighting challenges in zero-shot satire detection.

Searching arXiv for the specified paper to ground the article. SeLeRoSa is a sentence-level Romanian satire detection dataset for news articles, introduced as the first resource of this type for Romanian and designed to support research on satire, irony, and sarcasm at a granularity below the document level (Smădu et al., 31 Aug 2025). The dataset targets a setting in which figurative content is embedded in otherwise news-like discourse, making satire detection relevant not only to humor analysis but also to systems that may confuse satirical writing with factual reporting. SeLeRoSa comprises 13,873 manually annotated sentences drawn from multiple domains and is accompanied by baseline evaluations spanning transformer models, open-source LLMs in zero-shot and fine-tuning regimes, and closed-source LLMs in zero-shot settings (Smădu et al., 31 Aug 2025).

1. Conceptual scope and problem formulation

SeLeRoSa is motivated by the observation that satire, irony, and sarcasm are techniques typically used to express humor and critique, rather than deceive; however, they can occasionally be mistaken for factual reporting, akin to fake news (Smădu et al., 31 Aug 2025). The resource therefore addresses a narrower and technically harder problem than document-level satire classification: identifying whether an individual sentence within a news context should be labeled as satirical or non-satirical.

The annotation framework defines satire, irony, and sarcasm as the use of figurative language, including irony, exaggeration, and sarcasm, that conveys an opposite or critical stance, often humorous (Smădu et al., 31 Aug 2025). This definition places the dataset within figurative-language processing rather than misinformation detection per se. A common misconception is to treat satire detection as identical to fake-news detection; the dataset’s framing explicitly distinguishes these tasks, while also acknowledging that confusion between them can occur in practice (Smădu et al., 31 Aug 2025).

The sentence-level formulation is consequential because satirical information can be incorporated into news articles at a more granular level than whole-document labels permit. This suggests that SeLeRoSa is intended not only for binary classification experiments but also for hybrid pipelines in which local sentence signals can be aggregated into broader assessments of discourse stance or credibility.

2. Corpus construction and statistical profile

The dataset contains 13,873 sentences in total and covers four broad domains: Social issues, IT, Science, and Movies (Smădu et al., 31 Aug 2025). The source material was obtained by scraping news texts, with a cut-off in February 2018, from Romanian satirical outlets and factual news websites (Smădu et al., 31 Aug 2025). The construction pipeline began with an initial pool of 10,806 article titles, which was manually filtered to 3,692 articles; the resulting texts were cleaned from HTML and split into approximately 36,000 sentences, from which the final 13,873 sentences were retained after annotation (Smădu et al., 31 Aug 2025).

The class distribution follows the natural distribution of the collected material rather than an artificially balanced design. Non-satirical, or “regular,” sentences number 8,179, representing approximately 59% of the corpus, while satirical sentences number 5,694, or approximately 41% (Smădu et al., 31 Aug 2025). No further balancing beyond natural distribution was applied.

The train, validation, and test splits were created by grouping sentences from the same article, which reduces leakage across partitions. The train split contains 9,800 sentences, divided into 5,705 regular and 4,095 satirical instances. The validation split contains 2,000 sentences, divided into 1,240 regular and 760 satirical instances. The test split contains 2,073 sentences, divided into 1,234 regular and 839 satirical instances (Smădu et al., 31 Aug 2025).

Sentence length is relatively compact, with a median length of approximately 19 words, a mean of approximately 20.1 words, and a standard deviation of approximately 10.1 (Smădu et al., 31 Aug 2025). The examples supplied with the dataset illustrate the intended contrast. A regular sentence is: “Ministerul Educației va introduce astăzi un nou plan de examene pentru facultăți.” A satirical sentence is: “Guvernul va tipări bancnote personalizate cu chipul câinelui premierului pentru a spori încrederea publică.” (Smădu et al., 31 Aug 2025)

3. Annotation protocol and human agreement

SeLeRoSa was annotated by six native Romanian-speaking annotators aged 20–25, of whom 2 were male and 4 were female (Smădu et al., 31 Aug 2025). Each sentence was labeled as “satirical,” “neutral,” or “uncertain,” and “uncertain” labels were discarded (Smădu et al., 31 Aug 2025). Final decisions were determined by majority vote, defined as at least 2 of 3 annotators (Smădu et al., 31 Aug 2025).

The reported inter-annotator agreement scores indicate that the task is non-trivial even for human annotators. Fleiss’ $\kappa = 0.4036$, Kendall’s $W = 0.4378$, and the average pairwise Cohen’s $\kappa = 0.4053$ (Smădu et al., 31 Aug 2025). The average raw agreement is 71.1%, while agreement with the majority vote, used as a proxy for a human upper bound, is 85.5% (Smădu et al., 31 Aug 2025). The paper characterizes this as moderate agreement, which is consistent with the figurative and context-sensitive nature of satire, irony, and sarcasm.

These agreement values are important for interpreting downstream model performance. They imply that sentence-level Romanian satire detection is not a near-deterministic labeling problem and that evaluation scores should be read against a human reference that is itself imperfect. A plausible implication is that future gains may depend not only on larger models but also on richer contextualization and clearer operationalizations of figurative phenomena.

4. Modeling baselines and experimental configuration

The baseline study evaluates both conventional transformer architectures and multiple LLM families (Smădu et al., 31 Aug 2025). The transformer-based fine-tuned baselines are Romanian BERT (bert-base-romanian-cased-v1) and RoGPT2-large. The open-source LLMs, evaluated in zero-shot and LoRA-fine-tuned settings, include the Gemma 3 family at 1B, 4B, 12B, and 27B parameters, as well as RoMistral 7B, RoLlama 3.1 8B, RoGemma 2 9B, and Llama 3.1 8B. The closed-source LLMs, evaluated in zero-shot settings only, are GPT-4o base, GPT-4o mini, GPT-4.1 nano, GPT-4.1 mini, GPT-4.1 base, and the o4-mini reasoning model (Smădu et al., 31 Aug 2025).

Zero-shot inference used default temperature $= 0.8$, except for Gemma 3 which required $1.0$, with top-$k = 64$, top-$p = 0.95$, and a maximum output length of 50 tokens; the exception was o4-mini reasoning, for which the maximum was 30,000 tokens (Smădu et al., 31 Aug 2025). For transformer fine-tuning, the configuration used AdamW with weight decay 0.01, learning rate $2 \times 10^{-3}$, 40 epochs, and batch size 64 (Smădu et al., 31 Aug 2025). For LLM adaptation, QLoRA was used with 4-bit quantization and LoRA, rank $R = 16$, $\alpha = 32$, dropout $W = 0.4378$0, maximal learning rate $W = 0.4378$1 with 10% warm-up, paged 8-bit AdamW, 3 epochs, batch size 32, and maximum input length 1024 (Smădu et al., 31 Aug 2025).

Evaluation employed accuracy and macro-averaged $W = 0.4378$2-score, with

$W = 0.4378$3

The use of macro-averaged $W = 0.4378$4 is notable because the class distribution is not balanced, and macro-averaging therefore prevents the majority regular class from dominating the summary score.

5. Empirical results and error structure

The reported test-set results show a marked gap between zero-shot performance and fine-tuned performance, and they also indicate that stronger general-purpose LLMs are not uniformly reliable at sentence-level satire detection without task adaptation (Smădu et al., 31 Aug 2025).

Model	Setting	Test accuracy / macro-$W = 0.4378$5
RoBERT-cased	Fine-tuned	76.58% / 70.82%
RoGPT2-large	Fine-tuned	70.92% / 65.77%
Gemma 3	Zero-shot	42.7–56.4% / 22.0–45.8%
Gemma 3	Fine-tuned	54.0–76.2% / 53.7–79.2%
RoMistral 7B	Fine-tuned	74.12% / 69.41%
Llama 3.1 8B	Fine-tuned	74.26% / 71.19%
RoGemma 2 9B	Fine-tuned	76.88% / 80.72%
GPT-4o mini	Zero-shot	61.54% / 64.81%
GPT-4.1 base	Zero-shot	65.22% / 67.60%
o4-mini	Zero-shot	72.53% / 72.74%

Among the reported baselines, RoGemma 2 9B fine-tuned with QLoRA achieves the best test performance at 76.88% accuracy and 80.72% macro-$W = 0.4378$6 (Smădu et al., 31 Aug 2025). The best closed-source zero-shot result is o4-mini at 72.53% accuracy and 72.74% macro-$W = 0.4378$7, which remains below the best fine-tuned open-source result (Smădu et al., 31 Aug 2025). This directly supports the paper’s central claim that current models, including recent LLMs, still exhibit limitations on the sentence-level satire detection task.

The error analysis identifies three recurrent failure modes: subtle irony or sarcasm without explicit lexical markers; domain-specific jargon, such as financial or medical language, that confuses satire cues; and longer or multi-clausal sentences in which contextual drift hides the punchline (Smădu et al., 31 Aug 2025). Topic-level observations show that rare topics such as “Sanctions and Legal Regulations” are easier, with $W = 0.4378$8, whereas broad topics such as “Financial Credit Unemployment” and “Books and Media” are harder, with $W = 0.4378$9–$\kappa = 0.4053$0 (Smădu et al., 31 Aug 2025).

The false-positive and false-negative behavior is particularly informative. Zero-shot models suffer from high false-positive rates, with FPR up to 52%, and tend to overpredict satirical labels; fine-tuning reduces FPR to below 20%, while false-negative rates remain below 15% (Smădu et al., 31 Aug 2025). This suggests that off-the-shelf LLM prompting may be biased toward interpreting stylistic markedness as satire, whereas task-specific adaptation substantially improves calibration.

6. Research significance, applications, and future directions

SeLeRoSa is publicly available under CC BY-NC-SA 4.0, including anonymized and pre-processed variants hosted at Hugging Face, with code and annotation details released through Zenodo (Smădu et al., 31 Aug 2025). The dataset is explicitly positioned as groundwork for advanced research on figurative language processing in Romanian and other low-resource languages (Smădu et al., 31 Aug 2025).

Several downstream and adjacent applications are identified. For deception and fake-news detection, sentence-level satire flags can be combined with document-level credibility assessment (Smădu et al., 31 Aug 2025). For humor and figurative-language analysis, the dataset supports joint modeling of sarcasm, irony, and hyperbole in low-resource languages (Smădu et al., 31 Aug 2025). For cross-lingual transfer and multilingual benchmarking, SeLeRoSa can serve as a target resource for cross-lingual annotation projection or adapter fine-tuning (Smădu et al., 31 Aug 2025). The study also proposes investigating robust prompting and in-context learning, including chain-of-thought or step-by-step reasoning prompts, to better capture subtle humor cues (Smădu et al., 31 Aug 2025).

The paper’s discussion of improvement strategies is methodologically specific. It suggests incorporating auxiliary signals such as sentiment scores and punctuation patterns, using hierarchical or document context to provide background for sentence cues, and applying multi-task learning with related figurative-language detection tasks (Smădu et al., 31 Aug 2025). These proposals indicate that the main bottleneck is not merely model scale, but the interaction between figurative semantics, discourse context, and domain-specific knowledge.

Within Romanian NLP, SeLeRoSa occupies a distinctive position because it provides sentence-level supervision rather than only article-level labels. More broadly, it functions as a stress test for LLM claims of robust zero-shot language understanding. The baseline evidence indicates that even high-capacity models remain vulnerable to overprediction, context loss, and domain confusion on this task, while fine-tuning yields substantial but still incomplete improvement (Smădu et al., 31 Aug 2025).