---
title: 'ArSarcasm-v2: Arabic Sarcasm Detection Dataset'
url: https://www.emergentmind.com/topics/arsarcasm-v2
type: topic
---

# ArSarcasm-v2: Arabic Sarcasm Detection Dataset

Searching arXiv for the specified papers to ground the article in the cited literature.
ArSarcasm-v2 is an Arabic Twitter dataset used for sarcasm detection and sentiment analysis in EACL WANLP-2021 Shared Task 2, and later for joint modeling of sentiment, sarcasm, and dialect. In the shared-task setting, it comprises 12,548 Arabic tweets for training/validation and a held-out 3,000 tweets for testing. In later work, the same split is treated as a 15,548-tweet resource with labels for sentiment, sarcasm, and dialect, which makes it useful both as a task-specific evaluation set and as a substrate for multi-task learning over correlated social-media phenomena [2103.01679], [2301.02521].

## 1. Shared-task role and corpus definition

ArSarcasm-v2 is the official training set for EACL WANLP-2021 Shared Task 2, subtask 1 for sarcasm detection and subtask 2 for sentiment analysis. The shared-task formulation distinguishes a binary sarcasm label and a three-way sentiment label. For sarcasm detection, the labels are **TRUE** for a sarcastic tweet and **FALSE** for a non-sarcastic tweet. For sentiment, the labels are **NEG**, **NEU**, and **POS**.

Later work uses the same data split as a 15,548-tweet collection consisting of 12,548 training examples and 3,000 test examples, and treats each tweet as carrying three labels: sentiment, sarcasm, and dialect. In that formulation, dialect takes one of five values: **MSA**, **EGY**, **Gulf**, **LEV**, or **NOR**. This makes ArSarcasm-v2 not merely a benchmark for isolated sarcasm detection, but also a dataset for studying dependencies among sentiment, sarcasm, and regional language variation [2103.01679], [2301.02521].

| Partition | Tweets | Use |
|---|---:|---|
| Training set | 12,548 | Training/validation in the shared-task split |
| Test set | 3,000 | Held-out shared-task test set |
| Total | 15,548 | Corpus size as used in later work |

## 2. Annotation scheme and label distributions

The annotation guidelines reported for the shared task specify that each tweet was examined by multiple annotators. A tweet is marked **TRUE** if at least two annotators agree that it conveys sarcasm; otherwise it is **FALSE**. Annotators were instructed to look for explicit markers of irony or incongruity between literal content and real-world facts or expectations. Sentiment annotations followed standard polarity guidelines: **NEG** for negative affect, **POS** for positive affect, and **NEU** when no clear polarity.

On the 12,548-tweet training set, the label counts reported in later work are 2,180 positive, 5,747 neutral, and 4,621 negative tweets for sentiment; 2,168 sarcastic and 10,380 non-sarcastic tweets for sarcasm; and 8,562 MSA, 2,675 EGY, 644 Gulf, 624 LEV, and 43 NOR tweets for dialect. The same study reports cross-task regularities: approximately 90% of sarcastic tweets are labeled **NEG**, approximately 50% of **MSA** tweets are labeled **NEU**, and approximately 50% of **EGY** tweets are labeled **NEG**. These correlations motivate architectures in which sentiment prediction is explicitly informed by sarcasm and dialect predictions [2103.01679], [2301.02521].

| Task | Label | Count |
|---|---|---:|
| Sentiment | POS | 2,180 |
| Sentiment | NEU | 5,747 |
| Sentiment | NEG | 4,621 |
| Sarcasm | Sarcastic | 2,168 |
| Sarcasm | Non-sarcastic | 10,380 |
| Dialect | MSA | 8,562 |
| Dialect | EGY | 2,675 |
| Dialect | Gulf | 644 |
| Dialect | LEV | 624 |
| Dialect | NOR | 43 |

For model development in the 2021 shared-task system, the 12,548 tweets were re-split into 90% train and 10% dev, yielding 11,293 training tweets and 1,255 development tweets. In that re-split, sarcasm counts were 9,356 **FALSE** and 1,937 **TRUE** in train, and 1,024 **FALSE** and 231 **TRUE** in dev. Sentiment counts were 4,139 **NEG**, 5,197 **NEU**, and 1,957 **POS** in train, and 482 **NEG**, 550 **NEU**, and 223 **POS** in dev [2103.01679].

## 3. Preprocessing and morphological segmentation

One reported treatment of ArSarcasm-v2 applies four successive transformations to each tweet before transformer finetuning. The first stage is **cleaning and noise removal**, which strips HTML line-break tags such as `<br>` and other markup, deletes non-standard emoticons or non-text unicode symbols, collapses character repeats beyond two, and normalizes extra whitespace. The stated purpose is to remove social-media artifacts that carry little or no signal for sarcasm or sentiment and to reduce token-vocabulary sparsity.

The second stage is **Farasa segmentation** for model variants that expect segmented input. Farasa segmentation splits clitics and affixes from stems; the summary gives the example \( \text{Al+kitābuhu} \rightarrow [\text{Al+}][\text{kitāb}][\text{+hū}] \). The segmentation scheme is based on the open-source Farasa segmenter and applies affix lookup, statistical stem isolation, and recoding of enclitics. It is explicitly described as morphological tokenization rather than character-level heuristics. The paper reports use of Farasa’s default settings and does not report custom thresholds or beam sizes. It also notes that Farasa internally relies on a Viterbi-style decoding to choose the highest-scoring segmentation path, although those equations are not restated.

The third stage is **whitespace insertion around mixed scripts**, with spaces inserted before and after English letters or digits adjacent to Arabic script, and around parentheses. The fourth stage is **placeholder replacement**, mapping URLs to `[<رابط>]`, emails to `[~<بريد>]`, and user mentions to `[<مستخدم>]`. These operations abstract away arbitrary strings while retaining their functional presence. A plausible implication is that the preprocessing is designed not only for normalization, but also for preserving tweet-level discourse cues that may remain relevant after lexical identity is removed [2103.01679].

## 4. Transformer-based baselines and segmentation effects

The 2021 shared-task system feeds each cleaned, and optionally segmented, tweet into a Hugging-Face Arabic BERT or ELECTRA tokenizer. The input sequence is wrapped as `[CLS] … tokens … [SEP]` and truncated or padded to `max_length = 256` tokens. The model’s original subword vocabulary and positional embeddings are used without modification, and no additional task-specific embeddings or adapters are injected. The experiments compare multiple variants of two transformer-based models, **AraELECTRA** and **AraBERT**.

For sarcasm detection, the reported validation results permit a direct comparison between pre-segmented and non-segmented runs. On subtask 1, **AraBERTv0.2-base** without segmentation reaches a weighted \(F_1\) of **84.62%**; **AraBERTv1-base** with segmentation reaches **84.86%**; and **AraBERTv2-base** with segmentation reaches **85.55%**. The summary states that moving from unsegmented AraBERTv0.2-base to segmented AraBERTv1-base yields approximately \(+0.24\) points in \(F_1\), while AraBERTv2-base reaches approximately \(+0.9\) points over the unsegmented variant. It further states that, while the absolute effect of Farasa segmentation is under 1 \(F_1\) point, it consistently improves model robustness, especially on the minority sarcastic class.

The final shared-task submission was ranked seventh in sarcasm detection and fourth in sentiment detection. On the official test set, the system obtained **58.72%** for subtask 1, where the official metric is \(F_1\) on the **TRUE** class, and **65.31%** for subtask 2, where the official metric is Macro-\(F_1\) on **POS/NEG** [2103.01679].

## 5. Multi-task exploitation of sarcasm, sentiment, and dialect

A later line of work treats ArSarcasm-v2 as a jointly labeled dataset and introduces **SAIDS**, a system that predicts sentiment, sarcasm, and dialect of Arabic tweets. SAIDS uses **MARBERTv2** as a language model to generate a sentence embedding, then passes that embedding to sarcasm and dialect models, concatenates their outputs with the original embedding, and feeds the result to a sentiment classifier. Tweets are tokenized by the MARBERTv2 tokenizer and truncated or padded to a maximum length of 128.

The architecture is specified as follows. A tokenized tweet is encoded by MARBERTv2 into a pooled sentence embedding of size \(D = 768\). Two first-layer classifiers then operate on that embedding: a sarcasm classifier mapping \(\mathbb{R}^{768} \rightarrow \mathbb{R}^{2}\) and a dialect classifier mapping \(\mathbb{R}^{768} \rightarrow \mathbb{R}^{5}\). Both are shallow multi-layer nets with zero hidden layers, namely a single linear layer followed by Softmax. The final sentiment classifier takes the concatenation of the 768-dimensional MARBERT embedding, the 2-dimensional Softmax output of the sarcasm model, and the 5-dimensional Softmax output of the dialect model, giving \(768 + 2 + 5 = 775\) input units, followed by one linear layer and a three-way Softmax.

The losses are binary cross-entropy for sarcasm, categorical cross-entropy for dialect, and categorical cross-entropy for sentiment:
$$
\mathcal{L}_{\mathrm{sar}} = -\frac{1}{N}\sum_{i=1}^N \bigl[y_i \log \hat{y}_i + (1-y_i)\log(1-\hat{y}_i)\bigr]
$$

$$
\mathcal{L}_{\mathrm{dial}} = -\frac{1}{N}\sum_{i=1}^N \sum_{c=1}^{5} y_{i,c}\log \hat{y}_{i,c}
$$

$$
\mathcal{L}_{\mathrm{sent}} = -\frac{1}{N}\sum_{i=1}^N \sum_{c\in\{\mathrm{POS,NEU,NEG}\}} y_{i,c}\log \hat{y}_{i,c}
$$

The total training objective is the sum of the three losses,
$$
\mathcal{L} = \mathcal{L}_{\mathrm{sent}} + \lambda_{\mathrm{sar}}\mathcal{L}_{\mathrm{sar}} + \lambda_{\mathrm{dial}}\mathcal{L}_{\mathrm{dial}},
$$
with \(\lambda_{\mathrm{sar}} = \lambda_{\mathrm{dial}} = 1\) in practice. Training uses Adam, a learning rate tuned in \(\{10^{-4},10^{-5},10^{-6}\}\), typically 5 epochs, up to 20 epochs for the smallest learning rate, batch size 32, and no additional data augmentation or dropout or weight decay. The best training sequence, denoted **Seq 1**, consists of one epoch training only the sarcasm and dialect heads with MARBERT frozen, followed by four epochs training all three heads and MARBERT together. The study also introduces **limited backprop**, in which errors from the sentiment loss are prevented from flowing into the final output layers of the sarcasm and dialect heads; the stated reason is to ensure that each head learns its own task robustly [2301.02521].

## 6. Evaluation protocols, results, and interpretive significance

The 2023 study defines the evaluation metrics explicitly. For a single class,
$$
\mathrm{Precision} = \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}, \quad
\mathrm{Recall} = \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}, \quad
F_1 = 2 \cdot \frac{\mathrm{Precision}\times\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}.
$$
For sentiment, the reported metric is **FPN**, the average of \(F_1\) over positive and negative classes:
$$
\mathrm{FPN} = \tfrac{1}{2}\bigl(F_1^{\mathrm{POS}} + F_1^{\mathrm{NEG}}\bigr).
$$
For sarcasm, the metric is **FSar**, the \(F_1\) score on the sarcastic class only. For dialect, the metric is **WFS**, the weighted average \(F_1\) over all five dialects, with weights proportional to class support.

On the shared 3,000-tweet test set, SAIDS reports **75.98** FPN for sentiment analysis, **59.09** FSar for sarcasm detection, and **71.13%** weighted \(F_1\) for dialect identification. In the same comparison table, the reported values are 71.60/58.41 for Baseline 1, 72.53/58.61 for Baseline 2, 73.11/58.62 for Baseline 3, 74.80/60.00 for el-mahdaouy et al. (2021), 73.92/61.27 for DeepBlueAI (song et al. 2021), 73.21/56.62 for Abdel-Salam (2021), and 72.55/58.72 for Wadhawan (2021). The study states that SAIDS outperforms the state-of-the-art model for the sentiment analysis task, while the highest listed sarcasm score in that table is DeepBlueAI’s 61.27 FSar.

Ablation studies on a random 10% validation split of the training data further characterize the dataset’s task dependencies. For sentiment prediction, the uninformed baseline yields **72.40%** FPN, informing with sarcasm only yields **73.67%**, informing with dialect only yields **74.41%**, and informing with both yields **75.23%**, a gain of **+2.83** percentage points over the uninformed baseline. Zero hidden layers in each head perform best at **75.23%**, compared with **74.90%** for one hidden layer and **74.89%** for two. Softmax on the heads yields **75.23%**, whereas raw scores yield **72.15%**. For backpropagation control, full limit gives **74.23%**, partial limit gives **74.89%**, and unlimited end-to-end training gives **72.31%**. For training sequence, all tasks together from epoch 1 gives **74.35%**, Seq 1 gives **75.23%**, Seq 2 gives **73.49%**, and Seq 3 gives **73.01%**. A proof-of-concept Random Forest experiment increases FPN from **59.36%** to **62.34%** when informed by sarcasm and dialect outputs. This suggests that ArSarcasm-v2 supports architectures that exploit label interdependence rather than treating sentiment as conditionally independent of sarcasm and dialect [2301.02521].

A common simplification is to describe ArSarcasm-v2 solely as a binary sarcasm corpus. The reported evidence does not support that reduction. In the shared-task setting it is simultaneously a sarcasm-detection and sentiment-analysis dataset, and in later work it is additionally used as a dialect-labeled resource. Another simplification is to treat preprocessing as incidental. The 2021 experiments report consistent, if modest, gains from Farasa-based segmentation, whereas the 2023 experiments show materially larger gains from explicitly modeling correlations among sarcasm, sentiment, and dialect. Taken together, these results position ArSarcasm-v2 as a dataset in which both Arabic-specific preprocessing and multi-task dependency structure are empirically consequential [2103.01679], [2301.02521].

Source: https://www.emergentmind.com/topics/arsarcasm-v2