EXIST: Sexism Detection in Social Networks

• Sexism Identification in Social Networks is a benchmark framework for detecting and characterizing sexist content in posts and memes with bilingual and cross-platform data.
• The task framework includes binary classification, source-intention analysis, and fine-grained categorization to capture both overt and subtle sexist cues.
• It employs disagreement-aware evaluation and multimodal modeling, enhancing robustness and interpretability in automatic sexism detection.

Sexism Identification in Social Networks (EXIST) is a shared-task and benchmark line for automatic detection and characterization of sexist content in social media. It began as the first shared task on sEXism Identification in Social neTworks at IberLEF 2021 and later continued in CLEF settings that emphasized bilingual English–Spanish modeling, richer task hierarchies, and, from 2023 onward, explicit treatment of annotator subjectivity under learning with disagreements. Across its editions, EXIST has framed sexism as binary identification, source-intention analysis, and fine-grained categorization over social-media texts and memes, while the surrounding literature has expanded the paradigm toward interpretability, multimodality, and demographic perspective modeling (Schütz et al., 2021, Paula et al., 2023, Labadie-Tamayo et al., 28 Jul 2025).

1. Benchmark scope and task evolution

In EXIST 2021, sexism identification was organized as a coarse-grained binary task and a fine-grained classification task. The binary task predicted sexist versus non-sexist content, while the fine-grained task used the labels ideological-inequality, objectification, stereotyping-dominance, misogyny-non-sexual-violence, sexual-violence, and non-sexist over English and Spanish posts from Twitter and Gab (Schütz et al., 2021). This formulation established the core benchmark intuition that sexism detection in social networks is not exhausted by a yes/no moderation decision and that category structure matters.

By EXIST 2023, the benchmark had been reformulated around three tasks under a disagreement-aware regime. Task 1, Sexism Identification, was a binary classification problem over sexist versus non-sexist tweets. Task 2, Source Intention, used the labels Direct, Reported, and Judgemental. Task 3, Sexism Categorization, used the labels Ideological and Inequality, Stereotyping and Dominance, Objectification, Sexual Violence, and Misogyny and Non-sexual Violence. The 2023 framing is technically important because it treated disagreement not as annotation noise to be collapsed away, but as a target for learning and evaluation (Paula et al., 2023).

EXIST 2024 retained the bilingual English–Spanish setting and, in the system paper available here, was addressed through Task 1, sexism identification, and Task 2, source intention. EXIST 2025 Task 1 was explicitly hierarchical, with Subtask 1.1 “Sexism Identification in Tweets,” Subtask 1.2 “Source Intention in Tweets,” and Subtask 1.3 “Sexism Categorization in Tweets,” again over English and Spanish (Azadi et al., 2024, Tian et al., 15 Jul 2025).

A recurrent boundary condition in the literature is that not every sexism-related NLP task is an EXIST task. “A Holistic Indicator of Polarization to Measure Online Sexism” is adjacent to EXIST, but its unit of final analysis is the online community, operationalized as a subreddit corpus, rather than the individual post. Its output is a community-level targeted-toxicity index, not an instance-level sexism label (Ghafouri et al., 2024).

2. Data, annotation, and disagreement-aware supervision

The 2021 EXIST dataset contained 6,977 training instances and 4,368 test instances, with 3,426 English and 3,541 Spanish posts in training, and 2,208 English and 2,160 Spanish posts in test. Training data came from Twitter, while test data included Twitter and Gab, which introduced domain shift at evaluation time (Schütz et al., 2021, Davies et al., 2021). This bilingual and cross-platform design made transfer, robustness, and language-specific modeling central from the outset.

The 2023 and 2024 benchmark line introduced a more explicit annotation structure. The EXIST 2023 dataset contained more than 10,000 tweets in English and Spanish, split into 6,920 training tweets, 1,038 development tweets, and 2,076 test tweets. For Task 1, each tweet had a set of labels in labels_task1, each being YES or NO. For Task 2, labels_task2 contained DIRECT, REPORTED, JUDGEMENTAL, --, and UNKNOWN. For Task 3, labels_task3 contained IDEOLOGICAL-INEQUALITY, STEREOTYPING-DOMINANCE, OBJECTIFICATION, SEXUAL-VIOLENCE, MISOGYNY-NON-SEXUAL-VIOLENCE, --, and UNKNOWN. The label -- marked tweets deemed non-sexist for Tasks 2 and 3, while UNKNOWN marked cases where annotators did not provide a label (Paula et al., 2023).

The EXIST 2024 dataset, as described in the CLEF 2024 notebook, also contained more than 10,000 tweets in English and Spanish with the same 6,920/1,038/2,076 split. Each tweet was annotated by six crowdsourced annotators recruited through Prolific, following guidelines from gender experts, and the released data included demographic metadata such as gender, age, country, study level, and ethnicity. The benchmark was explicitly presented within the learning with disagreements paradigm, although not all participant systems trained directly on the disagreement distributions (Azadi et al., 2024).

This distinction became methodologically salient. AI-UPV’s EXIST 2023 system trained directly from the data with disagreements, without using any aggregated label, by converting multiple annotations into soft labels. By contrast, the 2024 BAZI system used majority voting to determine a final label for training, even though the challenge itself supported disagreement-aware evaluation (Paula et al., 2023, Azadi et al., 2024).

The 2025 textual dataset continued this line. It contained over 10,000 X/Twitter posts, balanced between English and Spanish, split into 6,920 training posts, 1,038 development posts, and 2,076 test posts, with each post annotated by six individuals and accompanied by demographic metadata including gender, age group, ethnicity, education level, and country (Labadie-Tamayo et al., 28 Jul 2025).

3. Modeling paradigms

The first EXIST systems were dominated by multilingual transformer adaptation, language-specific encoders, and ensembles. AIT_FHSTP used multilingual BERT and XLM-R, comparing unsupervised pre-training on EXIST, HatEval2019, and MeTwo against supervised fine-tuning with augmented data. Their best system, XLM-R-PreT-EHM, performed unsupervised masked-language-model pre-training before classification fine-tuning and reached macro F1 0.7752 on the binary task and 0.5589 on the multiclass task (Schütz et al., 2021). AI-UPV’s 2021 winning system combined multilingual BERT, English BERT, and BETO with translation and ensemble strategies, and found that an ensemble over all individual models using standardized prediction values was strongest for both tasks (Paula et al., 2021).

Other 2021 systems reinforced the same pattern. MiniTrue used three BERT-based models and a hard majority-voting rule for Subtask 1, obtaining better accuracy and F1 than any of its individual components (Feng, 2021). codec.ai trained language-specific transformer models for English and Spanish and used majority voting across differently initialized models for binary detection, while keeping a hierarchical Task 1→Task 2 pipeline for categorization (Davies et al., 2021). A separate English-only study on EXIST 2021 found that BERT with a multi-filter CNN head was the strongest among the tested deep models, and that data augmentation helped the multiclass task more than the binary task (Kalra et al., 2021).

The 2023 benchmark shifted the modeling center of gravity toward disagreement-aware supervision. AI-UPV used mBERT and XLM-RoBERTa in three pipelines: a model-selection run, an ensemble run averaging predicted probabilities, and an adjusted ensemble that projected predictions onto feasible annotator-induced distributions using cosine similarity. This system was trained on soft labels rather than majority-vote labels and targeted both soft and hard evaluation (Paula et al., 2023).

The 2024 notebooks broadened the comparison to instruction-following LLMs. BAZI fine-tuned XLM-RoBERTa and separately used GPT-3.5 with few-shot prompts built from six bilingual demonstrations that included annotator votes. XLM-RoBERTa outperformed GPT-3.5 on both Task 1 and Task 2, indicating that supervised adaptation remained stronger than few-shot prompting for bilingual sexism identification and source-intention prediction under benchmark conditions (Azadi et al., 2024).

By 2025, two distinct research directions had become prominent. Mario used hierarchical LoRA and 4-bit QLoRA on Llama 3.1 8B, with conditional adapter routing across Subtasks 1.1, 1.2, and 1.3 and LoRA applied to all linear transformations rather than only attention projections (Tian et al., 15 Jul 2025). FHSTP introduced interpretable concept bottlenecks through SCBM and SCBMT, using adjective-based human-interpretable concepts scored by an LLM, and then fusing them with XLM-RoBERTa-large representations in SCBMT (Labadie-Tamayo et al., 28 Jul 2025). In parallel, related benchmark work outside EXIST but highly relevant to it showed that label-conditioned pairwise classification with natural-language label descriptions could substantially improve fine-grained sexism categorization, as in CL-UZH’s EDOS 2023 system based on DeBERTa-V3-large and incremental fine-tuning (Goldzycher, 2023).

4. Evaluation regimes and representative performance

EXIST evaluation evolved from conventional accuracy and macro-F1 toward disagreement-aware metrics. In 2021, systems were primarily compared with accuracy and macro-F1. AIT_FHSTP’s XLM-R-PreT-EHM ranked 5th in task 1 with macro F1 0.7752 and 6th in task 2 with macro F1 0.5589 (Schütz et al., 2021). AI-UPV’s 2021 paper states that the system ranked first in both tasks; it also reports test-table values of 0.789 accuracy and 0.785 F1-binary for task 1, and 0.703 accuracy and 0.689 F1-macro for task 2, while noting abstract/table discrepancies (Paula et al., 2021).

In EXIST 2023, soft evaluation compared predicted probability distributions against human annotation distributions, with ICM-Soft as the official metric and Cross Entropy additionally reported. Hard evaluation compared hard predictions against threshold-derived hard labels, with ICM, reported as ICM-Hard, and F1 Score. AI-UPV’s ensemble approach was best under soft evaluation for Task 1 and Task 2, and the adjusted ensemble ranked first globally in Task 3 soft evaluation with ICM-Soft = -2.3183 and ICM-Soft Norm = 0.7879 (Paula et al., 2023).

The 2024 BAZI system reported both hard-hard and soft-soft evaluations under the official ICM-based framework. For Task 1, its XLM-RoBERTa run obtained ICM-Soft = 0.82, ICM-Soft Norm = 0.63, Cross Entropy = 0.98, ICM-Hard = 0.55, ICM-Hard Norm = 0.78, and F1 = 0.78, and ranked 4th in soft-soft evaluation. For Task 2, it achieved ICM-Soft = -1.35, ICM-Soft Norm = 0.39, Cross Entropy = 1.78, ICM-Hard = 0.19, ICM-Hard Norm = 0.56, and F1 = 0.48, ranking 2nd in soft-soft evaluation (Azadi et al., 2024).

The 2025 Mario system reported official EXIST 2025 test-set ICM-Hard values of 0.6774 for binary sexism detection, 0.4991 for source intention detection, and 0.6519 for multilabel sexism categorization, ranking first in all three subtasks. The same paper reported that joint bilingual training over English and Spanish improved validation F1 across all three subtasks relative to separate-language training (Tian et al., 15 Jul 2025). FHSTP’s SCBMT did not dominate binary classification, where fine-tuned XLM-RoBERTa remained strongest, but it was more competitive for Source Intention and Sexism Categorization, illustrating that interpretability-oriented concept representations can be especially effective on nuanced tasks (Labadie-Tamayo et al., 28 Jul 2025).

5. Conceptual boundaries and adjacent formulations

A central conceptual boundary in the literature is between instance-level sexism identification and broader measurements of gender-targeted hostility. “A Holistic Indicator of Polarization to Measure Online Sexism” builds a community-level targeted-toxicity index by combining a supervised toxicity detector with unsupervised word-embedding association, and it explicitly cautions that the result should not be projected back onto comments or users. Its relevance to EXIST is therefore methodological rather than task-equivalent: it suggests ways to separate toxicity detection from target inference and to reduce target-specific annotation needs, but it is not a direct post-level sexism detector (Ghafouri et al., 2024).

Other adjacent systems are closer to the binary end of the EXIST spectrum but remain simpler than the shared-task taxonomy. The “Anti-Sexism Alert System” builds a Spanish comment-level classifier over sexist versus non-sexist comments and aggregates predictions into a traffic-light warning for news, Twitter, and YouTube threads. It is highly relevant to Spanish sexism detection, but it does not model source intention, fine-grained sexism categories, or disagreement (Redondo et al., 2023).

The multimodal frontier extends beyond memes. MuSeD introduced a multimodal Spanish dataset for sexism detection in social media videos from TikTok and BitChute, with annotations at text, audio, and video levels, and found that visual information plays a key role in labeling sexist content for both humans and models (Grazia et al., 15 Apr 2025). FineMuSe extended this into a fine-grained, hierarchical, and multi-label Spanish video dataset that included sexist subcategories, non-sexist subcategories such as counter-speech and reported sexism, and rhetorical devices such as irony and humor (Grazia et al., 17 Feb 2026). On EXIST memes specifically, MemeWeaver proposed inter-meme graph reasoning and reported strong gains on a custom internal split of the meme task, arguing that reasoning across related memes is useful for multimodal sexism detection (Italiani et al., 13 Jan 2026).

Related shared tasks outside EXIST reinforce similar design pressures. EDOS 2023 formulated sexism detection, coarse categorization, and fine-grained categorization over English social-media posts and showed that multi-task learning with label descriptions improved fine-grained sexism modeling substantially (Goldzycher, 2023). A plausible implication is that EXIST and EDOS occupy neighboring positions in a broader benchmark ecology: both treat sexism as more than binary offense detection, but they emphasize different taxonomies, media types, and evaluation regimes.

6. Persistent challenges and research directions

A major critique of sexism detection benchmarks is that construct validity is difficult to maintain. “Call me sexist, but...” grounded sexism in psychological scales, derived a codebook separating sexist content from sexist phrasing, and showed that many existing datasets cover only narrow aspects of sexism. Its adversarial experiments indicated that current machine learning models pick up on a very narrow set of linguistic markers of sexism and do not generalize well to out-of-domain examples, while diverse data and adversarial examples improve robustness (Samory et al., 2020). This critique applies directly to EXIST-like evaluation: high in-domain scores do not guarantee sensitivity to subtle, civil, or ideologically framed sexism.

The difficulty is visible in benchmark error analyses. In an English-only EXIST 2021 study, sexual-violence was the best recognized sexist category at 54%, while ideological-inequality and stereotyping-dominance were frequently collapsed into non-sexist, showing that explicit abuse remains easier than subtle stereotype or inequality discourse (Kalra et al., 2021). The same asymmetry reappears in FineMuSe, where multimodal LLMs were competitive with human annotators on nuanced forms of sexism but struggled to capture co-occurring sexist types when these were conveyed through visual cues, especially objectification (Grazia et al., 17 Feb 2026).

A second persistent challenge is perspectivism. Studies using EXIST 2023 and EXIST 2024 demographic annotations found that LLMs do not reliably reproduce the diversity of human perceptions across groups. One paper using Krippendorff’s alpha showed that all tested LLMs aligned more closely with female annotators than with male annotators, while persona-style demographic prompting did not reliably induce stable demographic viewpoints (Paula et al., 17 May 2025). A later study on the EXIST 2024 tweet dataset reported that LLMs can moderately approximate the aggregate perception of sexism but do not accurately replicate the diversity of perceptions among different demographic groups, with age appearing more informative than gender for correlation with subgroup labels (Tavarez-Rodríguez et al., 25 Aug 2025). This suggests that disagreement-aware sexism detection is not reducible to prompting a generic LLM with a demographic persona.

Taken together, these works suggest a research trajectory for EXIST beyond simple binary classification. The likely direction is toward multilingual and multimodal architectures, finer-grained and possibly multi-label taxonomies, explicit modeling of annotator disagreement, and interpretable outputs that distinguish sexist content, sexist phrasing, counter-speech, reported sexism, and rhetorical framing. The benchmark history already contains all of these ingredients in partial form; the open problem is integrating them without collapsing subtle sexism into generic toxicity or reducing perspective-sensitive judgment to a single canonical label.