---
title: SMM4H 2025 Shared Task 3
url: https://www.emergentmind.com/topics/smm4h-2025-shared-task-3
type: topic
---

# SMM4H 2025 Shared Task 3

Searching arXiv for the cited SMM4H papers and closely related Task 3 material.
SMM4H 2025 Shared Task 3 designates the third shared-task track in the Social Media Mining for Health workshop series, but its exact 2025 definition is not directly described in the cited literature. The available arXiv system papers instead document earlier, year-specific Task 3 formulations: in 2023, Task 3 was a Named Entity Recognition task for detecting and extracting COVID-19 symptoms from tweets written in Latin American Spanish; in 2024, Task 3 was a 4-class classification task over Reddit posts concerning the impact of outdoor environments on symptoms of social anxiety [2312.10652] [2404.19714] [2410.15998]. This establishes a central fact about the SMM4H series: the label “Task 3” is not semantically fixed across years.

## 1. Series context and the instability of task numbering

Within SMM4H, task numbering is reused across yearly shared-task editions rather than reserved for a single enduring benchmark. The 2023 system paper by the Explorers team explicitly situates its work in “The Eighth Social Media Mining for Health Applications (#SMM4H) Workshop and Shared Task,” where the team participated in Task 1, Task 3, and Task 4 and achieved first place in Task 3 [2312.10652]. The 2024 papers by ThangDLU and 1024m both treat Task 3 as a Reddit classification problem about outdoor spaces and mental health rather than an extraction problem [2404.19714] [2410.15998].

Earlier SMM4H literature reinforces this year-specific interpretation. In the 2017 workshop, Task 3 was “mapping a free-text ADR to a standardized MEDDRA term,” a task not addressed by NRC-Canada in its paper on Tasks 1 and 2 [1805.04558]. A plausible implication is that any discussion of SMM4H 2025 Shared Task 3 must begin by resisting cross-year assumptions based solely on the task number.

## 2. Prior Task 3 formulations most relevant to 2025

The most direct precedent in the provided literature is SMM4H 2023 Task 3. In that setting, Task 3 was a Named Entity Recognition task focused on detecting and extracting COVID-19 symptoms from tweets written in Latin American Spanish, including both personal self-reports and third-party mentions of symptoms. The input consisted of tweets in Latin American Spanish and the output was extracted COVID-19 symptom entities. The paper indicates a single extraction target class, COVID-19 symptoms, and does not mention multiple symptom subtypes, assertion labels, normalization targets, or nested entity categories. The reported dataset sizes were 6021 for training, 1979 for validation, and 2150 for test [2312.10652].

SMM4H 2024 Task 3, by contrast, was a multiclass document classification task. ThangDLU describes it as classification of Reddit posts from the r/socialanxiety subreddit mentioning predetermined keywords related to outdoor spaces into one of four categories: positive effect, neutral / no effect, negative effect, and unrelated. The paper reports 3000 total annotated posts, with a 6:2:2 split of 1800 training, 600 validation, and 600 testing examples, and notes that organizers provided a test set with 1200 posts “to hide the real ones” [2404.19714]. The 1024m paper describes the same Task 3 label space numerically as 0 for unrelated to the task, 1 for had a positive impact, 2 for is neutral or had no effect, and 3 for had a negative effect over 3000 Reddit posts from r/socialanxiety [2410.15998].

These precedents imply two distinct Task 3 archetypes within SMM4H. One is span-level health mention extraction from short, noisy social media posts; the other is post-level multiclass inference over longer Reddit discourse. This suggests that any technically serious treatment of SMM4H 2025 Shared Task 3 must remain conditional until the year-specific task definition is known.

## 3. Data characteristics and annotation regimes

The 2023 extraction benchmark foregrounds canonical social-media NLP difficulties. The Explorers paper emphasizes informal expression, colloquial wording, misspellings, noise, ambiguity, data sparsity, usernames and emojis with limited direct semantic value, and hashtags that may still carry information. It also gives a concrete lexical-variation example: the model recognizes “gripe” but misses forms such as “gripes” and “gripa” [2312.10652]. The annotation target is symptom mention extraction, but the paper does not explicitly describe BIO/BILOU tagging, span boundary conventions, discontinuous spans, nested entities, or ontology normalization.

The 2024 Reddit classification task exhibits a different data regime. Reddit posts are longer and more discursive than tweets, and the ThangDLU system used `max_source_length = 768`, retaining the first 256 tokens and the last 512 tokens when posts exceeded that limit [2404.19714]. The 1024m paper reports RoPE scaling for texts longer than 2048 tokens in fine-tuned LLM experiments [2410.15998]. Neither 2024 paper provides detailed annotation guidelines beyond the class definitions, and both leave class balance largely unspecified in the visible text.

From a comparative perspective, the main invariant across these settings is not a common label space but the use of user-generated health-related discourse as primary evidence. A plausible implication is that transfer across SMM4H Task 3 editions depends less on task number and more on shared properties of noisy social-media language, limited supervision, and label ambiguity.

## 4. Modeling paradigms associated with Task 3

For the 2023 NER task, the Explorers system departs from conventional token-labeling formulations and adopts W2NER. The paper states that W2NER captures neighboring relations between entity words using Next-Neighboring-Word (NNW) and Tail-Head-Word (THW), and represents unified NER as a 2D grid of word pairs. For Task 3, the system used three Spanish BERT-family backbones—`bert-base-spanish-wwm-cased-xnli`, BETO, and BETO_NER—together with continual pre-training and Multi-Model Fusion Voting, with final prediction obtained through “Mean-Pooling” [2312.10652]. The paper does not provide W2NER equations, decoder specifics, relation-label inventory, or token-to-word alignment details.

The 2024 classification literature explores two different paradigms. ThangDLU frames classification as sequence-to-sequence generation using BART-base, reporting that training on raw text without preprocessing worked better than training with preprocessing. The model selection criterion was validation F1-macro, and the reported Task 3 hyperparameters were 10 epochs, batch size 4, and `max_source_length = 768` [2404.19714].

The 1024m paper compares fine-tuned Transformer classifiers, zero-shot proprietary LLMs, and fine-tuned open-source LLMs. For Task 3, the strongest result came from a two-stage BART-large classifier that first separates unrelated posts from related ones and then classifies related posts as positive, neutral, or negative. The paper also reports exact prompt formulations for direct and staged LLM inference, constraining output to a single character in `0/1/2/3` or `0/1` and `1/2/3` formats [2410.15998].

Taken together, these papers associate Task 3 with two substantially different technical toolkits: relation-based span extraction for NER-style problems and staged or seq2seq classification for multiclass discourse-level problems.

## 5. Evaluation conventions and reported performance

The 2023 Task 3 extraction system reports test-set Precision \(= 0.94\), Recall \(= 0.93\), and F1 \(= 0.94\), with the explicit statement that its F1-score was the highest among all submissions for Task 3. The abstract repeats that the method achieved first place in Task 3 [2312.10652].

For SMM4H 2024 Task 3, the evaluation metric was macro-averaged F1-score across all categories in the ThangDLU paper. Its BART-base results were 0.595 F1-macro for training only, 0.601 for training plus paraphrased data, and 0.627 for training plus validation, with corresponding accuracies of 0.631, 0.640, and 0.670. The paper reports mean and median team scores of 0.518 and 0.579 macro-F1, respectively [2404.19714].

The 1024m paper provides both dev and test comparisons. On dev, BART-large improved from 0.603 Macro-F1 in direct 4-way classification to 0.670 in two-stage classification; the same directional pattern held across longformer-large, RoBERTa-large, and DeBERTa-large. On test, `Bart-Large* (2-stage)` trained without dev data reached F1 0.673, `Bart-Large (direct)` reached 0.654, and `Bart-Large (2-stage)` reached 0.679, compared with mean 0.519 and median 0.580 [2410.15998].

These results show that metric choice and benchmark structure are year-specific. The 2023 extraction setting uses precision, recall, and F1 on entity extraction, while the 2024 Reddit setting emphasizes macro-F1 for multiclass post classification.

## 6. Error patterns, limitations, and methodological cautions

The most explicit Task 3 error analysis in the provided literature concerns the 2023 symptom-extraction system. The authors state that they were unable to address all issues, such as misspellings and informal or colloquial expressions in Task 3, and illustrate the problem with “gripe,” “gripes,” and “gripa” [2312.10652]. The same paper also warns that continual pre-training is “quite unstable,” sometimes yielding positive impacts and sometimes negative influences, depending on the data used for pre-training.

The 2024 classification papers identify a different set of limitations. ThangDLU notes that seq2seq classification can produce out-of-scope labels, which they map to a default label “0,” and concludes that the relatively low F1 for Task 3 indicates that the task is difficult and that more training data is needed [2404.19714]. The 1024m paper reports that paraphrasing existing data did not improve results and that Transformer models were less effective when language was atypical or when criteria for classification appeared in one sentence and were referred to indirectly later on; by contrast, LLMs were described as performing equally well on all kinds of data, even though their aggregate Task 3 scores were lower than those of the best supervised Transformer system [2410.15998].

A recurring methodological caution across the literature is that absolute F1 values should not be compared across years unless dataset, platform, label space, split policy, and metric are aligned. This caution is explicit in the 2024 classification literature and is also strongly supported by the cross-year redefinition of Task 3 itself [2404.19714].

## 7. Implications for interpreting SMM4H 2025 Shared Task 3

Because none of the cited papers directly specifies SMM4H 2025 Task 3, the most rigorous interpretation is conditional. If the 2025 task is an NER-style symptom extraction problem from social media, the 2023 Explorers system is the closest direct precedent: lightweight social-media preprocessing, continual pre-training on in-domain text, W2NER-style span modeling, and multi-backbone fusion across Spanish encoders [2312.10652]. If the 2025 task is instead a small-data multiclass classification benchmark over user-generated posts, the 2024 literature offers two distinct baselines: BART-base seq2seq classification with minimal preprocessing, and BART-large with a two-stage decomposition that isolates the unrelated class before effect classification [2404.19714] [2410.15998].

Several transferable lessons remain stable across these otherwise different benchmarks. Social-media text remains informal, colloquial, noisy, ambiguous, and lexically variable. More labeled data can help, but validation-data reuse depends on shared-task rules. Zero-shot proprietary LLM prompting should not be assumed to dominate supervised baselines. Paraphrase augmentation is not consistently beneficial. Two-stage decomposition can be valuable when one class is qualitatively different from the others. Relation-based span extraction can outperform conventional token labeling in noisy NER settings [2312.10652] [2410.15998].

The literature therefore supports a restrained but technically clear characterization of SMM4H 2025 Shared Task 3: not as a single already-defined benchmark in the available papers, but as a year-specific SMM4H task whose most relevant precedents split between social-media health NER and multiclass mental-health-related post classification. The decisive first question for any 2025 system is consequently not which architecture is fashionable, but which prior Task 3 lineage the 2025 benchmark actually follows.

Source: https://www.emergentmind.com/topics/smm4h-2025-shared-task-3