SemEval 2025 Task 4: LLM Unlearning

• SemEval-2025 Task 4 is a benchmark for targeted unlearning in LLMs that requires models to forget designated documents without retraining from scratch.
• The task evaluates methods using regurgitation tests, knowledge recovery, and privacy leakage metrics to balance forgetting effectiveness with utility preservation.
• Subtasks cover synthetic creative texts, PII-laden biographies, and real Wikipedia data, highlighting challenges like scale effects and layer-specific optimization.

SemEval-2025 Task 4, titled “Unlearning sensitive content from LLMs”, is a shared task on machine unlearning for generative LLMs. It asks whether a model that has already memorized designated documents can be modified so that it forgets a specified forget set, retains a specified retain set, and preserves broad language ability, all without retraining from scratch. The benchmark instantiates this objective across three use cases—synthetic creative text, synthetic biographies containing personally identifiable information, and real biographies sampled from the model’s training distribution—and evaluates both direct regurgitation and knowledge recovery, together with privacy leakage and model utility (Ramakrishna et al., 2 Apr 2025).

1. Problem setting and rationale

SemEval-2025 Task 4 is motivated by the fact that LLMs can memorize training data and later regurgitate private, copyrighted, or otherwise sensitive text. The task framing emphasizes practical pressures such as privacy, legality, confidentiality, and GDPR-style “right to be forgotten” requirements. Full retraining is treated as too expensive, so the benchmark focuses on selective removal of already memorized content while preserving unrelated knowledge and downstream usefulness. The organizers also stress that unlearning in generative LLMs is harder than in conventional classification because the output space is open-ended and because evaluation must distinguish visible forgetting from hidden residual traces (Ramakrishna et al., 2 Apr 2025).

The official setting provides a pretrained target model together with a Forget set $F$ and a Retain set $R$. The desired post-unlearning behavior is that the model acts as if it had been trained only on retained data. Two public target models were used: OLMo-7B-0724-Instruct-hf and OLMo-1B-0724-hf. Both had already been fine-tuned to memorize the task documents, making the benchmark a test of post hoc removal rather than conventional fine-tuning (Ramakrishna et al., 2 Apr 2025).

A central conceptual distinction in the task is between forgetting effectiveness and utility preservation. The benchmark does not treat removal of the forget set as sufficient by itself; it also requires low privacy leakage and acceptable general performance. This design makes the task an instance of constrained model editing rather than unrestricted destructive retraining.

2. Benchmark design and dataset construction

The benchmark contains 1,387 unique documents and 4,394 unique examples derived from them. It is divided into three subtasks, each with paired forget and retain partitions (Ramakrishna et al., 2 Apr 2025).

Subtask	Content type	Size
1	Long-form synthetic creative documents	199 forget, 194 retain, 393 total
2	Short-form synthetic biographies with PII	203 forget, 202 retain, 405 total
3	Real biographies sampled from the target model’s training data	295 forget, 294 retain, 589 total

Subtask 1 models removal of copyrighted or otherwise sensitive creative text while avoiding ambiguity from naturally recurring web text. The organizers generated synthetic short novels with Mixtral 8x7B by sampling a genre from Action, Fantasy, Thriller, Comedy, Mystery, Science Fiction, Young Adult, and Romance; sampling 1 to 4 synthetic character names; sampling locations from a random address generator for most genres and a Dungeons & Dragons town generator for Fantasy; and prompting Mixtral to produce a 150–200 word short story. All stories were manually reviewed by two authors, and near-duplicates were filtered out (Ramakrishna et al., 2 Apr 2025).

Subtask 2 targets memorization of personally identifiable information. The organizers generated 500 synthetic personal biographies containing a random first and last name, a birthday sampled between 01/01/1964 and 01/01/1991, an SSN in the 900-xx-xxxx range, a 10-digit phone number, an email heuristically formatted as [email protected], and a non-existent home address produced by combining a street address from one state with a city and ZIP code from another. Mixtral 8x7B then produced biographies incorporating these fields (Ramakrishna et al., 2 Apr 2025).

Subtask 3 evaluates unlearning on real data from the target model’s own training distribution. The organizers sampled 750 biographies of length 100–200 words from the Wikipedia subset of Dolma v1.6, which was part of OLMo training. This makes the benchmark substantially more realistic than purely synthetic removal settings because the target information belongs to the pretraining distribution rather than an isolated synthetic corpus (Ramakrishna et al., 2 Apr 2025).

The benchmark was split into public and private subsets. Participants developed algorithms on public data, but official evaluation ran submitted Python unlearning scripts on hidden private forget and retain subsets. This design was intended to prevent trivial retraining-only solutions and to preserve the privacy of the hidden evaluation data (Ramakrishna et al., 2 Apr 2025).

3. Evaluation protocol and scoring

For each of the three subtasks, the benchmark defines two memorization tests: regurgitation tests and knowledge tests. This yields $3 \times 2 = 6$ sub-evaluations, and because both forget and retain behavior are measured, the organizers report 12 memorization metrics in total (Ramakrishna et al., 2 Apr 2025).

In regurgitation tests, the model receives a sentence-completion prompt formed by choosing a random position in the second half of a document and supplying the preceding sentences as context. The generated continuation is compared against the original continuation using ROUGE-L. In knowledge tests, the model answers a generated question about the document. For Tasks 1 and 3, question-answer pairs were produced with an agentic workflow in which Mixtral 8x7B generated an unambiguous question with a single concise answer, and three verifier LLMs—Claude 3 Sonnet, Titan Text Express, and Mixtral 8x7B—validated that the answer could be recovered from the full document. For Task 2, QA generation was template-based because the biographies contain explicit fields such as birth date, SSN, phone number, and address (Ramakrishna et al., 2 Apr 2025).

The task memorization component combines ROUGE-L for sentence completion and case-insensitive exact match for QA. Forget-set scores are inverted so that better forgetting yields higher benchmark scores. These 12 values are then aggregated with a harmonic mean to form the task aggregate. Privacy leakage is measured through a membership inference attack score: $$\text{MIA Score} = 1 - 2 \cdot \left| \text{mia\_auc} - 0.5 \right|.$$ This score is maximal when the attack behaves like random guessing, i.e., when $\text{mia\_auc}=0.5$. Utility is measured with MMLU, specifically 57 STEM subjects. For the 7B model, leaderboard eligibility required MMLU accuracy above

$0.371,$

which corresponds to 75\% of the pre-unlearning checkpoint’s accuracy; this threshold was not enforced for the 1B model because its baseline MMLU was already low. The final score is the arithmetic mean of the task aggregate, MIA score, and MMLU (Ramakrishna et al., 2 Apr 2025).

Official evaluation was executed on an AWS EC2 p4d.24xlarge node with 8 A100 40GB GPUs and DeepSpeed ZeRO. Runs were timed, overly slow submissions were discarded, and each team could submit up to 5 code files (Ramakrishna et al., 2 Apr 2025).

4. Baselines and dominant methodological families

The official overview defines four core unlearning baselines. Gradient Ascent (GA) uses the negated forget-set cross-entropy,

$-\mathcal{L}_{CE}(F; \theta),$

to push the model away from forget-set targets. Gradient Difference (GD) adds retain-set preservation,

$$-\mathcal{L}_{CE}(F; \theta) + \mathcal{L}_{CE}(R; \theta).$$

KL Regularization combines forget-set ascent with a retain-set KL term,

$$-\mathcal{L}_{CE}(F; \theta) + \mathcal{L}_{KL}(R; \theta, \theta_{ref}),$$

and Negative Preference Optimization (NPO) is included as a preference-based unlearning objective $\mathcal{L}_{NPO}(F; \theta)$ (Ramakrishna et al., 2 Apr 2025).

The competition overview reports that the baselines displayed a systematic scale effect: with the chosen hyperparameters, the 7B model tended toward over-unlearning, while the 1B model tended toward under-unlearning. This observation became one of the task’s main methodological lessons, since many later systems explicitly targeted this instability through parameter-efficient tuning, staged optimization, or layer selection (Ramakrishna et al., 2 Apr 2025).

Across submissions, several recurrent method families emerged. The organizers identify LoRA / parameter-efficient unlearning, selective layer or parameter unlearning, model merging, random-label or randomized-token approaches, task-vector subtraction, stabilized training and adaptive learning-rate control, novel loss reformulations, and distillation-based methods as the dominant technical patterns. A plausible implication is that the field had not converged on a single unlearning primitive; instead, SemEval-2025 Task 4 became a comparative testbed for several partially compatible design philosophies.

5. Leaderboard outcomes and representative systems

The task received over 100 submissions from 24 teams, representing nearly 70 individuals and 30+ institutions. The strongest overall system was AILS-NTUA, which ranked first on both model tracks. Their method, Sequential Unlearning with Gradient Difference (SUGD), combined gradient ascent on forget data, gradient descent on retain data, parameter-efficient adaptation, and a stabilizing data chunking strategy. On the private test set, they report 0.706 final score for the 7B model and 0.688 for the 1B model. The best 7B configuration used chunk size 32, forget-retain ratio 1:7, LoRA rank $R$0, LoRA alpha $R$1, learning rate $R$2, effective batch size 8, and 5 epochs per chunk in the final LoRA setup (Premptis et al., 4 Mar 2025).

ZJUKLAB ranked second among 26 teams with an online Task Aggregate of 0.944 and an Overall Aggregate of 0.487. Their central idea was to train two complementary LoRA-based unlearning models—one biased toward stronger forgetting and one toward stronger retention—and then merge them with TIES-Merging. Their local merged model analysis reported Task Aggregate 0.939, MIA AUC 0.501, MMLU Avg 0.480, and Aggregate 0.806, which they interpreted as evidence that model merging can mediate between over-forgetting and under-forgetting (Xu et al., 27 Mar 2025).

SHA256 proposed “Selective Amnesia -- Constrained Unlearning for LLMs via Knowledge Isolation” and reported 2nd place in the 1B model track. Their method first used causal tracing / causal mediation analysis on a 125-sample QA subset to localize subject–attribute associations, concluding that layers 0–5, especially MLP modules, are the main carriers of this memorization in OLMo 1B and 7B. They then froze upper layers and optimized only the early layers with a joint loss

$R$3

They report Final Score 0.652 for the 1B track and Final Score 0.711 for the 7B track, while also noting a substantial MMLU drop in the 7B setting (Agrawal et al., 17 Apr 2025).

Mr. Snuffleupagus ranked 4th on the official leaderboard of both 1B parameter and 7B parameter models with Adaptive Representation Misdirection Unlearning (Adaptive RMU). Their method pushed forget-example activations toward a random direction while anchoring retain examples to the frozen original model, and it adaptively scaled the control vector by the activation norm of the frozen model. Their layerwise study concluded that later decoder layers are best for sensitive factual content: 12,13,14 for 1B and 24,25,26 for 7B (Dosajh et al., 19 Jun 2025).

iShumei-Chinchunmei ranked 5th on the competition leaderboard with a balanced multi-task framework based on Effective Unlearning Loss (EUL),

$R$4

Their framework alternated retain-set supervised fine-tuning with forget-set EUL batches, and also explored negative response replacement and length-based data augmentation. On OLMo-1B, their best local configuration, RD + DA + EUL, reported MIA = 0.593, TAS = 0.395, MMLU = 0.275, and Final = 0.421 (Sun et al., 22 Jul 2025).

Other representative approaches broadened the methodological range. Atyaephyra combined NPO, retain loss, KL regularization, and LoRA, reporting validation results around 0.516 final score for their strongest setting and emphasizing the efficiency advantage of computing the KL term by ignoring LoRA updates in the reference branch (Bronec et al., 17 Mar 2025). Lacuna Inc. proposed LIBU, a two-phase pipeline combining influence-function-style inverse-Fisher weighting with Sophia second-order refinement; their strongest reported run achieved MIA 0.0 and MMLU 0.483, though overall aggregate scores remained below the top leaderboard systems (Kudelya et al., 4 Jun 2025). Cyber for AI studied global weight modification methods, including gradient ascent, gradient descent on the retain set, controlled ascent, KL-regularized ascent, and staged pipelines, obtaining test-set aggregate scores of 0.409 on 7B and 0.389 on 1B (P et al., 2 Mar 2025).

6. Main findings, controversies, and research significance

The official overview’s main conclusion is that unlearning is possible, but balancing forgetting and utility remains difficult. The strongest systems generally combined a forget objective with some form of retain anchoring, parameter efficiency, or structural restriction. The best methods were not those that maximized forgetting alone, but those that kept the model usable while suppressing forget-set memorization and reducing privacy leakage (Ramakrishna et al., 2 Apr 2025).

One major controversy concerns evaluation adequacy. The organizers emphasize that high task aggregate scores do not always imply good privacy, and they explicitly note that memorization metrics alone are insufficient. This is reinforced by system papers. ZJUKLAB argues that MIA scores and ROUGE-based metrics alone are insufficient to fully evaluate successful unlearning, because a model may obtain favorable benchmark values while still producing repetitive or degenerate outputs (Xu et al., 27 Mar 2025). This suggests that the benchmark measures an important but incomplete operationalization of unlearning.

A second recurring issue is where sensitive knowledge is stored. The leading system descriptions do not agree on a universal answer. SHA256 finds that early transformer layers, especially MLP modules in layers 0–5, are the main carriers of subject–attribute memorization in their causal tracing analysis (Agrawal et al., 17 Apr 2025). Mr. Snuffleupagus, by contrast, concludes that for factual and PII-like content the most effective unlearning occurs in later decoder layers, because those layers yield better robustness to membership inference attacks (Dosajh et al., 19 Jun 2025). This divergence is best interpreted not as a contradiction in the benchmark record, but as evidence that layer locality depends on both the unlearning mechanism and the knowledge type being targeted.

A third lesson is that model scale changes the optimization regime. Cyber for AI reports that smaller models can sometimes be handled effectively by direct retain-set training or controlled ascent, whereas larger models may require carefully staged procedures such as gradient difference pipelines (P et al., 2 Mar 2025). The organizers similarly report that baseline hyperparameters that over-unlearn the 7B model can under-unlearn the 1B model (Ramakrishna et al., 2 Apr 2025). A plausible implication is that selective unlearning does not yet admit scale-invariant default settings.

The benchmark’s broader significance lies in turning LLM unlearning into a reproducible shared-task problem with realistic constraints: hidden private data, code-based evaluation, explicit privacy scoring, and utility preservation. It establishes three durable points for later work. First, memorization, privacy leakage, and utility are distinct objectives and cannot be collapsed into one metric. Second, parameter-efficient and layer-selective methods are central because brute-force updates readily destabilize LLMs. Third, evaluation design matters: the benchmark’s use of private data and execution of submitted code on controlled infrastructure directly addresses shortcut solutions that would otherwise blur the distinction between genuine unlearning and retraining (Ramakrishna et al., 2 Apr 2025).

SemEval-2025 Task 4 therefore occupies an important place in the emerging literature on machine unlearning for LLMs. It does not resolve the problem, but it sharply defines it: successful unlearning is a constrained intervention on parametric memory that must remove targeted content, preserve retained knowledge, resist privacy attacks, and avoid unacceptable degradation in general reasoning.