NoisyAG-News Benchmark

• NoisyAG-News is a benchmark for instance-dependent label noise, leveraging human-annotated noisy labels from the AG-News dataset.
• It employs three aggregation strategies to generate noise levels from 10% to 38%, revealing limitations of synthetic noise simulations.
• Experiments with PLMs show that real instance-dependent noise leads to steeper performance drops compared to synthetic noise, highlighting challenges for noise-robust methods.

NoisyAG-News is a large-scale benchmark for studying instance-dependent label noise in text classification. Designed to address limitations of synthetic noise benchmarks—which fail to capture the complexity of real-world annotation errors—NoisyAG-News provides systematically annotated, human-generated noisy labels on the AG-News corpus. The benchmark exposes the distinctive challenges posed by instance-specific noise, enabling comprehensive evaluation of noise-robust learning methods under conditions that closely resemble practical text classification tasks (Huang et al., 2024).

1. Construction and Annotation Protocol

NoisyAG-News builds upon the AG-News dataset (Zhang et al., 2015), comprising news samples categorized into four classes: World, Sports, Business, and Sci/Tech. The original training set (50,000 samples, 12,500 per class) is reshuffled and partitioned into 45,000 training and 5,000 validation samples. The original 7,600 AG-News test samples serve as the held-out clean evaluation set.

Crowdsourcing is employed, with 60 non-expert annotators divided into three groups. Each group independently labels all 50,000 samples, yielding a triple annotation $\{y_n^{(1)}, y_n^{(2)}, y_n^{(3)}\}$ per instance. A pilot study on 4,000 samples yields Cohen’s $\kappa \approx 0.75$, Gwet’s Gamma $\approx 0.75$, and underperforming annotators are retrained. The AG-News ground-truth is designated as the clean label $y_n$.

Three distinct noisy training sets are produced via aggregation rules:

Set	Aggregation Rule	Accuracy	Noise Rate
NoisyAG-NewsBest	If any annotation matches $y_n$, adopt $y_n$; else random from three.	~90%	~10%
NoisyAG-NewsMed	Majority voting.	~80%	~20%
NoisyAG-NewsWorst	If all annotations differ from $y_n$, random from three; else $y_n$.	~62%	~38%

Class-wise noise rates (NoisyAG-NewsMed) derive from the estimated transition matrix $\bar{T}$ (see Figure 1d):

• World: $\bar{T}_{World\to World} \approx 0.869$ (noise $\kappa \approx 0.75$0 13.1%)
• Sports: $\kappa \approx 0.75$1 (noise $\kappa \approx 0.75$2 6.9%)
• Business: $\kappa \approx 0.75$3 (noise $\kappa \approx 0.75$4 29.7%)
• Sci/Tech: $\kappa \approx 0.75$5 (noise $\kappa \approx 0.75$6 35.5%)

The overall average noise rate for NoisyAG-NewsMed is $\kappa \approx 0.75$7.

2. Instance-Dependent Noise Characteristics

NoisyAG-News enables systematic analysis of instance-dependent noise (IDN), where the distribution of noisy labels is a function of both the original class and the individual input $\kappa \approx 0.75$8. The noise process is formalized via a sample-dependent transition matrix $\kappa \approx 0.75$9 with entries $\approx 0.75$0. Feature-driven label errors are confirmed through quantitative and qualitative analysis.

The per-class noise rate is given as: $\approx 0.75$1

Cluster-based diagnostics further stratify samples: the feature space is partitioned into $\approx 0.75$2 regions per class, and transition vectors $\approx 0.75$3 estimate $\approx 0.75$4. A two-sample $\approx 0.75$5-test of squared $\approx 0.75$6 distances between human-annotated and synthetic-noise clusters yields $\approx 0.75$7, rejecting the null hypothesis of feature-independence and empirically confirming instance dependence.

Key observations:

• Sci/Tech samples are most frequently mislabeled as World or Business.
• Business often confuses with World.
• Sports has highest self-transition and lowest noise rate.
• The noisy transition matrix (NTM) for human annotations features disorderly, semantically biased off-diagonal structure, divergent from the structured patterns of synthetic noise.

3. Synthetic Noise Counterparts

For controlled comparison, four synthetic noise constructions are applied to the clean training set:

• Uniform (symmetric) noise: $\approx 0.75$8, $\approx 0.75$9.
• Single-flip (“pairwise”) noise: $y_n$0, $y_n$1.
• Synthesized IDN (Xia et al., 2020): $y_n$2 computed by projecting $y_n$3 via a weight matrix $y_n$4; flips occur per feature similarity.
• Flip-by-NTM: class flips follow the empirical NoisyAG-NewsMed NTM.

Empirically, pre-trained LLMs (PLMs) show near invariance to even high-rate synthetic noise (accuracy drops ≤3% at $y_n$5). In contrast, real instance-dependent noise triggers steeper degradation: drops of ~3.5% at 10%, ~8.0% at 20%, and ~16.0% at 38%. Learning curves indicate that, under synthetic noise, PLMs initially focus on clean samples and only later overfit noise, making early stopping effective. This separation collapses in the presence of real IDN, with the models fitting noisy patterns early and generalizing inconsistently.

4. Experimental Framework and Baselines

Experiments evaluate five modern PLMs:

• BERT-base
• RoBERTa-base
• DeBERTa-v3
• BART-base
• XLNet-base

All models are optimized via AdamW (learning rate $y_n$6, batch size 32, maximum steps 20,000) with early stopping based on validation accuracy. Training is conducted on 8 × A6000 GPUs.

A suite of robust learning baselines is benchmarked, primarily on RoBERTa:

Method	Key Feature
WN	Warm-up (no noise handling)
CM	Noise adaptation layer (Jindal et al., 2019)
CMGT	Improved transition estimation (Yao et al., 2020)
CT	Co-Teaching (Han et al., 2018)
LS	Label smoothing (Zhang et al., 2021)
NLS	Noisy label smoothing (Wei et al., 2021)
BTLS	Bi-tempered logistic loss (Amid et al., 2019)

Primary evaluation metric is accuracy: $y_n$7 Macro-F$y_n$8 is considered for class-balanced analysis.

Empirical findings demonstrate resilience to synthetic noise across methods but pronounced vulnerability to IDN in NoisyAG-News. CMGT outperforms others on NoisyAG-NewsWorst but lags behind clean performance by approximately 7%. Both Co-Teaching and straightforward transition-matrix corrections fail under real IDN, demonstrating that feature-independence assumptions are invalid in this regime.

5. Open Challenges and Research Directions

Textual instance-dependent noise presents unique difficulties:

• Annotation errors are neither random nor purely class-conditional but respond intricately to lexical, topical, and syntactic variations.
• Pre-trained LLMs are prone to rapid memorization of instance-specific annotation errors, which undermines generalizability.
• Most noise models and loss correction strategies presume feature-independent or simple parametric label corruption, a presumption not met with human-generated noise.

Critical open problems highlighted include:

• Estimating high-dimensional, feature-conditioned transition matrices $y_n$9.
• Developing regularization or sample selection techniques that utilize language semantic and syntactic structure.
• Quantifying and leveraging annotator label consistency in the learning objective.
• Employing causal or contrastive frameworks to disentangle inherent content features from annotation noise.

NoisyAG-News supports:

• Controlled, variable noise-rate experimentation (10–38%).
• Direct comparisons of real and synthetic instance-dependent noise.
• Fine-grained, cluster-based diagnostics of model performance.

A plausible implication is that adapted methods—such as meta-learning of $y_n$0, contrastive self-supervision, and architectures integrating text features with per-sample annotation distributions—will be necessary to address text-based IDN phenomena.

6. Impact and Applications

NoisyAG-News establishes itself as a critical benchmark for the advancement of noise-robust text classification. By characterizing realistic, instance-dependent annotation noise at large scale, it provides a testbed for diagnosing the failure modes of current robust learning methods and fostering the development of text-specific algorithmic solutions. Researchers can leverage its multi-view, real-vs-synthetic, and per-sample diagnostics to develop and evaluate methods that generalize better in the face of naturally occurring annotation errors, addressing a fundamental deficiency in prior noisy-label NLP benchmarks (Huang et al., 2024).