Few-NERD: Few-Shot NER Dataset

• The paper introduces Few-NERD, a large-scale, human-annotated dataset for few-shot NER with hierarchically-structured entity types.
• It employs a rigorous annotation protocol over 188,238 sentences using dense, fine-grained labels to ensure high-quality NER evaluation.
• Empirical results across SUP, INTRA, and INTER settings reveal challenges in span detection and generalization, motivating advances in meta-learning.

Few-NERD is a large-scale, human-annotated dataset specifically designed for few-shot Named Entity Recognition (NER) in the context of fine-grained and hierarchically-structured entity types. Addressing longstanding gaps in NER evaluation, Few-NERD provides both a dense, richly-typed corpus and a suite of benchmark protocols for systematic assessment of generalization to new types, outperforming prior re-organizations of supervised NER corpora in scope and granularity. Its release targets the research community interested in meta-learning, transfer learning, and robust evaluation of NER under practical low-resource settings (Ding et al., 2021).

1. Dataset Construction and Annotation Protocol

The corpus is constructed from the full English Wikipedia dump. For each candidate fine-grained entity type, a “distant” dictionary is built by harvesting FIGER-annotated mentions and manually performing denoising. Using Lucene, Wikipedia paragraphs are indexed and for each fine type, up to 10 paragraphs are retrieved that contain mentions from this dictionary. Subsequently, 1,000 paragraphs per type are randomly sampled for manual annotation, generating a total of 66,000 paragraphs with an average of 61.3 tokens each, tokenized and split using NLTK. The final dataset contains 188,238 sentences and 4,601,160 tokens.

Annotation is performed according to strict guidelines: each token receives an IO-format label (I–〈fine type〉 if within an entity mention, O otherwise). The coarse entity type is implied from the fine type. The annotation pool includes 70 linguistically trained annotators and 10 experts, following a detailed manual that includes positive and negative examples for every type. Each paragraph is independently labeled by two annotators with disagreements adjudicated by a senior expert. Quality checks involve re-annotating a random 10% of each batch (1,000–3,000 sentences); batches under 95% sentence-level accuracy are redone. Inter-annotator agreement is measured as Cohen’s κ = 0.7644.

2. Hierarchical Entity Typology

Few-NERD establishes a two-level type system: 8 coarse-grained classes and 66 fine-grained subtypes. The hierarchy is as follows:

Coarse Type	Fine-grained Types (examples)
Person	Actor, Athlete, Director, Politician, Scholar, Soldier, etc.
Location	GPE, BodyOfWater, Island, Mountain, Park, Road/Transit, etc.
Organization	Company, Education, Government, Media, SportsTeam, etc.
Art	Broadcast, Film, Music, Painting, WrittenArt, etc.
Building	Airport, Hospital, Library, Restaurant, SportsFacility, etc.
Product	Airplane, Car, Food, Game, Ship, Software, Weapon, etc.
Event	Attack, Election, NaturalDisaster, Protest, SportsEvent, etc.
Miscellaneous	Astronomy, Award, Biology, Currency, Disease, Law, etc.

Each type is annotated in context, supporting both dense and fine-grained entity recognition scenarios.

3. Benchmark Task Design

Few-NERD defines a meta-learning framework for N-way, K-shot sequence-labeling episodic tasks. In each episode:

• Support set S: labeled sentences cumulatively covering N target fine types, with at least K labeled mentions per type.
• Query set Q: unlabeled sentences containing the same N types (labels hidden during evaluation).
• K–2K greedy sampling is used since dense paragraph annotation violates strict K-shot settings: sentences are sampled so that (i) no type’s count in S exceeds 2K and (ii) no more than N types are introduced. This process continues until each of the N types reaches at least K mentions.

Four principal configurations are provided: 5-way 1–2-shot, 5-way 5–10-shot, 10-way 1–2-shot, and 10-way 5–10-shot.

Three distinct benchmark settings are established:

1. FEW-NERD (SUP): Standard supervised split—66 types present in each of train (70%), dev (10%), and test (20%) sets.
2. FEW-NERD (INTRA): Coarse-type disjoint—coarse families are partitioned so that train, dev, and test do not share any coarse type (e.g., train: Person, MISC, Art, Product; test: Organization, Location). Mentions of test/dev types in training are labeled as O.
3. FEW-NERD (INTER): Fine-type disjoint, coarse-type shared—within each coarse type, fine types are assigned to train, dev, test splits (~60%/20%/20%), with out-of-split mentions relabeled as O.

4. Evaluation Metrics and Formulas

Standard NER evaluation metrics are adopted, computed at the token level:

• Precision: $Precision = \frac{TP}{TP + FP}$
• Recall: $Recall = \frac{TP}{TP + FN}$
• F1 score: $$F_1 = 2\;\frac{Precision \times Recall}{Precision + Recall}$$

For modeling approaches beyond standard supervised learning, additional loss and scoring formulations are used:

• BERT-Tagger loss: Token-wise cross-entropy over the IO label set.
• Prototypical networks: Class prototypes $c_k$ are centroids over support set embeddings; query likelihoods use softmax over $L_2$ distance: $$p(y=k|x) = \frac{\exp(-d(f(x), c_k))}{\sum_{j=1}^{N} \exp(-d(f(x), c_j))}$$.
• NNShot: Nearest-neighbor emission: predicted label minimizes embedding distance between query token and support set.
• StructShot: NN emissions plus a learned transition model, with sequence decoding by Viterbi algorithm over transition and emission scores.

5. Results and Empirical Insights

The following summarizes the core empirical findings:

• BERT-Tagger, trained on Few-NERD (SUP), achieves F1 = 67.13% for 66 types—22 points below performance on OntoNotes-5.0 (F1 = 89.11%) and CoNLL-03 (F1 = 91.34%).
• In few-shot (5-way 1–2-shot, INTRA) settings, F1 is low for all meta-learning approaches: Proto 20.8±0.8, NNShot 25.8±0.9, StructShot 30.2±0.9.
• Results improve with increased shots: For 5-way 5–10-shot, INTRA, Proto and StructShot reach F1 ≈ 38–42%; for INTER, highest F1 (StructShot) is 51.9±0.7 (1–2-shot) and 57.3±0.6 (5–10-shot).
• INTRA (coarse-type disjoint) is substantially harder than INTER (coarse-type shared).
• Span boundary errors dominate performance loss: incorrect segmentation of entity extents leads to both false positives and negatives. Once the span is correct, type assignment within the coarse type is more reliable.
• The results emphasize that even simple few-shot NER tasks remain challenging for current algorithms.

6. Usage, Experimental Protocol, and Access

Few-NERD provides official dataset splits and scripts for preprocessing, formatting, and benchmarking. The dataset and code are available at https://ningding97.github.io/fewnerd/ and https://github.com/thunlp/Few-NERD. Data is formatted with one token per line and its IO label, with sentences separated by blank lines. JSON conversion scripts are included.

Official splits (SUP, INTRA, INTER), fixed seeds, and pre-computed development/test episodes ensure strong reproducibility. Hyperparameters are standardized (BERT-base uncased, AdamW learning rate $1\times 10^{-4}$, batch sizes per paper, 12,000 training episodes, and 5,000 evaluation episodes). The evaluation code computes micro-averaged Precision, Recall, and F1 over all query tokens. Training is designed for two NVIDIA V100 GPUs (approx. 120M parameters). Model checkpoints for baselines are provided.

7. Applications, Limitations, and Research Directions

Few-NERD is established as the first few-shot NER benchmark constructed at this scale and granularity with human annotation. The dataset’s complexity—especially the hierarchical and dense entity labeling—exposes clear limitations in current few-shot and meta-learning approaches. Failure cases concentrate on mention boundary detection, suggesting that approaches integrating explicit span modeling or leveraging type hierarchies may yield improvements. The benchmark further motivates research in prompt- or generation-based strategies and incorporation of external knowledge sources for robust transfer, especially across disjoint coarse-type scenarios. Few-NERD is positioned as a foundational resource for systematic progress in few-shot and generalizable NER (Ding et al., 2021).