Scaling Performance and Low-Resource Annotation with Many-Shot In-Context Learning for Named Entity Recognition

Abstract: In-context learning (ICL) with LLMs has emerged as a powerful alternative to fine-tuning for Named Entity Recognition (NER), achieving strong performance with minimal annotation and no additional training. However, prior work has shown that despite their adaptability, LLMs still lag behind fully supervised models such as fine-tuned BERT in structured tasks like NER. While existing studies on ICL for NER have mainly explored few-shot settings, the potential of scaling to hundreds of demonstrations has not been thoroughly investigated. To address this gap, we conduct a comprehensive investigation of many-shot ICL for NER and further explore its effectiveness in annotating and refining data for low-resource NER tasks. Specifically, we evaluate various LLMs across multiple domains using hundreds of ICL examples and then assess the feasibility of using many-shot ICL as a data annotation framework. Our experiments demonstrate that: (1) scaling to hundreds of in-context examples enables LLMs to match or even surpass the performance of fully supervised BERT models; and (2) using about one hundred human-labeled examples as demonstrations, many-shot in-context annotation can generate high-quality labeled data, leading to approximately 10% absolute F1 improvement over existing state-of-the-art approaches when used to fine-tune BERT on low-resource NER.

• The paper demonstrates that many-shot in-context learning with hundreds of demonstrations enhances NER performance and can match supervised models.
• The ICA framework uses minimal human annotations combined with LLM-based refinement to generate high-quality, cost-efficient labeled data.
• Experiments reveal a steady F1 improvement across benchmark datasets, emphasizing effective retrieval-based demonstration selection and error-aware refinement.

Many-Shot In-Context Learning for NER: Performance Scaling, Annotation Efficiency, and the ICA Framework

Introduction and Motivation

The paper "Scaling Performance and Low-Resource Annotation with Many-Shot In-Context Learning for Named Entity Recognition" (2606.21890) provides an in-depth empirical and methodological investigation into the use of many-shot in-context learning (ICL) with LLMs for Named Entity Recognition (NER). The authors address two central limitations in existing LLM-based NER research: (1) the underexplored potential of scaling ICL to hundreds of demonstrations; and (2) the challenge of obtaining high-quality annotated data in low-resource domains without expensive human supervision.

The focus is on leveraging increased context lengths in modern LLMs (32K-128K tokens and beyond) to provide hundreds of in-context demonstrations, systematically measuring performance scaling, and utilizing LLMs for data annotation rather than real-time inference in low-resource NER. Critically, the paper introduces the In-Context Annotation (ICA) framework, a pipeline that uses many-shot ICL to annotate large quantities of unlabeled data with minimal human supervision, followed by a set of LLM-based refinement strategies tailored to typical NER errors.

Many-Shot In-Context Learning for NER: Task and Empirical Characterization

The setup treats NER as conditional generation: given a sentence, the LLM is prompted to annotate and output both the spans and types of entities within XML-style tags. Demonstrations are included in the prompt, ranging from a few to several hundred examples, with two selection strategies: random and retrieval-based (BM25 similarity).

Figure 1: XML-style tagging prompt with multiple in-context demonstrations encoding both span and entity type information.

The many-shot ICL scaling experiments cover four benchmark datasets: MIT-Movie, MIT-Restaurant (fine-grained, domain-specific), CoNLL2003 (newswire), and WNUT2017 (social media).

Figure 2: F1 performance on MIT-Movie, MIT-Restaurant, CoNLL2003, and WNUT2017 as the number of in-context demonstrations increases.

Key empirical findings:

• Monotonic scaling: For all LLMs tested, F1 rises monotonically as the number of demonstrations increases, plateauing at different rates by dataset/type.
• Retrieval-based demonstrations provide clear but diminishing benefits over random sampling as demonstration count increases.
• BERT-FT parity: With hundreds of demonstrations, competitive LLMs (DeepSeekV3, GPT-4o, Qwen-72B) match or exceed fully supervised BERT fine-tuning on thousands of annotated examples, despite requiring far less direct annotation.
• Open vs. closed source: Larger open-source models (Qwen-72B) close the performance gap with proprietary LLMs, but smaller versions and less-capable LLMs lag substantially even in many-shot ICL.

The ICA Framework: Many-Shot In-Context Annotation for Efficient Data Generation

Given the prohibitive inference cost of deployment-time many-shot ICL, the ICA framework is introduced to decouple LLM annotation from deployment. ICA consists of three components:

1. Seed annotation: 100 (occasionally up to 200) high-quality human annotations serve as fixed in-context demonstrations.
2. Bulk LLM annotation: The LLM, prompted with these demonstrations, labels thousands of unlabeled sentences in batch. Structured outputs (XML) are converted to standard BIO for downstream NER.
3. LLM-based refinement: Dedicated strategies are applied to low-confidence outputs, targeting three key error types: span/type self-consistency, self-correction (revision prompting), and Error-Aware Refinement (EAR), which decomposes error handling into focused sub-prompts.

   Figure 3: Three refinement methods (self-consistency, self-correction, EAR) in the ICA framework for systematically reducing annotation errors.

This pipeline yields annotated corpora used to train efficient BERT-based NER models. It is evaluated on the CrossNER benchmark, spanning 5 challenging domains (AI, Literature, Music, Politics, Science)—all with very limited initial labeled data.

Experimental Results: Data Efficiency, Refinement Gains, and Comparison to SOTA

The ICA framework is comprehensively benchmarked against a spectrum of baseline methods:

• LLM in-context inference (zero-shot and many-shot)
• Traditional data augmentation: DAGA, NERDA, GPDA, ProgGen
• Current SOTA NER transfer/augmentation systems: LST-NER, DoSEA, DTrans-SMix, DH-GAT, Three-KPNs, DTrans-MPrompt, PromptNER, B2NER, IF-WRANER, GLiNER

Strong Quantitative Results

• ICA with EAR achieves an average improvement of ~10 F1 points over previous best approaches on low-resource CrossNER domains, using only 100 human-labeled demonstrations per domain.
• Direct LLM inference (many-shot ICL) is competitive with strong transfer baselines, but the downstream BERT models trained on ICA-generated data further outperform the direct ICL outputs by 2-5 F1.
• Refinement strategies yield additional significant gains (EAR > self-correction/self-consistency), especially in low-resource and long-tail entity categories.

Figure 4: NER performance with increasing LLM-annotated (ICA) training data on the AI and Literature domains; early gains plateau after 2K-2.5K examples.

Figure 5: NER performance as the number of human-labeled seed annotations increases; major improvements saturate around 75–100 in-context examples.

Figure 6: Impact of LLM model choice on annotation utility; GPT-4o and large Qwen/DeepSeek models yield the highest downstream NER accuracy.

Ablation and Scaling Analyses

• Downstream model performance saturates after 1.5K-2.5K ICA-labeled examples (Figure 4); larger annotation sets provide only marginal returns given fixed-domain distribution.
• Human-labeled seed requirements saturate at 75–100 demonstrations (Figure 5), indicating practical feasibility for rapid data bootstrapping.
• Model choice is paramount (Figure 6): open-source 70B+ models approach closed model quality while smaller architectures fail to match SOTA.
• Refinement (especially EAR) more effectively addresses error modes unique to LLM generation (spurious entities, missing entities, type confusion) than simple majority voting/self-correction.

Qualitative Assessment and Case Studies

Case studies confirm that:

• Self-consistency and self-correction mitigate type and span boundary errors but are sycophantic if applied too generally.
• EAR is particularly effective at removing model hallucinations (spurious capitalized phrases) and recovering missed or misclassified entities, directly corresponding to the error modes uncovered in error analysis.

  Figure 7: ICA case study illustrating correction of entity type and recovery of missing/incorrect entities by refinement.

  

  Figure 8: ICA refinement removing spurious entities and correcting labels in a complex sentence.

Practical and Theoretical Implications

The results establish several key implications for both practice and future research:

• Many-shot ICL offers a practical alternative to expensive annotation or broad transfer learning in low-resource NER. Only hundreds of demonstrations suffice for competitive LLM annotation quality; ICA turns this into practical labeled datasets.
• Combining structured output formats and targeted refinement is crucial: Customized prompt engineering for span-type annotation (XML tags) and modular error handling unlock significant quality gains over naïve zero/few-shot prompting.
• Resource efficiency is improved: ICA achieves superior F1 at a small fraction of prior human annotation cost, and tagged data enables deployment of small, efficient models rather than computationally expensive LLM inference.
• The approach highlights the importance of in-context learning for data creation (dataset bootstrapping) rather than just inference, informed by scaling trends for annotation and LLM capabilities.

For future directions, hybrid systems could exploit the separation between LLM-based offline annotation and efficient downstream model training; further research on generalization to other languages, annotation schemes, or more ambiguous contexts remains critical. Additionally, as LLM architectures evolve, the demonstration scaling curve and the minimal requirements for high-quality labeling may shift, requiring ongoing re-evaluation for both open-source and proprietary LLMs.

Conclusion

This work rigorously demonstrates that many-shot in-context learning with LLMs yields monotonic improvements in NER performance with increasing demonstrations, and—when embedded in the ICA framework—enables the creation of high-quality, large-scale labeled data from minimal initial human effort. The ICA methodology, supported by LLM-centric refinement techniques, offers a highly data- and cost-efficient path toward accurate NER in low-resource domains. These findings urge a rethinking of annotation and deployment strategies for NER and related sequence labeling tasks, emphasizing the centrality of ICL for both efficiency and quality under realistic annotation bottlenecks.