LC-ICL: Label-Guided Contrastive In-Context Learning for Robust Information Extraction

Abstract: There has been increasing interest in exploring the capabilities of advanced LLMs in the field of information extraction (IE), specifically focusing on tasks related to named entity recognition (NER) and relation extraction (RE).Although researchers are exploring the use of few-shot information extraction through in-context learning with LLMs, they tend to focus only on using correct or positive examples for demonstration, neglecting the potential value of incorporating incorrect or negative examples into the learning process.In this paper, we present LC-ICL a novel few-shot technique that leverages both correct and incorrect sample constructions to create in-context learning demonstrations. This approach enhances the ability of LLMs to extract entities and relations by combining positive samples with negative samples annotated by error-cause labels. These labels expose more detailed error features in erroneous examples, enabling the model to understand why similar predictions fail and avoid repeating such errors during inference.Specifically, our proposed method taps into the inherent contextual information and valuable information in hard negative samples and the nearest positive neighbors to the test and then applies the in-context learning demonstrations based on LLMs. Our experiments on various datasets indicate that LC-ICL outperforms previous few-shot in-context learning methods, delivering substantial enhancements in performance across a broad spectrum of related tasks. These improvements are noteworthy, showcasing the versatility of our approach in diverse scenarios.

• The paper introduces LC-ICL, a contrastive in-context learning framework that utilizes error-labeled negative examples to enhance extraction robustness.
• The methodology combines KNN-based retrieval with structured error annotation, achieving up to a 159.8% improvement in F1 scores on relation extraction tasks.
• The paper highlights practical implications for improving LLM error awareness and performance in low-resource and complex information extraction scenarios.

LC-ICL: Label-Guided Contrastive In-Context Learning for Robust Information Extraction

Introduction

Information extraction (IE) comprises core tasks such as named entity recognition (NER) and relation extraction (RE), which benefit significantly from recent advances in LLMs. While in-context learning (ICL) with few-shot prompts has enabled LLMs to perform IE with minimal supervision, prevailing approaches predominantly utilize only positive, correct exemplars. The paper "LC-ICL: Label-Guided Contrastive In-Context Learning for Robust Information Extraction" (2606.29407) addresses the underexploited utility of negative, incorrect exemplars in ICL for IE tasks. LC-ICL presents a systematic framework to inject both positive and negative samples—where negatives are annotated with fine-grained error labels—into in-context demonstrations. The central hypothesis is that such contrastive setups provide LLMs with richer supervision, enhancing error awareness and ultimately improving extraction robustness and precision.

Methodology

LC-ICL constructs each in-context prompt as a composite of instructions, positive samples, negative samples with error labels, and the target test sentence. Unlike traditional ICL that retrieves positive nearest neighbors based on embedding similarity, LC-ICL additionally performs inference over a stratified pool of training samples to identify "hard" negatives—examples the model itself mispredicts. An auxiliary labeling model then annotates each negative with structured error labels, systematically categorizing error modes specific to NER and RE (e.g., "Wrong entity type", "Missing relation", "Reversed entities").

Figure 1: Core mechanism of LC-ICL in RE—feedback via error labels leads to refined, lower-error predictions.

Figure 2: End-to-end pipeline for the LC-ICL framework: construction and integration of positive/negative examples with error labeling.

For retrieval, KNN is employed in the embedding space to maximize contextual compatibility for both positive and negative exemplars relative to the target input. At inference, the LLM thus receives not only typical success patterns but also explicit representations of failure and the linguistic circumstances causing those failures.

Experimental Evaluation

Datasets and Settings

LC-ICL is evaluated across three NER datasets (ACE04, ACE05, NCBI) and five RE datasets (CoNLL04, NYT10, NYT11, SciERC, ADE). Llama-3.1-8B-Instruct and Llama-3.3-70B-Instruct serve as the main backbone LLMs for inference, with DeepSeek used to generate high-fidelity error labels. Both strict F1 metrics and instruction-following scores are reported. To sample negatives efficiently, a pool of 2,500 training instances per dataset is processed.

Main Results

LC-ICL achieves substantial performance improvements over both zero-shot (no ICL) and conventional ICL (using only positive examples). Across all model scales and datasets, absolute F1 scores increase markedly—most notably, in RE on Llama-3.1-8B-Instruct, LC-ICL improves NYT10 from 15.73% to 38.69% (+146%) and NYT11 from 8.28% to 21.52% (+159.8%). On Llama-3.3-70B-Instruct, RE gains remain robust, e.g., CoNLL04 climbs from 48.42% (PE) to 50.71% (LC-ICL).

NER tasks display similar trends, especially in medical-domain settings (NCBI), where gains exceed +18 percentage points in some configurations. Gains are consistently demonstrated even in random retrieval scenarios, indicating that the core advantage derives from negative label supervision rather than retrieval strategy.

Figure 4: Comparative F1 performance for RE/NER across random retrieval, showing outperformance by LC-ICL with label guidance.

Ablation and Analysis

Negative Sample Annotation Schemes

LC-ICL’s error annotation strategy was ablated by substituting structured error labels with free-form explanations. Structured labels consistently outperformed explanation-based negatives, especially for multi-relational and complex-structured datasets. For tasks with homogeneous or singular label spaces, the benefit of explicit error categorization diminished.

Figure 5: Impact of structured label annotation vs. free-form explanation in negative sample construction across NER and RE tasks.

Label Definition Granularity

Further ablations compared labels alone to labels augmented with explicit definitional explanations. Contrary to initial intuitions, providing label explanations attenuated performance, likely due to increased prompt complexity and a rise in distractor information.

Figure 3: Performance differential between using error labels alone versus labels combined with definitions.

Case Studies and Prompt Design

Detailed case analysis confirmed that negative examples—paired with explicit error labels—enhance the model’s ability to discriminate correct versus incorrect outputs in structurally ambiguous or noisy examples.

Figure 6: Prompt construction in LC-ICL (left, with labeled negative) vs. standard ICL (right, positive only); LC-ICL avoids false relations by explicitly signaling past relation prediction errors.

Theoretical and Practical Implications

LC-ICL makes several contributions of broader import to few-shot learning with LLMs:

• Contrastive demonstration composition—leveraging error label-guided negative examples—enriches supervision and yields more robust generalization compared to positive-only demonstration paradigms.
• Structured error taxonomy annotation creates a feedback channel allowing LLMs to reflect on and rectify common failure dynamics during multi-turn or domain-shifted scenarios.
• The demonstrated gains suggest potential application in other low-resource or structure-intensive IE subfields, as well as for broader LLM alignment and instruction-following tasks.
• Methodologically, the findings support a move toward more fine-grained and task-aware demonstration engineering, with direct analogues in computer vision and other structured prediction settings.

The approach remains effective as model scale increases, suggesting that error context remains salient for LLMs with substantial inherent capacity.

Limitations and Future Directions

The current work is primarily validated on English-language NER and RE, and on a limited set of information extraction paradigms. Extension to other IE settings (e.g., event extraction, cross-lingual settings), the integration of richer negative retrieval strategies, and scaling to multi-turn or open-domain scenarios are natural avenues for future research. Additionally, exploring soft labeling or uncertainty quantification for negative exemplars could further enhance adaptability.

Conclusion

LC-ICL introduces and empirically validates a label-guided, contrastive framework for in-context learning in information extraction. By explicitly integrating error-labeled negatives into the in-context prompt, the method demonstrates substantial and robust gains over positive-only ICL. These results underscore the value of negative supervision in few-shot LLM application and open pathways for further investigation into structured contrastive context design for robust language understanding and extraction tasks.