ContrastNER: Prompt-Based NER Framework

• ContrastNER is a prompt-based NER framework that combines discrete and continuous tokens with contrastive learning to forecast entity types in few-shot settings.
• It replaces manual verbalizer design with representation-space forecasting and kNN retrieval, reducing sensitivity to prompt variations.
• Empirical results show that ContrastNER achieves competitive performance in both rich-resource and low-resource scenarios while enhancing robustness.

ContrastNER is a prompt-based named entity recognition framework that employs both discrete and continuous tokens in prompts and uses a contrastive learning approach to learn the continuous prompts and forecast entity types. It was introduced for few-shot NER, where only $K$ labeled instances per entity type are available, and it targets the sensitivity of prompt-based NER to discrete template design and handcrafted verbalizers by replacing verbalizer-based prediction with representation-space forecasting (Layegh et al., 2023).

1. Conceptual basis and problem setting

ContrastNER is formulated within the standard NER setting in which a sentence $X=\{x_1,\ldots,x_m\}$ is assigned token-level labels $y_i \in Y$, where $Y$ includes named entity types and $O$ (“not an entity”). Its motivation is rooted in two observations. First, prompt-based NER reframes downstream prediction as masked language modeling with a template and a verbalizer, but discrete prompts are highly sensitive and small wording changes can cause large performance variations. Second, manual prompt engineering and verbalizer design are labor-intensive and brittle across domains, especially in few-shot settings (Layegh et al., 2023).

The method therefore combines discrete (“hard”) and continuous (“soft”) prompt tokens, and removes the manual verbalizer by training the model to forecast entity types via contrastive learning in a learned embedding space. In the paper’s experimental protocol, the few-shot regime is instantiated with $K \in \{10,20,50,100,200,500\}$, and evaluation includes cross-domain low-resource targets such as MIT Movie Review, MIT Restaurant Review, and ATIS, as well as CoNLL-2003 in both rich-resource and downsampled configurations (Layegh et al., 2023).

A central design choice is that ContrastNER operates at the token level without explicit BIO tags in the prompt. Each token is treated as a candidate entity, including the $O$ class, and token-level predictions are subsequently aggregated into the final NER output. This differentiates it from span-enumeration systems and from prompt formulations that depend on label words standing in for entity categories (Layegh et al., 2023).

2. Prompt construction and representational architecture

For a sentence $X=\{x_1,\ldots,x_m\}$, ContrastNER generates $m$ prompted inputs by slotting each token $x_i$ as a candidate entity in a hard prompt and appending $X=\{x_1,\ldots,x_m\}$0 learnable continuous prompt tokens $X=\{x_1,\ldots,x_m\}$1. Its canonical hard template is:

$X=\{x_1,\ldots,x_m\}$2

The resulting prompted input has the form:

$X=\{x_1,\ldots,x_m\}$3

For the sentence “Steve Jobs was born in America,” the paper instantiates prompts such as “Steve Jobs was born in America $X=\{x_1,\ldots,x_m\}$4 Steve is a [MASK] entity.” and “Steve Jobs was born in America $X=\{x_1,\ldots,x_m\}$5 America is a [MASK] entity.” (Layegh et al., 2023).

The pretrained LLM is RoBERTa from Hugging Face. For each prompted input, the hidden representation at the $X=\{x_1,\ldots,x_m\}$6 position is extracted as $X=\{x_1,\ldots,x_m\}$7, and this vector functions as the label embedding for the candidate token in context. The method does not introduce a prompt encoder, and the forecasting path does not rely on a label-specific classifier or CRF head. Similarity is computed directly with cosine similarity on these $X=\{x_1,\ldots,x_m\}$8-position representations (Layegh et al., 2023).

The paper also studies prompt sensitivity under several alternative discrete templates, including “The entity type of <candidate_entity> is <entity_type>.” and “<candidate_entity> belongs to <entity_type> category.” The reported conclusion is that the hybrid soft-hard prompt makes the framework less sensitive to the exact discrete wording than TemplateNER, because the learnable soft tokens adapt during training (Layegh et al., 2023).

3. Contrastive training objective and verbalizer-free inference

ContrastNER optimizes two losses. The first is a supervised contrastive term $X=\{x_1,\ldots,x_m\}$9 that clusters examples of the same label and separates examples of different labels. The second is a cross-entropy term $y_i \in Y$0 that stabilizes optimization of the soft prompts by supervising the $y_i \in Y$1 prediction with the ground-truth entity type. For a batch $y_i \in Y$2, with representation $y_i \in Y$3 for example $y_i \in Y$4, positives

$y_i \in Y$5

and

$y_i \in Y$6

the paper uses cosine similarity with temperature $y_i \in Y$7 and defines an InfoNCE-style supervised contrastive loss as

$y_i \in Y$8

The total training objective is

$y_i \in Y$9

In the reported implementation, $Y$0 and $Y$1 (Layegh et al., 2023).

Inference is verbalizer-free. Given a test example with representation $Y$2, the model retrieves the $Y$3 nearest labeled training examples by cosine similarity and assigns the majority label among neighbors, optionally with similarity weighting. This nearest-neighbor forecasting eliminates the need for a handcrafted verbalizer while keeping label prediction in the learned embedding space. In high-level terms, training jointly updates RoBERTa and the soft-prompt embeddings with Adam, using learning rate $Y$4 and batch size $Y$5 (Layegh et al., 2023).

This design suggests a metric-learning interpretation of prompt tuning: the prompt is not only an input reformulation device, but also a mechanism for shaping a label-aware geometry in which entity types are recovered by neighborhood structure rather than lexical label-word mapping.

4. Empirical behavior, benchmark performance, and robustness

On rich-resource CoNLL-2003, ContrastNER reports Precision $Y$6, Recall $Y$7, and F1 $Y$8. In that setting it is competitive with high-resource NER systems such as TemplateNER, sequence-labeling BERT, sequence-labeling BART, and BART-NER, although LUKE remains higher at F1 $Y$9 (Layegh et al., 2023).

In the paper’s in-domain few-shot CoNLL-03 setting, PERSON and ORGANIZATION are treated as rich-resource while LOCATION and MISCELLANEOUS are downsampled to $O$0 instances each, producing a training set with $O$1 samples: $O$2 PERSON, $O$3 ORGANIZATION, $O$4 MISCELLANEOUS, and $O$5 LOCATION. Under that setup, ContrastNER reports PERSON $O$6, ORGANIZATION $O$7, LOCATION $O$8, MISCELLANEOUS $O$9, and Overall $K \in \{10,20,50,100,200,500\}$0 F1. It exceeds TemplateNER’s overall F1 $K \in \{10,20,50,100,200,500\}$1 by $K \in \{10,20,50,100,200,500\}$2 and slightly surpasses LightNER’s $K \in \{10,20,50,100,200,500\}$3 by $K \in \{10,20,50,100,200,500\}$4 (Layegh et al., 2023).

The cross-domain few-shot results are described as more consistent than those of sequence-labeling BERT, sequence-labeling BART, and TemplateNER. A specific example given in the paper is that on one target, MIT movies, at $K \in \{10,20,50,100,200,500\}$5-shot, ContrastNER reaches $K \in \{10,20,50,100,200,500\}$6 F1, exceeding the baselines’ results even at $K \in \{10,20,50,100,200,500\}$7-shot. In the transfer setting that trains on CoNLL-03 and evaluates on target domains, prompt-based methods outperform sequence labeling across $K \in \{10,20,50,100,200,500\}$8, and ContrastNER achieves the best overall performance, surpassing LightNER once $K \in \{10,20,50,100,200,500\}$9 (Layegh et al., 2023).

Prompt robustness is another reported empirical property. ContrastNER is less sensitive to the choice of discrete template than TemplateNER, and the paper attributes this to the soft-hard prompting mechanism. The contrastive term appears to become particularly effective beyond approximately $O$0-shot, indicating that the embedding-space forecasting mechanism benefits from additional support examples when domain transfer is involved (Layegh et al., 2023).

5. Position within contrastive NER research

ContrastNER belongs to a larger line of work that treats NER as a representation-learning problem, but its specific combination of prompt tuning, continuous prompt tokens, and verbalizer-free nearest-neighbor prediction is distinct. Contemporary and subsequent research diversified contrastive NER along several orthogonal axes (Layegh et al., 2023).

One line reframes NER as span–type matching rather than prompt-based token classification. BINDER uses two isomorphic, decoupled Transformer encoders to map candidate mention spans and entity types into the same vector space, turning NER into a matching problem and introducing dynamic thresholding to separate entities from non-entities without an explicit $O$1 class. It supports nested and flat NER and reports new state of the art on datasets including ACE2004, ACE2005, GENIA, NCBI, BC5CDR, and JNLPBA (Zhang et al., 2022).

Another line focuses on few-shot token representations without prompt tuning. CONTaiNER models each token as a diagonal Gaussian and optimizes a contrastive objective over symmetric KL distances between token distributions, rather than point embeddings, in order to improve generalization across domains and entity inventories. MsFNER, by contrast, splits few-shot NER into entity-span detection and entity classification, adds entity-aware supervised contrastive learning over span representations, and uses a hybrid KNN-plus-prototype decision rule at inference (Das et al., 2021, Liu et al., 2024).

Cross-lingual variants extend contrastive learning to translation pairs, token-label alignment, or pseudo-labeled span clusters. ConCNER combines Translation Contrastive Learning at the sentence level with Label Contrastive Learning at the token level, together with knowledge distillation on unlabeled target-language data. mCL-NER reformulates CrossNER as token-to-token relation classification and aligns both sentence semantics and token-pair relations across languages. ContProto incorporates supervised contrastive self-training and prototype-based pseudo-label refinement for span classification under cross-lingual self-training (Fu et al., 2022, Mo et al., 2023, 2305.13628).

Other extensions move contrastive NER toward label-aware prompting, multimodality, and LLMs. A unified label-aware few-shot framework injects label semantics as suffix prompts and combines context–context with context–label contrastive objectives. 2M-NER aligns sentence-level text and image representations and then performs multimodal collaboration for multilingual and multimodal NER. C-ICL transfers contrastive principles into in-context learning by combining nearest positive demonstrations with hard negative demonstrations that are explicitly flagged and corrected inside the prompt (Zhang et al., 2024, Wang et al., 2024, Mo et al., 2024).

This suggests that ContrastNER occupies a specific point in a broader design space: it is prompt-centric and token-centric, whereas related systems distribute contrastive learning across spans, distributions, cross-lingual alignments, retrieval-style bi-encoders, multimodal alignment, and in-context demonstrations.

6. Limitations, computational profile, and prospective extensions

The principal limitation explicitly noted for ContrastNER is computational overhead. Because each sentence is processed $O$2 times, once per token candidate, training and inference are more expensive than in single-pass sequence labeling, and memory and compute scale with the number of token-level prompts per batch. Inference also requires kNN search over stored labeled examples, although the paper notes that this can be accelerated through approximate nearest-neighbor indexing (Layegh et al., 2023).

A second limitation is representational granularity. The framework operates at the token level without explicit BIO tags in the prompt, and the paper identifies explicit handling of multi-token spans as a potential extension. Relatedly, the number of soft tokens $O$3 and their initialization are not detailed, and systematic exploration of prompt length or initialization strategies is left open. Code availability is not specified in the provided summary, although the paper does report implementation details including RoBERTa, Adam, learning rate $O$4, batch size $O$5, Tesla T4 GPU, $O$6 CPU cores, and $O$7 GB RAM (Layegh et al., 2023).

The paper frames future work in terms of extending ContrastNER to relation extraction and evaluating on real-world datasets from the DataCloud project, including digital health, sports content, and manufacturing analytics. Relative to later span-based and cross-lingual contrastive models, a plausible implication is that several unresolved issues in ContrastNER—especially token-level overhead, lack of explicit span modeling, and dependence on stored labeled neighbors—became organizing themes for subsequent research on span-based bi-encoders, multi-stage decoding, and prototype-driven contrastive transfer (Zhang et al., 2022, Liu et al., 2024).

As a result, ContrastNER remains notable both as a specific few-shot prompt-tuning framework and as an early, clear formulation of verbalizer-free contrastive forecasting for NER: entity types are not decoded from label words, but recovered from a learned representation space shaped jointly by soft prompts and supervised contrast.