CoLAP: Contrastive Alignment with Prompting

• The paper introduces CoLAP, which combines prompted classification with a contrastive loss to narrow high-resource versus low-resource gaps in multilingual models.
• It employs XRCL and XCCL objectives to align task-specific representations across languages using either translated pairs or class label agreement.
• Empirical findings show that CoLAP outperforms traditional fine-tuning and in-context learning, especially in very low-resource few-shot settings.

Searching arXiv for the cited CoLAP paper and closely related contrastive prompting work to ground the article with current references. Contrastive Language Alignment with Prompting (CoLAP) is a prompt-based transfer method for few-shot cross-lingual adaptation that combines prompted task formulation with contrastive alignment of multilingual task representations. In the formulation introduced in "Bridging Language Gaps: Enhancing Few-Shot Language Adaptation" (Borchert et al., 26 Aug 2025), a multilingual pretrained LLM is first fine-tuned on a high-resource source language and then adapted to a target language using only a few labeled examples. During this adaptation stage, CoLAP augments prompted classification with a contrastive objective that pulls semantically corresponding source- and target-language representations together while pushing mismatched examples apart. The method is presented as a response to the persistent high-resource versus low-resource gap in multilingual pretrained LLMs, including mBERT, XLM-R, and newer multilingual decoder-only models (Borchert et al., 26 Aug 2025).

1. Problem formulation and motivation

CoLAP is studied in the setting of few-shot cross-lingual transfer (FS-XLT). In this setting, a multilingual pretrained LLM is fine-tuned on a high-resource source language—English in the reported experiments—and then adapted to a target language using only a few labeled examples (Borchert et al., 26 Aug 2025). The motivating observation is that multilingual pretrained LLMs are not equally competent across languages because their pretraining corpora are heavily skewed toward high-resource languages. As a result, high-resource languages receive richer and more discriminative representations, low-resource languages receive weaker representations, and languages unseen during pretraining are especially disadvantaged (Borchert et al., 26 Aug 2025).

The central hypothesis behind CoLAP is that this disparity is not only a language-modeling problem but also a task-specific representation problem. The method assumes that if target-language task representations can be aligned with the stronger English task space, then discriminative knowledge can be transferred more effectively even when labeled target-language data is scarce (Borchert et al., 26 Aug 2025). This makes CoLAP a task-level adaptation method rather than a general multilingual pretraining remedy.

A plausible implication is that CoLAP belongs to a broader family of techniques that treat prompting not merely as an inference interface, but as a representational scaffold for transfer. In this respect it differs from other uses of contrastive prompting in the literature. For example, "Customizing LLM Responses with Contrastive In-Context Learning" (Gao et al., 2024) uses positive and negative in-context examples to steer generation preferences, while "Contrastive Language Prompting to Ease False Positives in Medical Anomaly Detection" (Park et al., 2024) uses positive and negative prompts to suppress false positive attention in biomedical visual-language anomaly detection. CoLAP, by contrast, uses prompting as the front end for cross-lingual classification and contrastive learning as the mechanism for aligning task representations across languages (Borchert et al., 26 Aug 2025).

2. Prompt-based task formulation

CoLAP is built on prompt-based classification rather than standard sequence classification (Borchert et al., 26 Aug 2025). Given an input $x$, the model applies a template $T$ to produce a prompted input

$x^{prompt} = T(x)$

and predicts a label word from the vocabulary rather than directly predicting a class identifier (Borchert et al., 26 Aug 2025). For masked LLMs such as XLM-R, a $\langle mask \rangle$ token is inserted into the prompt and its hidden state is used for prediction. For causal decoder-only models, the hidden state of the $\langle EOS \rangle$ token is used instead (Borchert et al., 26 Aug 2025).

The prompted prediction objective is written as

$$p(y \mid x) = p(\langle EOS \rangle = w \mid x^{prompt}) = \frac{\exp(w \cdot h_{\langle EOS \rangle})}{\sum_{v \in V} \exp(w_v \cdot h_{\langle EOS \rangle})}$$

where $h_{\langle EOS \rangle}$ is the hidden state at $\langle EOS \rangle$, $w$ is the token embedding or output vector for the label word, and $V$ is the vocabulary (Borchert et al., 26 Aug 2025). The paper uses English label words and language-agnostic prompts (Borchert et al., 26 Aug 2025).

The reported templates are deliberately simple. For XNLI / AmNLI, XLM-R uses

$T$0

while Gemma and Mistral use

$T$1

For MultiTACRED, XLM-R uses

$T$2

and Gemma / Mistral use

$T$3

(Borchert et al., 26 Aug 2025).

This prompting formulation matters because it converts multilingual classification into a language-modeling style objective. The paper argues that this makes transfer more natural for pretrained LLMs and provides a stable interface for the additional contrastive objective (Borchert et al., 26 Aug 2025). This suggests that in CoLAP, prompting is not an auxiliary embellishment but a structural component of the transfer mechanism.

3. Contrastive alignment objectives

The defining feature of CoLAP is the addition of a contrastive loss during few-shot adaptation to the target language, after English task fine-tuning (Borchert et al., 26 Aug 2025). The total loss is

$T$4

where $T$5 is the standard cross-entropy loss for prompted classification (Borchert et al., 26 Aug 2025).

XRCL: Cross-lingual Representation Contrastive Learning

The first variant, XRCL, requires parallel translations between source and target language (Borchert et al., 26 Aug 2025). For each target-language example $T$6, the corresponding source-language translation $T$7 is treated as a positive pair, while other examples in the batch serve as negatives (Borchert et al., 26 Aug 2025). Representations are extracted from the prompted model as

$T$8

where $T$9 is the pretrained LLM and the representation is typically taken from the hidden state of $x^{prompt} = T(x)$0 or $x^{prompt} = T(x)$1 (Borchert et al., 26 Aug 2025).

The loss is InfoNCE-like:

$x^{prompt} = T(x)$2

with cosine similarity

$x^{prompt} = T(x)$3

where $x^{prompt} = T(x)$4 is the number of instances and $x^{prompt} = T(x)$5 is a temperature hyperparameter (Borchert et al., 26 Aug 2025). The intended effect is instance-level alignment: the target-language representation is explicitly pulled toward its English translation in the task-specific representation space (Borchert et al., 26 Aug 2025).

XCCL: Cross-lingual Class Contrastive Learning

The second variant, XCCL, is designed to be more data-efficient because it does not require parallel translations (Borchert et al., 26 Aug 2025). Instead of pairing translated instances, it pairs examples across languages that share the same class label. For a target-language representation $x^{prompt} = T(x)$6, positives are source-language examples with the same label and negatives are source-language examples with different labels (Borchert et al., 26 Aug 2025). The loss is written as

$x^{prompt} = T(x)$7

(Borchert et al., 26 Aug 2025).

The conceptual distinction is explicit in the paper: XRCL performs instance-level alignment using translations, whereas XCCL performs class-level alignment using label agreement (Borchert et al., 26 Aug 2025). XCCL therefore reduces annotation cost because translated pairs are not required, making it more practical in low-resource classification settings (Borchert et al., 26 Aug 2025).

This contrastive design places CoLAP in a broader methodological space in which positive alignment and negative separation are jointly used to improve representational specificity. Related work in vision-language systems exhibits a similar pattern. In medical anomaly detection, CLAP defines $x^{prompt} = T(x)$8 so that negative prompts attenuate false alarms on normal tissue (Park et al., 2024). In zero-shot object detection, automated prompt refinement based on the Contrastive Class Alignment Score (CCAS) ranks prompts by similarity to the target class and dissimilarity to confounding classes (Choi et al., 14 May 2025). CoLAP extends this contrastive logic to multilingual task representations rather than prompt strings or spatial attention maps (Borchert et al., 26 Aug 2025).

4. Training procedure, models, and experimental regime

The training recipe has three stages. First, the model is fine-tuned on English task data with prompting and cross-entropy loss. Second, it is adapted to the target language using few-shot examples. Third, during target-language adaptation, either $x^{prompt} = T(x)$9 or $\langle mask \rangle$0 is added to the objective (Borchert et al., 26 Aug 2025).

The paper reports the following hyperparameters for CoLAP: batch size 64, learning rate 2e-5, optimizer AdamW, English fine-tuning for 5 epochs, and few-shot adaptation for 10 epochs (Borchert et al., 26 Aug 2025). No dedicated validation set is used, in order to remain faithful to few-shot conditions and avoid consuming extra labeled data (Borchert et al., 26 Aug 2025).

The method is evaluated on both encoder-only and decoder-only multilingual pretrained LLMs:

Model family	Model	Notes
Encoder-only	XLM-R Base	270M parameters
Decoder-only	Gemma 2 2B	4-bit quantization, LoRA
Decoder-only	Mistral v0.3 7B	4-bit quantization, LoRA

For Gemma and Mistral, the reported adapter settings are LoRA rank $\langle mask \rangle$1 and alpha $\langle mask \rangle$2 (Borchert et al., 26 Aug 2025).

Few-shot target-language adaptation is tested with

$\langle mask \rangle$3

and for each $\langle mask \rangle$4, episodes are randomly sampled from the target-language training set (Borchert et al., 26 Aug 2025). For contrastive loss computation, an episode contains $\langle mask \rangle$5 target-language instances and, for XRCL or XCCL, also $\langle mask \rangle$6 source-language instances (Borchert et al., 26 Aug 2025). Results are averaged over five random seeds (Borchert et al., 26 Aug 2025).

The evaluation covers three multilingual benchmarks:

Dataset	Scope	Metric
XNLI	15 languages, natural language inference	accuracy
AmericasNLI (AmNLI)	10 indigenous languages of the Americas	accuracy
MultiTACRED	12 languages, multilingual relation extraction	accuracy

These datasets were chosen to include both languages seen during multilingual pretraining and languages that are essentially unseen or severely underrepresented (Borchert et al., 26 Aug 2025). This experimental design is significant because it probes not only transfer among relatively resourced languages, but also transfer into genuinely low-resource and pretraining-sparse regimes.

5. Empirical findings and comparative performance

The reported empirical trend is that CoLAP consistently improves few-shot cross-lingual transfer over standard FT and CA, and often surpasses PCT as well (Borchert et al., 26 Aug 2025). The baselines considered are regular fine-tuning (FT), checkpoint averaging (CA), prompt-learning from cross-lingual templates (PCT), and in-context learning (ICL) for decoder-only models in the $\langle mask \rangle$7 setting (Borchert et al., 26 Aug 2025).

Several findings are emphasized. First, gains are especially visible in very low-resource settings such as $\langle mask \rangle$8 and $\langle mask \rangle$9 (Borchert et al., 26 Aug 2025). Second, CoLAP helps both languages included in multilingual pretraining and languages not seen in pretraining (Borchert et al., 26 Aug 2025). Third, the method is described as architecture-agnostic because improvements are observed for both encoder-only and decoder-only models (Borchert et al., 26 Aug 2025).

Against PCT, the paper reports average gains of up to 1.84% for Gemma 2 in some settings (Borchert et al., 26 Aug 2025). Against ICL in the $\langle EOS \rangle$0 decoder-only setting, CoLAP exceeds inference-time in-context learning by about +6.41% average for Gemma 2 and about +6.93% average for Mistral (Borchert et al., 26 Aug 2025). The paper identifies this as a key result, arguing that training-time cross-lingual alignment outperforms purely prompt-based inference-time few-shot learning in low-resource multilingual transfer (Borchert et al., 26 Aug 2025).

The dataset-level summaries reported in the paper are also specific. On XNLI, CoLAP improves over FT and PCT across XLM-R, Gemma 2, and Mistral; both XCCL and XRCL help substantially in low-shot settings; and gains remain visible even at $\langle EOS \rangle$1, although relative margins shrink (Borchert et al., 26 Aug 2025). On AmNLI, which the paper characterizes as the most challenging benchmark, CoLAP substantially improves over FT and PCT, including for languages not present in pretrained LLM pretraining (Borchert et al., 26 Aug 2025). On MultiTACRED, the method shows strong improvements over FT, CA, and PCT, with particularly notable benefits for decoder-only models, where ICL is weaker (Borchert et al., 26 Aug 2025).

A plausible implication is that CoLAP’s strongest empirical contribution lies less in absolute architectural novelty than in demonstrating that task-level representation alignment can systematically compensate for the data imbalance inherited from multilingual pretraining. The paper explicitly frames this as evidence that the cross-lingual performance gap can be narrowed efficiently without large target-language datasets (Borchert et al., 26 Aug 2025).

6. Ablations, variants, and interpretive analyses

The paper includes several analyses intended to isolate why CoLAP works (Borchert et al., 26 Aug 2025). The most direct ablation shows that removing the contrastive loss causes a notable performance drop, indicating that contrastive alignment is a key contributor beyond prompting alone (Borchert et al., 26 Aug 2025). It also tests combinations with checkpoint averaging and PCT; the reported takeaway is that contrastive alignment is essential, checkpoint averaging can improve robustness, and combining PCT-style multilingual prompt augmentation with CoLAP is not clearly better than CoLAP alone (Borchert et al., 26 Aug 2025).

XRCL versus XCCL

A central comparison concerns the tradeoff between XRCL and XCCL. The paper reports that XRCL is often slightly stronger because it uses exact translated pairs, while XCCL is nearly as good, usually within less than 1% average performance loss in many settings (Borchert et al., 26 Aug 2025). In very low-resource settings, XCCL can sometimes match or exceed XRCL (Borchert et al., 26 Aug 2025). This matters because XCCL removes the need for parallel translations and therefore reduces data preparation cost (Borchert et al., 26 Aug 2025).

Which layer to align

The paper also analyzes which transformer layer yields the best representations for contrastive learning. For XLM-R, the 10th layer works best for NLI tasks (Borchert et al., 26 Aug 2025). The interpretation given is that early layers capture low-level syntax, middle layers encode more general semantic and task-relevant information, and final layers become more specialized for prediction (Borchert et al., 26 Aug 2025). This supports the claim that contrastive transfer works best when applied to mid-level task representations (Borchert et al., 26 Aug 2025).

Exemplar selection by representation similarity

Another analysis explores choosing few-shot exemplars by representation similarity rather than random sampling. The method computes class prototypes from English representations and selects target examples with either high similarity or low similarity to these prototypes (Borchert et al., 26 Aug 2025). The reported result is that similarity-based exemplar selection improves data efficiency, and with as few as $\langle EOS \rangle$2, this can outperform random $\langle EOS \rangle$3 selection in some settings (Borchert et al., 26 Aug 2025). This suggests that the geometry of the shared task space is itself a usable resource for few-shot adaptation.

These analyses situate CoLAP within a broader interpretive framework. In other contrastive prompting paradigms, the contrastive signal may be carried by positive and negative examples in-context (Gao et al., 2024), by positive and negative medical prompts that are subtracted in attention space (Park et al., 2024), or by prompt-ranking scores that penalize similarity to confounders (Choi et al., 14 May 2025). CoLAP’s distinctive claim is that a comparable contrastive principle can be implemented at the level of multilingual task representations and can materially improve few-shot cross-lingual adaptation (Borchert et al., 26 Aug 2025).

7. Significance, limitations, and relation to adjacent work

The paper characterizes CoLAP as a relatively simple but effective method: use prompts to define the task, then use contrastive learning to align multilingual task representations (Borchert et al., 26 Aug 2025). Its reported significance lies in several points: strong few-shot cross-lingual transfer, better performance than FT, CA, PCT, and ICL in many settings, especially strong gains in low-resource and unseen-language regimes, a practical translation-free variant in XCCL, and evidence that representation alignment at the task level can narrow the multilingual performance gap efficiently (Borchert et al., 26 Aug 2025).

At the same time, the limitations are explicit. CoLAP improves transfer but does not fix the underlying imbalance in pretraining data (Borchert et al., 26 Aug 2025). XRCL still requires translated pairs, introducing a translation or annotation step (Borchert et al., 26 Aug 2025). XCCL removes this translation dependency but is limited to classification-style tasks where class labels are shared across languages (Borchert et al., 26 Aug 2025). The method also assumes access to a source language such as English with enough task data, and the paper notes that further work is needed to combine CoLAP more effectively with other multilingual prompting methods (Borchert et al., 26 Aug 2025).

In the broader landscape, the phrase “contrastive language alignment with prompting” has been used for several distinct methodological patterns. In multilingual NLP, CoLAP denotes prompted classification with cross-lingual contrastive losses over task representations (Borchert et al., 26 Aug 2025). In preference-aligned generation, contrastive in-context learning uses positive and negative examples, sometimes coupled with a self-analysis step, to steer response style, tone, or preference without parameter updates (Gao et al., 2024). In medical visual-language anomaly detection, contrastive language prompting uses positive lesion prompts and negative normality prompts to reduce false positive attention and improve downstream reconstruction-based anomaly detection (Park et al., 2024). In zero-shot object detection, automated prompt refinement based on Contrastive Class Alignment Score (CCAS) ranks candidate prompts by semantic closeness to the target class and distance from confounding classes, improving average precision without additional model training (Choi et al., 14 May 2025).

These approaches are not methodologically identical, but they share a common design principle: prompting becomes more discriminative when the system is given not only a target signal but also an explicit mechanism for suppressing confounders, distractors, or undesired alternatives. In CoLAP proper, that principle is instantiated as cross-lingual representation alignment under prompt-based task conditioning, with English serving as the anchor space for low-resource adaptation (Borchert et al., 26 Aug 2025).