Contrastive Language-Instance Alignment (CLIA)

• Contrastive Language-Instance Alignment (CLIA) is a method that aligns language observations with corresponding instances (e.g., images, audio, video) in a shared embedding space.
• It utilizes contrastive objectives, such as CLIP-style and InfoNCE losses, to separate non-matching pairs and optimize alignment across different modalities and languages.
• CLIA’s flexible design supports various supervision regimes and efficient parameter updates, enabling improved retrieval performance and cross-modal generalization.

Searching arXiv for the cited CLIA-relevant papers and closely related work. Contrastive Language-Instance Alignment (CLIA) is best understood, in the contemporary literature, as a family of contrastive representation-learning setups in which a language observation is aligned to a corresponding “instance” in a shared embedding space, while non-corresponding instances are separated. The instance may be an image, an object candidate, an audio segment, a video object, a multilingual realization of the same sentence meaning, or another non-linguistic or weakly linguistic anchor. In the cited literature, the term is explicit in the REIR baseline CLARE, but closely related formulations also appear in cross-lingual image-caption tuning, multilingual sentence embedding, video-language segmentation, audio–lyrics alignment, and spoken language understanding (Hao et al., 23 Jun 2025, Krasner et al., 19 May 2025, Zhao et al., 2023).

1. Conceptual scope

CLIA replaces or supplements direct language-to-language supervision with alignment to a shared instance space. In bitext-style multilingual representation learning, a sentence in language $a$ is directly pulled toward its translation in language $b$. In CLIA-style formulations, a sentence in language $a$ is pulled toward an instance, and a sentence in language $b$ is also pulled toward instances drawn from the same semantic universe; the shared instance space therefore acts as a pivot (Krasner et al., 19 May 2025). This can be explicit, as in gallery-wide referring-expression–object alignment, or implicit, as in multilingual image-caption tuning where cross-lingual sentence alignment emerges as a byproduct rather than as a direct objective (Hao et al., 23 Jun 2025, Krasner et al., 19 May 2025).

The notion of “instance” varies by task. In REIR, the aligned visual units are candidate object embeddings extracted from a Deformable-DETR-style branch, and the language unit is a referring expression embedding (Hao et al., 23 Jun 2025). In multilingual sentence embedding, the instance may be a semantic sentence group with multiple translations, so the contrastive objective aligns one anchor to several language-conditioned realizations of the same meaning (Zhao et al., 2023). In audio–lyrics alignment, the aligned units are local text symbols and local audio embeddings, with alignment recovered from a similarity matrix plus monotonic decoding (Durand et al., 2023). In video-language segmentation, the aligned visual units are sentence-conditioned object instances aggregated from masks across frames, contrasted against unreferred objects and background (Liang et al., 2021).

The literature does not present CLIA as a single standardized framework. Some papers are direct instances, some are described as methodological relatives, and some are best viewed as neighboring approaches rather than CLIA proper. This suggests that CLIA is most precise as an umbrella description for contrastive language-to-instance alignment mechanisms rather than as a single canonical algorithm (Hao et al., 23 Jun 2025, Zhao et al., 2023).

2. Objective functions and alignment geometry

The simplest CLIA objectives are CLIP-style or InfoNCE-style batchwise alignment losses. In multilingual image-caption tuning, caption embeddings $E_c$ and image embeddings $E_i$ are projected into a shared $512$-dimensional space and trained with

$S = E_c \cdot E_i^\top * t$

$L(E_i,E_c) = \mathrm{CrossEntropy}(S, I),$

where $t$ is a learned temperature and $b$0 is the identity target over matched image-caption pairs (Krasner et al., 19 May 2025). In that formulation, the objective as written is one-directional: captions are contrasted against images, and in-batch non-matching images serve as negatives.

Other CLIA formulations are symmetric. LilT uses the standard dual-encoder image-to-text and text-to-image contrastive objective

$b$1

with cosine similarity on normalized embeddings and all other items in the batch as negatives (Khan et al., 2023). The trimodal time-series/vision/language study likewise uses a symmetric pairwise InfoNCE loss for each modality pair and sums the three pair losses,

$b$2

over $b$3-normalized embeddings (Yashwante et al., 22 Feb 2026).

A second axis of variation concerns the positive set. Standard single-positive training uses one anchor, one positive, and multiple negatives. MPCL replaces that with a set of multilingual positives,

$b$4

and optimizes

$b$5

thereby updating pairwise similarity distributions into group-wise similarity ones (Zhao et al., 2023). This is explicitly motivated by transitive similarity among translations and by the claim that multilingual sentence groups form coherent semantic clusters.

A third axis concerns target structure. Soft contrastive learning replaces one-hot translation labels with teacher-derived soft labels. Given translation sentence pairs $b$6, the student is trained so that cross-lingual similarities follow monolingual teacher similarities through a bidirectional soft-target cross-entropy,

$b$7

with $b$8 derived from teacher similarities rather than from hard identity labels (Park et al., 2024). Theoretical work goes further by showing that a broad class of contrastive objectives can be reframed as distribution alignment problems over a coupling matrix $b$9,

$a$0

which suggests a more distribution-aware view of CLIA than hard diagonal matching alone (Chen et al., 27 Feb 2025).

3. Supervision regimes and mediating instance types

The central design question in CLIA is what serves as the pivot. Recent work spans several distinct regimes.

Setting	Language side	Instance side
Cross-lingual image-caption tuning (Krasner et al., 19 May 2025)	captions in English, Spanish, Japanese, Hindi, Quechua	images
REIR / CLARE (Hao et al., 23 Jun 2025)	referring expressions	candidate object instances
Audio–lyrics alignment (Durand et al., 2023)	characters or phonemes in context	local audio frames
Video-language segmentation (Liang et al., 2021)	sentence embedding	object instances across frames
GL-CLeF (Qin et al., 2022)	utterance and token representations	multilingual views of the same utterance

In multilingual image-caption alignment, images replace bitext as the pivot. The training data use MS-COCO’s $a$1k English image-caption pairs and Google-Translate captions in Spanish, Japanese, Hindi, and Quechua, producing a $a$2-way parallel caption set over the same images; however, in the “Multilingual” and “Multilingual+Quechua” training sets each image is paired with only one caption language chosen by rotation rather than all captions jointly (Krasner et al., 19 May 2025). The alignment signal is therefore weaker and more indirect than explicit same-image multilingual grouping, but it remains a genuine instance-mediated cross-lingual process.

In GL-CLeF, the pivot is not a non-linguistic modality but an utterance instance with multilingual code-switched views generated from bilingual dictionaries. This produces explicit sentence-level local intent transfer, token-level local slot transfer, and semantic-level global transfer across intent and slot (Qin et al., 2022). A plausible implication is that CLIA need not require a non-linguistic anchor if the training setup still aligns language-conditioned realizations of the same underlying instance.

At finer granularity, audio–lyrics alignment and video-language segmentation show that CLIA is not confined to whole-image or whole-sentence embeddings. Audio–lyrics alignment trains on weak segment-level supervision—“the start and end of lyrical lines in the audio”—and uses a local max-sim loss

$a$3

together with a regression-style contrastive loss over positive and negative symbols (Durand et al., 2023). Video-language segmentation uses sentence-to-instance InfoNCE over object masks aggregated across frames, then strengthens the alignment with Language-relevant Channel Filter and Relative Hard Instance Construction (Liang et al., 2021).

The trimodal time-series/vision/language study adds another supervision lesson: independently pretrained encoders from these modalities exhibit near-orthogonal geometry before explicit coupling, but post-hoc contrastive alignment over frozen backbones can recover useful shared structure, and images can act as effective intermediaries between time series and language (Yashwante et al., 22 Feb 2026).

4. Representative empirical results

The clearest cross-lingual CLIA evidence comes from image-mediated sentence alignment. On Flores-200 bitext retrieval, multilingual image-caption tuning achieved mean accuracy $a$4, compared with $a$5 for the direct Eng-Pivot text-text baseline and $a$6 for raw XLM-R; the Multilingual+Quechua variant reached $a$7 (Krasner et al., 19 May 2025). On XNLI, the same models reached average accuracies $a$8 and $a$9, improving over the untuned XLM-R baseline at $b$0 but remaining below the direct text-text baseline at $b$1 (Krasner et al., 19 May 2025). Particularly notable is the Quechua result: Quechua is outside XLM-R pretraining, yet Quechua-to-English retrieval rose from $b$2 without Quechua captions in tuning to $b$3 after adding Quechua image captions (Krasner et al., 19 May 2025).

MPCL shows that multiple multilingual positives improve retrieval, semantic similarity, classification, and unseen-language transfer over single-positive contrastive training. Representative gains include LaBSE BUCC $b$4, XLM-R STS17 $b$5, XLM-R STS22 $b$6, and XLM-R MTOP domain classification $b$7 (Zhao et al., 2023). The paper also reports stronger excluded-language transfer, for example XLM-R STS17 excluded pairs $b$8 and STS22 excluded pairs $b$9 (Zhao et al., 2023).

Soft contrastive learning shows that preserving graded semantic neighborhoods can outperform hard-label translation ranking. On bilingual Tatoeba, soft contrastive learning achieved $E_c$0 vs $E_c$1 on en-ko, $E_c$2 vs $E_c$3 on en-fr, and $E_c$4 vs $E_c$5 on en-ja; on BUCC it achieved $E_c$6 vs $E_c$7 on en-fr and $E_c$8 vs $E_c$9 on en-ru (Park et al., 2024). In multilingual training, mE5 + soft contrastive reached Tatoeba $E_i$0, slightly above LaBSE at $E_i$1, while on FLORES-200 xSIM error it improved pretrained mE5 from $E_i$2 to $E_i$3, though LaBSE remained best at $E_i$4 (Park et al., 2024).

The REIR baseline CLARE provides the paper in which CLIA is explicit. Its ablation shows that removing CLIA reduces BR@10 from $E_i$5 to $E_i$6; a mini-batch-only CLIA reaches $E_i$7, and the full method reaches BR@1 $E_i$8, BR@5 $E_i$9, BR@10 $512$0 (Hao et al., 23 Jun 2025). This is direct evidence that gallery-wide language-instance contrastive supervision, not only intra-image grounding, is central to retrieval across images.

At local alignment granularity, audio–lyrics alignment yields an average absolute error below $512$1 seconds on Jamendo and, with contextual character modeling plus language conditioning, reaches all-language AAE $512$2 and PCO $512$3 on JamendoLyrics Multi-Lang (Durand et al., 2023). Video-language segmentation shows similar value for explicit sentence-to-instance CLIA: adding cross-modal contrastive learning to an implicit-fusion baseline raises A2D mAP from $512$4 to $512$5, and adding both hard-mining strategies reaches $512$6 (Liang et al., 2021).

5. Architectural patterns and efficiency regimes

A recurring CLIA pattern is to start from strong pretrained unimodal backbones and learn a relatively small alignment interface. In multilingual image-caption tuning, the text encoder is XLM-Roberta-Large, the vision encoder is ViT-Base-patch16-224-in21k, both modalities are projected by linear layers to a shared $512$7-dimensional space, token-level text representations are mean-pooled, and the projection layers are warmed up before the encoders are thawed halfway through the first epoch (Krasner et al., 19 May 2025). In the trimodal frozen-backbone setting, all encoders remain frozen and a shared projection-head template with LayerNorm, GELU, Dropout, and output dimension $512$8 is trained post hoc (Yashwante et al., 22 Feb 2026).

Parameter-efficient CLIA is also viable. LilT shows that contrastive vision-language alignment can be achieved by minimally updating already trained encoders: $512$9 trainable parameters can match full-model CLIP-style training, and $S = E_c \cdot E_i^\top * t$0 can reach about $S = E_c \cdot E_i^\top * t$1 of full retrieval performance while matching classification (Khan et al., 2023). With bert-base + ViT-B/16, LilT$S = E_c \cdot E_i^\top * t$2 at $S = E_c \cdot E_i^\top * t$3 trainable parameters reaches retrieval close to or slightly above full CLIP depending on regime, whereas LilT$S = E_c \cdot E_i^\top * t$4 and LilT$S = E_c \cdot E_i^\top * t$5 at $S = E_c \cdot E_i^\top * t$6 and $S = E_c \cdot E_i^\top * t$7 give strong but incomplete retrieval (Khan et al., 2023). In multilingual zero-shot retrieval after English-only alignment, LilT$S = E_c \cdot E_i^\top * t$8 beats full CLIP on $S = E_c \cdot E_i^\top * t$9 tasks with average $L(E_i,E_c) = \mathrm{CrossEntropy}(S, I),$0 absolute improvement, suggesting that constrained updates better preserve language knowledge (Khan et al., 2023).

Some systems operate at instance scale rather than whole-image scale. CLARE uses a dual-stream design: a SigLIP vision encoder, a Deformable-DETR-based object extractor with $L(E_i,E_c) = \mathrm{CrossEntropy}(S, I),$1 object queries, and a text branch refined by MORE, where one shared expert and routed experts produce the final query embedding $L(E_i,E_c) = \mathrm{CrossEntropy}(S, I),$2 that is compared by dot product with candidate object features $L(E_i,E_c) = \mathrm{CrossEntropy}(S, I),$3 (Hao et al., 23 Jun 2025). This design turns CLIA into gallery-scale instance retrieval with localization rather than image retrieval alone.

Other systems show that CLIA may need auxiliary structure beyond pure contrastive matching. Audio–lyrics alignment separates local similarity learning from structured monotonic decoding (Durand et al., 2023). CG-VLM uses pooled patch-derived image descriptors and token embeddings, but because there is no explicit patch-token grounding in standard caption datasets, it combines a contrastive alignment loss with a generative captioning loss, and the combined objective outperforms either alone (Liu et al., 2023). A neighboring practical lesson is that when local correspondence is unavailable, weak pooled contrastive signals are often stabilized by additional supervision.

6. Misconceptions, limitations, and related directions

A common misconception is that CLIA is equivalent to global image-text retrieval. The literature is broader. Some methods align whole captions to whole images; others align referring expressions to candidate objects, sentences to masked instances across video frames, or contextual characters to local audio embeddings (Hao et al., 23 Jun 2025, Liang et al., 2021, Durand et al., 2023). Conversely, not every alignment method with a language-guided objective is CLIA in the strict contrastive sense. The sign-language method summarized in the cited literature is better described as denoising-diffusion alignment rather than a contrastive language-instance alignment method, even though it targets a closely related sign–gloss correspondence problem (Guo et al., 2023).

Another misconception is that CLIA always requires direct language-language supervision. The multilingual image-caption study shows the opposite: multilingual visual grounding can induce cross-lingual sentence alignment without bitext, and can even post hoc integrate a language unseen in language-model pretraining (Krasner et al., 19 May 2025). At the same time, image-caption grounding is weaker and noisier than translation equivalence, and the best image-mediated results still trail direct bitext supervision on some benchmarks (Krasner et al., 19 May 2025).

A third misconception is that a single-positive hard-label formulation is intrinsic to CLIA. Multi-positive translation-group learning, soft-label distribution matching, and optimal-transport reinterpretations all argue otherwise (Zhao et al., 2023, Park et al., 2024, Chen et al., 27 Feb 2025). Theoretical work suggests that what appears as pairwise contrastive learning is “secretly” a distribution alignment problem over a coupling matrix, which opens the door to soft many-to-many matches, structured target plans, and unbalanced alignment for noisy views (Chen et al., 27 Feb 2025). This suggests a broader design space for CLIA than diagonal-positive InfoNCE.

The main empirical limitations recur across domains. Global alignment does not guarantee local neighborhood agreement; the trimodal study reports that good cosine margins and Procrustes scores can coexist with relatively weak mutual $L(E_i,E_c) = \mathrm{CrossEntropy}(S, I),$4-NN overlap (Yashwante et al., 22 Feb 2026). Weakly grounded text can cap performance: MIMIC reports, which describe diagnoses rather than explicit waveform structure, yield much weaker TS–TXT and IMG–TXT alignment than direct structural captions (Yashwante et al., 22 Feb 2026). In referring-expression retrieval, CLARE still struggles on complex relations and cluttered scenes with multiple similar objects (Hao et al., 23 Jun 2025). In multimodal LLM alignment, contrastive-only pooled patch/token alignment underperforms generative-only training, and the combined objective is best, indicating that a weak approximation to fine-grained CLIA does not yet substitute for richer supervision (Liu et al., 2023).

A plausible implication, stated explicitly in the multilingual image-caption study, is that the CLIA logic is not image-specific: any modality in which semantically corresponding language in multiple languages can be tied to the same instance—audio, video, speech segments, documents with metadata, grounded sensor events, or structured knowledge entries—could support the same alignment principle (Krasner et al., 19 May 2025). The cited literature therefore presents CLIA less as a narrow technique than as a general representational strategy: use contrastive learning to bind language to shared instance structure, then exploit that shared geometry for retrieval, transfer, grounding, or cross-lingual generalization.