---
title: Probabilistic Cross-Modal Embeddings (PCME++)
url: https://www.emergentmind.com/topics/probabilistic-cross-modal-embeddings-pcme
type: topic
---

# Probabilistic Cross-Modal Embeddings (PCME++)

Searching arXiv for the core PCME++ paper and closely related probabilistic cross-modal embedding work.
arXiv search query: "Probabilistic Cross-Modal Embeddings PCME++ image-text matching"
Probabilistic Cross-Modal Embeddings (PCME++) is a probabilistic image–text matching model for cross-modal retrieval that treats images and captions as distributions rather than deterministic point vectors. It was introduced as an improved form of the earlier Probabilistic Cross-Modal Embedding (PCME), with the specific aim of addressing two limitations of prior probabilistic image–text retrieval: heavy computation from Monte Carlo approximation and loss saturation under abundant false negatives. In PCME++, each image and caption is represented as a diagonal Gaussian embedding, and matching is driven by a closed-form probabilistic distance, pairwise binary matching loss, pseudo-positive relabeling, and mixed sample data augmentation. The method is motivated by the observation that image–caption datasets are inherently ambiguous because correspondences are many-to-many, sparsely annotated, and noisy, so a single deterministic embedding is often a poor fit for the underlying semantics [2305.18171].

## 1. Historical setting and problem formulation

PCME++ was proposed for image–text matching (ITM), typically evaluated as bidirectional retrieval: given an image, retrieve matching captions; given a caption, retrieve matching images. Its starting point is the claim that ITM datasets are not clean one-to-one matching datasets. They exhibit multiplicity or many-to-many correspondence, sparse annotation, and abundant false negatives. In the reported analysis of MS-COCO Caption, 88.2% of caption-to-image positives and 72.1% of image-to-caption positives are missing from the labeled positives. Under such conditions, deterministic dual encoders receive contradictory supervision: two captions may be semantically nearly identical, yet one is labeled positive with an image and the other is treated as a negative. The paper argues that hardest negative mining resolves this contradiction by pushing potential false negatives apart, producing a “twisted” embedding space [2305.18171].

This formulation extends the original PCME, which had already argued that deterministic functions are not sufficiently powerful to capture one-to-many correspondences in cross-modal retrieval. In PCME, images and captions were embedded as Gaussians, and match probabilities were defined by averaging sampled pairwise compatibilities under a soft contrastive objective. That earlier formulation established the core idea that uncertainty can encode semantic multiplicity and retrieval ambiguity, but it relied on Monte Carlo estimation during both training and high-quality inference [2101.05068].

A common misconception is to treat PCME++ primarily as a computational acceleration of PCME. It is computationally more efficient, but the paper also presents it as a change in how certainty is represented and optimized. The central claim is that the proposed distance better encodes sample certainty than alternatives such as Wasserstein distance, because it rewards both mean alignment and variance shrinkage for positive matches.

## 2. Probabilistic representation and closed-form sampled distance

PCME++ represents image and text embeddings as diagonal Gaussians:
$$
\mathbf{Z}_v \sim \mathcal{N}(\mu_v,\sigma_v^2), \qquad \mathbf{Z}_t \sim \mathcal{N}(\mu_t,\sigma_t^2),
$$
where $\mu_v,\mu_t,\sigma_v,\sigma_t \in \mathbb{R}^D$, and $\sigma^2$ denotes elementwise variance. The intended semantics are explicit: if a sample’s match relationships are certain, its variance should be small; if its match relationships are ambiguous, its variance should be large [2305.18171].

The defining contribution of PCME++ is the closed-form sampled distance (CSD), defined as the expected squared Euclidean distance between random samples from the two Gaussian embeddings:
$$
d(\mathbf{Z}_v,\mathbf{Z}_t) = \mathbb{E}_{\mathbf{Z}_v,\mathbf{Z}_t} \left[ \|\mathbf{Z}_v-\mathbf{Z}_t\|_2^2 \right].
$$
Under independent Gaussian assumptions, this yields the closed form
$$
d(\mathbf{Z}_v,\mathbf{Z}_t) = \|\mu_v-\mu_t\|_2^2 + \|\sigma_v^2+\sigma_t^2\|_1.
$$
The first term measures mean separation; the second measures total uncertainty mass. This construction is the key difference from the original PCME, which defined matching probability by Monte Carlo integration of a sigmoid over sampled distances:
$$
p(m \mid x_v, x_t) = \mathbb{E}_{Z_v,Z_t} \left[ \sigma\!\left(-a\|Z_v-Z_t\|_2+b\right) \right].
$$
Because that expectation is not available in closed form, PCME approximated it with sampled pairs, incurring $O(J^2)$ comparisons per image–text pair [2101.05068].

The paper explicitly contrasts CSD with the diagonal 2-Wasserstein distance,
$$
W_2^2 = \|\mu_v-\mu_t\|_2^2 + \|\sigma_v-\sigma_t\|_2^2.
$$
Its argument is that Wasserstein only encourages variances to match each other, not necessarily to shrink. By contrast, CSD makes positive matches closer when variances decrease. A technically important subtlety follows from this design: CSD is not a strict metric because
$$
d(\mathbf{Z},\mathbf{Z}) = 2\|\sigma^2\|_1.
$$
The paper treats this nonzero self-distance as desirable, since it reflects intrinsic uncertainty rather than an error in the geometry.

## 3. Matching loss, regularization, and optimization techniques

Using CSD, PCME++ defines a pairwise probabilistic matching objective as a binary negative log-likelihood:
$$
\mathcal{L}_{\text{match}} = - m_{vt}\log \sigma(-a\,d(\mathbf{Z}_v,\mathbf{Z}_t)+b) - (1-m_{vt})\log \sigma(a\,d(\mathbf{Z}_v,\mathbf{Z}_t)-b),
$$
where $m_{vt}\in\{0,1\}$ is the match indicator, $a,b$ are learnable scalars, and $\sigma(\cdot)$ is the sigmoid. Equivalently, with
$$
\ell_{vt}=-a\,d(\mathbf{Z}_v,\mathbf{Z}_t)+b,
$$
the loss is standard BCE between $\ell_{vt}$ and $m_{vt}$. This pairwise formulation is central: unlike triplet loss or InfoNCE, it treats all image–text pairs in the minibatch independently, which later enables pseudo-positives and soft labels from mixed sample augmentation [2305.18171].

To prevent variance collapse, PCME++ adds a Variational Information Bottleneck regularizer:
$$
\mathcal{L}_{\text{VIB}} =
D_{\mathrm{KL}}\!\big(\mathcal{N}(\mu_v,\sigma_v^2)\,\|\,\mathcal{N}(0,I)\big) +
D_{\mathrm{KL}}\!\big(\mathcal{N}(\mu_t,\sigma_t^2)\,\|\,\mathcal{N}(0,I)\big).
$$
For one modality, the diagonal Gaussian term is
$$
D_{\mathrm{KL}}\big(\mathcal{N}(\mu,\sigma^2)\,\|\,\mathcal{N}(0,I)\big)
= \frac12 \sum_{d=1}^D \left( \mu_d^2 + \sigma_d^2 - \log \sigma_d^2 - 1 \right).
$$
The paper reports that this regularizer improves not only numerical stability but also the correlation between learned uncertainty and retrieval difficulty.

The method’s first optimization refinement is pseudo-positives. For a positive pair $(v,t)$, another text $t'$ is relabeled as pseudo-positive with $v$ if
$$
d(\mathbf{Z}_v,\mathbf{Z}_{t'}) \le d(\mathbf{Z}_v,\mathbf{Z}_t).
$$
The intuition is that if an unlabeled text is already closer than the annotated positive, it may be a missing positive rather than a true negative. Let $\mathcal{L}_{\text{PP}}$ be the matching loss recomputed with pseudo-positive labels.

The second refinement is Mixed Sample Data Augmentation (MSDA). Only images are mixed; text is not. Pair labels are relaxed from binary values to $m_{vt}\in[0,1]$, which is directly compatible with the BCE formulation. In the reported experiments, 25% of minibatch images are mixed, $\lambda \sim \mathrm{Beta}(2,2)$, and Mixup or CutMix is chosen randomly each minibatch.

The full objective is
$$
\mathcal{L} = \mathcal{L}_{\text{match}} + \alpha\,\mathcal{L}_{\text{PP}} + \beta\,\mathcal{L}_{\text{VIB}},
$$
with default settings
$$
\alpha = 0.1, \qquad \beta = 10^{-4}.
$$
The paper also notes a trade-off: pseudo-positives mitigate missing-positive saturation, but under strong noisy correspondence they can themselves be wrong, so $\alpha$ should be reduced or disabled.

## 4. Architecture and implementation

PCME++ uses dual encoders initialized from pretrained CLIP. The image encoder is a Vision Transformer, and the text encoder is a 12-layer, width-512 Transformer following CLIP. Each modality has two output heads: a $\mu$ head for the mean embedding and a $\log \sigma^2$ head for the variance. The model maps each input to
$$
(\mu, \log \sigma^2) \in \mathbb{R}^D \times \mathbb{R}^D.
$$
The $\mu$ vector is $\ell_2$-normalized, while the variance head outputs $\log \sigma^2$, which is exponentiated to recover $\sigma^2$. The implementation duplicates the last transformer layer into separate one-layer transformer blocks for $\mu$ and $\log\sigma^2$; the $\log\sigma^2$ head is randomly initialized, and the $\mu$ head inherits pretrained initialization [2305.18171].

Feature aggregation uses the Generalized Pooling Operator (GPO), following VSE$\infty$. The paper reports that GPO improves both stability and performance. Main COCO-style experiments use 25 epochs, AdamP, batch size 128, initial learning rate $5\times 10^{-4}$, weight decay $10^{-4}$, $a=b=5$ initialization in the BCE logit, visual backbone learning-rate multiplier $0.01$, text backbone learning-rate multiplier $0.1$, layer-wise learning-rate decay $0.7$, visual backbone frozen for 2 epochs, and SizeAugment enabled. The embedding dimension is 1024.

The computational contrast with original PCME is explicit. With $J=8$, 25-epoch training takes 141,694 sec for PCME and 106,311 sec for PCME++, so PCME is about 33% slower. The paper also emphasizes retrieval practicality: because CSD decomposes into a mean-distance term and a variance term, approximate nearest-neighbor deployment becomes easier. The proposed strategy is to build a FAISS index on means $\mu$, retrieve top-$K$ by mean distance, and rerank using CSD with stored variances.

## 5. Benchmarks, empirical behavior, and applications

PCME++ is evaluated on MS-COCO Caption, CxC, and ECCV Caption, along with noisy-correspondence variants where training pairs are randomly corrupted at 20% and 50%. The paper emphasizes that ECCV mAP@R and R-Precision are more informative than plain Recall@K because they better reflect the missing-positive structure of caption retrieval datasets [2305.18171].

At ViT-B/32 scale, PCME++ reports ECCV mAP@R 40.1, R-P 49.7, and COCO RSUM 537.0, compared with PCME at 39.1, 48.9, and 532.0, and VSE$\infty$ at 40.0, 49.5, and 536.5. At ViT-B/16, PCME++ reaches mAP@R 42.1, R-P 51.2, and RSUM 548.0, again above PCME and slightly above strong deterministic baselines. At ViT-L/14, it reports mAP@R 42.1, R-P 50.8, ECCV R@1 88.8, CxC R@1 65.9, and COCO RSUM 554.7. One of the paper’s strongest claims is that deterministic methods become more vulnerable to false negatives as model capacity increases, whereas probabilistic methods scale more gracefully.

Under 50% correspondence noise with ViT-B/32, PCME++ reports ECCV mAP@R 35.7, R-P 45.8, and COCO RSUM 511.0, compared with PCME at 35.2, 45.5, and 508.0, InfoNCE at 33.6, 44.1, and 499.5, and VSE$\infty$ at 18.0, 28.5, and 394.1. This suggests that probabilistic matching is particularly useful when correspondence labels are unreliable.

Ablations isolate the role of each component. With ViT-B/32, the full combination of VIB, pseudo-positives, and MSDA gives RSUM 537.0 and mAP@R 40.1, compared with 535.9 and 38.9 when none are used. In distance ablations, KL divergence fails, JS divergence fails, Wasserstein-2 is very poor, Expected Likelihood Kernel is decent, Bhattacharyya distance is decent, PCME match probability is strong, and the proposed CSD is best overall in this setup. At inference, CSD-based retrieval slightly outperforms mean-only retrieval: RSUM 537.3 versus 536.3.

The paper also reports a side application in zero-shot classification via prompt filtering. On ImageNet, top-1 accuracy increases from 30.43 with a single prompt and 34.22 with all 80 prompts to 41.82 when selecting the best top-$K$ certain prompts per class. The paper notes that this is not a fully clean zero-shot protocol because filtering is searched on the validation set.

## 6. Relation to adjacent probabilistic and uncertainty-aware retrieval methods

PCME++ belongs to a broader literature on uncertainty-aware cross-modal representations, but its combination of probabilistic embeddings, closed-form matching, and pairwise BCE is distinctive. Relative to original PCME, it keeps Gaussian image–text embeddings but replaces Monte Carlo match-probability estimation with a closed-form expected distance and removes the primary sampling bottleneck [2101.05068]. Relative to the Gaussian retrieval retrofit in “Uncertainty-based Cross-Modal Retrieval with Probabilistic Representations,” it shares the premise that cross-modal ambiguity should appear as uncertainty in the embedding, but differs in objective design: that work retains hinge-based triplet ranking and evaluates KL- and Wasserstein-based similarities, whereas PCME++ centers the training procedure on pairwise BCE over CSD [2204.09268].

It also differs from later frozen-backbone methods. ProbVLM adds small post-hoc adapters on top of frozen CLIP or BLIP, uses generalized Gaussian distributions rather than diagonal Gaussian covariance, and keeps retrieval ranking deterministic while using uncertainty for calibration and downstream tasks [2307.00398]. GroVE is a post-hoc GPLVM-based alternative that learns a shared low-dimensional latent space over frozen VLM embeddings and derives Gaussian predictive posteriors from sparse GPs, prioritizing calibration over throughput [2505.05163]. PAU represents a more radical departure: it retains deterministic embeddings and estimates aleatoric uncertainty through cross-modal prototypes, Dempster–Shafer Theory, and Subjective Logic, so its uncertainty is evidential rather than distributional in the latent embedding sense [2309.17093].

A plausible implication is that PCME++ occupies a middle position in this landscape. It is more retrieval-native than post-hoc calibration methods and more computationally deployable than the original PCME, while remaining explicitly probabilistic in the representation itself.

## 7. Limitations, interpretation, and influence

Several limitations are explicit. PCME++ assumes diagonal covariance,
$$
\Sigma = \mathrm{diag}(\sigma^2),
$$
which is computationally efficient but ignores cross-dimensional covariance. Its closed-form distance is specific to Gaussian embeddings; more expressive uncertainty families would generally lose the same analytic simplicity. Pseudo-positives can hurt under severe correspondence noise, and on smaller backbones the gains over strong deterministic baselines are modest rather than dramatic [2305.18171].

The uncertainty interpretation is also specific. In PCME++, large variance is treated as data ambiguity rather than aleatoric sensor noise in the classical sense. The scalar uncertainty proxy is effectively $\|\sigma^2\|_1$, and the paper reports that higher variance correlates with lower COCO R@1. This suggests a semantics of uncertainty tied to many-to-many matchability and annotation incompleteness rather than model ignorance.

Historically, PCME++ has become a reference point for later work that either adapts probabilistic cross-modal embeddings to frozen vision–language models, replaces Gaussian dispersion with evidential uncertainty, or extends probabilistic embeddings to more general multimodal settings. Its main enduring contribution is not only the specific formula
$$
d(\mathbf{Z}_v,\mathbf{Z}_t)=\|\mu_v-\mu_t\|_2^2+\|\sigma_v^2+\sigma_t^2\|_1,
$$
but also the broader claim that retrieval ambiguity should be encoded directly in the representation geometry rather than treated as annotation noise to be ignored.

Source: https://www.emergentmind.com/topics/probabilistic-cross-modal-embeddings-pcme