Cross-Modal Deep Metric Learning

• Cross-modal deep metric learning comprises methods that embed heterogeneous data like images, text, and audio into shared spaces for direct comparison and retrieval.
• It employs various architectures including shared continuous, discrete, and class-center embeddings combined with tailored loss functions to optimize similarity across modalities.
• Recent advances incorporate distribution-aware regularization and plug-in similarity modules to enhance alignment and retrieval performance on standard benchmarks.

Cross-modal deep metric learning denotes a family of methods that learn modality-specific mappings, shared latent spaces, or explicit matching functions so that semantically corresponding samples from heterogeneous modalities become comparable for retrieval, matching, verification, or related downstream tasks. In the literature, this includes image-text, audio-visual, audio-text, and compositional image-plus-text retrieval; it also includes both continuous shared spaces and discrete Hamming spaces, as well as architectures that replace fixed Euclidean or cosine comparison with learned similarity operators (Huang et al., 2017, Qi et al., 2017, Jiang et al., 2016, Diao et al., 2024).

1. Historical emergence and problem formulations

An early formulation learns a direct bilinear cross-modal similarity rather than a within-modality Mahalanobis distance. "Cross-Modal Similarity Learning: A Low Rank Bilinear Formulation" defines $S_M(x_i,z_j)=x_i^\top M z_j$ for heterogeneous image and text features, regularizes $M$ with the nuclear norm, and optimizes a convex pairwise logistic objective with accelerated proximal gradient and singular-value thresholding. Its central premise is that image and text often have different dimensionalities, so a rectangular bilinear similarity is more natural than a same-space metric; the low-rank structure then yields an implicit shared latent space (Kang et al., 2014).

A second early trajectory is discrete. "Deep Cross-Modal Hashing" learns two modality-specific functions, $f(x_i;\theta_x)$ and $g(y_j;\theta_y)$, couples them through a cross-modal similarity matrix $S$, and constrains the learned representations to a shared binary code matrix $B\in\{-1,+1\}^{c\times n}$. This treats cross-modal retrieval as supervised similarity-preserving representation learning, but in a Hamming space rather than a continuous embedding space (Jiang et al., 2016).

By 2017, two influential image-text systems broadened the scope of cross-modal deep metric learning. "Cross-modal Deep Metric Learning with Multi-task Regularization" adds a semi-supervised contrastive loss and a quadruplet ranking loss, explicitly targeting relative similarity ranking and unlabeled paired data. "Cross-media Similarity Metric Learning with Unified Deep Networks" combines contrastive pretraining, double triplet fine-tuning, and a learned metric network on top of a shared embedding, emphasizing that cross-modal retrieval needs both shared representation learning and a learned similarity function beyond fixed Euclidean distance (Huang et al., 2017, Qi et al., 2017).

Subsequent work diversified both modalities and tasks. Audio-visual retrieval, audio-text retrieval, open-vocabulary keyword spotting, cross-modal biometric matching, compositional image retrieval, and multimodal fake-news detection all adopt cross-modal metric learning machinery, but they do so with different granularities, supervision signals, and end goals. This suggests that the field is less a single architecture family than a set of recurrent design choices around shared representations, similarity functions, and alignment constraints (Nawaz et al., 2019, Mei et al., 2022, Jung et al., 22 May 2025, Choi et al., 21 Jan 2025).

2. Representation spaces and architectural archetypes

A recurrent structural pattern is modality-specific encoding followed by alignment in a common space, but the literature realizes that pattern in several distinct ways.

Archetype	Representative papers	Characteristic
Shared continuous embedding	(Huang et al., 2017, Qi et al., 2017, Mei et al., 2022)	Two pathways map modalities into a common latent space
Discrete shared embedding	(Jiang et al., 2016)	Shared semantic Hamming space
Shared class-center space	(Nawaz et al., 2019)	Face and voice pulled to common class centers
Pair- or domain-conditioned regularization	(Ren et al., 2020, Zeng et al., 2024)	Pair-specific domains or anchor-specific proxies
Plug-in similarity module	(Diao et al., 2024)	Replace cosine with structured bilinear similarity
Label-recast paired-free space	(Pu et al., 10 Jan 2025)	Labels mapped back into common space via generalized inverse

In continuous shared-space systems, image and text are typically processed by separate encoders and projected into a same-dimensional embedding. CDMLMR uses modality-specific DBNs, a Bimodal Autoencoder for shallow shared representations, and then two fully connected metric pathways that output $Q^{(i)}=f(S^{(i)})$ and $Q^{(t)}=g(S^{(t)})$ (Huang et al., 2017). UNCSM likewise uses separate image and text pathways to produce $M^{(i)}=f_i(X^{(i)})$ and $M^{(t)}=f_t(X^{(t)})$, but then adds a second-stage metric network over concatenated embeddings rather than stopping at the common space (Qi et al., 2017).

Other systems alter the notion of what is shared. DCMH aligns image and text by enforcing $M$0, so the common space is explicitly binary (Jiang et al., 2016). SSNet instead uses one InceptionResNet-V1 backbone for both face images and voice spectrograms, then pulls both modalities toward the same identity center in a 128-dimensional latent space (Nawaz et al., 2019). DRCL trains modalities independently, selects a semantic prior $M$1, and then recasts labels into modality-invariant targets $M$2, turning labels themselves into anchors in the common embedding space (Pu et al., 10 Jan 2025).

A different architectural line treats similarity computation as the bottleneck. GSSF keeps the host encoders intact but replaces cosine with a structured bilinear family $M$3, with diagonal and block-diagonal masks to model channel relevancy and localized cross-channel dependency. The method is explicitly intended as a plug-in metric for dual encoders, local interaction models, attention, alignment matrices, knowledge distillation, and parameter-efficient tuning (Diao et al., 2024).

3. Objective functions and learned similarity

The literature uses several objective families, often within the same system. A common baseline is supervised pairwise similarity preservation. LRBS uses pairwise logistic loss over the bilinear score $M$4 (Kang et al., 2014). DCMH uses an inner-product-based logistic likelihood with $M$5, supplemented by quantization and bit-balance terms so that the final retrieval space is discrete (Jiang et al., 2016).

Pairwise margin objectives remain central. CDMLMR defines a semi-supervised contrastive loss over labeled pairs via a label matrix $M$6 and over unlabeled pairs via an online $M$7-NN graph $M$8, with Euclidean squared distances in the loss and cosine distance at retrieval time. It then adds a quadruplet ranking loss,

$M$9

to preserve bidirectional relative ranking constraints (Huang et al., 2017). UNCSM uses contrastive pretraining, then double triplet similarity losses with one branch anchored on images and the other on text, and finally trains a metric network with cross-entropy over matched versus unmatched image-text pairs (Qi et al., 2017).

Proxy- and center-based formulations provide alternatives to explicit pair or triplet mining. SSNet minimizes within-class scatter around shared face-voice centers while also training a softmax classifier, thereby eliminating pairwise or triplet supervision in favor of class-center supervision (Nawaz et al., 2019). CroMe applies a proxy-anchor-style loss separately to MAE image, BLIP2 image, BERT text, BLIP2 text, and BLIP2 image-text embeddings, with the explicit aim of capturing intra-modality relationships before cross-modal fusion (Choi et al., 21 Jan 2025).

Several works study the loss itself as the primary object. "Universal Weighting Metric Learning for Cross-Modal Matching" rewrites cross-modal losses as weighting schemes over positive and negative similarities, introduces informative negative mining $f(x_i;\theta_x)$0, and defines polynomial weighting functions $f(x_i;\theta_x)$1 and $f(x_i;\theta_x)$2 so that easy positives and easy negatives contribute less strongly (Wei et al., 2020). "On Metric Learning for Audio-Text Cross-Modal Retrieval" then compares triplet-sum, triplet-max, triplet-weighted, and bidirectional NT-Xent, finding that NT-Xent is stable across AudioCaps and Clotho and often outperforms triplet-style objectives (Mei et al., 2022).

Fine-grained audio-text alignment pushes metric learning below the utterance level. In open-vocabulary keyword spotting, cross-attention aligns acoustic and text phoneme sequences, Asymmetric Proxy loss structures phoneme-level embeddings, AdaMS makes $f(x_i;\theta_x)$3, $f(x_i;\theta_x)$4, and $f(x_i;\theta_x)$5 phoneme-class-specific, and utterance-level Relational Proxy loss remains active in parallel (Jung et al., 22 May 2025). This suggests that in some settings the metric space is most effective when global and local objectives coexist.

4. Beyond pointwise distance: alignment, distribution, and structure

A notable shift in the literature is from pure pointwise closeness toward mechanisms that also regularize distributions, internal dependencies, or local neighborhoods. TIRG-DIM argues that fusion-centric retrieval models do not adequately bridge the modality gap between image, text, and fusion features, and therefore introduces Deep InfoMax-style objectives: ITDIM maximizes mutual information between text and desired-image representations, while IFDIM maximizes mutual information between fusion and desired-image representations, both with global and local components (Gu et al., 2021).

ADDR makes a related but distinct argument: local region-word alignment and triplet ranking remain limited if they ignore the characteristics of hidden-space feature distributions. It therefore attaches a pair-specific discriminator $f(x_i;\theta_x)$6 to each matched image-text pair, adversarially aligns image and text features within that pair, and then regularizes the resulting pair-conditioned domains so that they remain discriminative relative to hard negatives (Ren et al., 2020). A plausible implication is that cross-modal retrieval quality depends not only on score inequalities such as $f(x_i;\theta_x)$7, but also on the geometry of the hidden distributions that produce those scores.

Graph- and neighborhood-based methods push this further. AADML builds within-modality $f(x_i;\theta_x)$8-NN correlation graphs, uses the paired sample from the opposite modality as query in multi-head attention, and constructs an anchor-aware proxy

$f(x_i;\theta_x)$9

which then replaces isolated anchors inside triplet or contrastive losses. The paper frames this as a response to “missing embeddings,” meaning sparse observations of the underlying semantic manifold (Zeng et al., 2024). Progressive self-distillation for audio-visual embedding learning likewise argues that hard labels underutilize latent audio-visual distributions, and therefore derives soft alignment matrices from batchwise cross-modal similarities, converts them into pseudo-relational supervision, and progressively increases the share of batch elements trained from those pseudo-alignments rather than ground-truth labels (Zeng et al., 16 Jan 2025).

Other work addresses structure directly in the similarity operator. GSSF’s diagonal form learns per-channel weights, while its block-diagonal form permits localized cross-channel mixing within blocks and suppresses global dense interactions. The paper emphasizes that the metric itself can be too weak or too expensive, and that structured sparsity offers a middle ground between cosine and dense bilinear matching (Diao et al., 2024). Taken together, these works suggest that cross-modal deep metric learning has broadened from “learn an embedding and measure distance” to “learn an embedding, a comparison rule, and a structural prior over how modalities should interact.”

5. Optimization, benchmarks, and empirical regularities

Across the cited works, image-text retrieval is evaluated on Wikipedia, NUS-WIDE, NUS-WIDE-10k, Pascal Sentences, Flickr30K, MSCOCO, Fashion200K, MIT-States, CSS, and INRIA-Websearch; audio-visual retrieval on VEGAS, AVE, and VoxCeleb; audio-text retrieval on AudioCaps, Clotho, WSJ, and LibriPhrase. Reported metrics include MAP, Recall@K, rSum, AP, EER, and AUC (Huang et al., 2017, Qi et al., 2017, Diao et al., 2024, Mei et al., 2022, Jung et al., 22 May 2025, Nawaz et al., 2019, Pu et al., 10 Jan 2025, Gu et al., 2021).

Modality/task family	Representative datasets	Common metrics
Image-text retrieval	Wikipedia, NUS-WIDE, Flickr30K, MSCOCO	MAP, Recall@K, rSum
Audio-visual retrieval	VEGAS, AVE, VoxCeleb	MAP, Precision-scope@K, AUC, EER
Audio-text retrieval	AudioCaps, Clotho, WSJ, LibriPhrase	Recall@K, AP, AUC, EER
Compositional image retrieval	Fashion200K, MIT-States, CSS	Recall@K
Paired-free multi-modal retrieval	XMedia, XMediaNet	MAP

Several empirical patterns recur. CDMLMR reports gains over prior shared-space and deep baselines on Wikipedia and NUS-WIDE-10k, and its ablations show that semi-supervised contrastive loss, quadruplet ranking loss, and their combination each improve over the base network (Huang et al., 2017). UNCSM shows that replacing cosine distance with its metric network improves average MAP on Wikipedia, NUS-WIDE, NUS-WIDE-10k, and Pascal Sentences, while contrastive pretraining also helps across all four datasets (Qi et al., 2017).

Later plug-in or add-on methods often improve strong backbones rather than replacing them. GSSF lifts SCAN, BFAN, NAAF, ESA, and other image-text systems on Flickr30K and MSCOCO, with especially large gains for interaction-heavy models; its Diag and B-Diag variants also add little runtime overhead relative to cosine (Diao et al., 2024). ADDR improves SCAN, VSRN, and BFAN on MS-COCO and Flickr30K, with larger changes at $g(y_j;\theta_y)$0 than at $g(y_j;\theta_y)$1 or $g(y_j;\theta_y)$2, which is consistent with better hard-negative disambiguation (Ren et al., 2020).

In audio-centered settings, AADML reports large improvements by replacing isolated anchors with anchor-aware proxies on VEGAS and especially AVE (Zeng et al., 2024). Audio-text retrieval experiments show that NT-Xent is more stable than triplet-max and triplet-weighted losses across datasets, frozen versus fine-tuned encoders, and batch sizes (Mei et al., 2022). Open-vocabulary keyword spotting then combines phoneme-level AsyP with AdaMS, utterance-level relational proxy loss, and modality adversarial learning; on WSJ and LibriPhrase, the full system improves over RP-loss and prior phoneme-level baselines (Jung et al., 22 May 2025).

Optimization protocols vary accordingly. LRBS uses APG with singular-value thresholding (Kang et al., 2014), DCMH alternates SGD/backprop updates with exact discrete sign updates for $g(y_j;\theta_y)$3 (Jiang et al., 2016), CDMLMR and UNCSM rely on SGD in two-pathway networks (Huang et al., 2017, Qi et al., 2017), and several later systems use Adam or AdamW with in-batch negatives, adaptive margins, or alternating discriminator-generator phases (Zeng et al., 2024, Ren et al., 2020, Jung et al., 22 May 2025). This suggests that the “metric” in cross-modal deep metric learning is inseparable from optimization design.

6. Conceptual boundaries, misconceptions, and open directions

A recurring misconception is that all methods in this area learn a strict metric in the mathematical sense. Several papers explicitly learn bilinear similarities, Softmax matching heads, or structured scores that are not constrained to be symmetric, positive semidefinite, or triangle-inequality preserving. LRBS describes $g(y_j;\theta_y)$4 as a similarity function rather than a proper metric, UNCSM notes that its learned score is a discriminative Softmax output rather than a true metric, and GSSF states that it does not impose positive semidefiniteness because the function is used as a similarity score (Kang et al., 2014, Qi et al., 2017, Diao et al., 2024).

A second boundary concerns supervision. Much of the literature assumes paired multimodal samples and class labels: CDMLMR uses labeled and unlabeled paired image-text data (Huang et al., 2017), UNCSM constructs positive and negative pairs from labels (Qi et al., 2017), AADML uses same-category graph edges (Zeng et al., 2024), and SSNet relies on identity supervision (Nawaz et al., 2019). Paired-free work such as DRCL replaces pair supervision with label-guided reversible recasting, but still depends on semantic labels and pre-extracted features (Pu et al., 10 Jan 2025). This suggests that “paired-free” in this literature often means “free of exact cross-modal pair training,” not “unsupervised.”

Several limitations are repeatedly acknowledged. CDMLMR does not report sensitivity analyses for margins, $g(y_j;\theta_y)$5-NN graph construction, or embedding size, and its unlabeled graph is built from shallow representations (Huang et al., 2017). UNCSM is not end-to-end and uses handcrafted inputs (Qi et al., 2017). GSSF fixes the sparsity mask rather than learning it, and the block size remains empirical (Diao et al., 2024). CroMe is classification-oriented rather than retrieval-oriented, applies proxy-anchor loss mainly within each stream, and leaves some proxy details ambiguous (Choi et al., 21 Jan 2025). ADDR introduces one discriminator per training pair, creating memory and training-complexity overhead (Ren et al., 2020).

A further caution is terminological. One 2025 paper claims “Cross-Modal Deep Metric Learning” for time-series anomaly detection in its title and abstract, but the detailed synthesis reports that its body instead develops an LSTM-based realized-volatility and EVT Value-at-Risk model; the requested triplet-, center-, and vMF-based CMDML machinery is not actually instantiated in the main text (Li et al., 16 Sep 2025). This suggests that the label “cross-modal deep metric learning” can sometimes be applied more loosely than the underlying formulation warrants.

Open directions are explicit in several papers. GSSF points to jointly learning $g(y_j;\theta_y)$6 and the structural mask $g(y_j;\theta_y)$7 (Diao et al., 2024). DRCL raises the possibility of integrating reversible semantic recasting with stronger instance-level objectives (Pu et al., 10 Jan 2025). Audio-text and audio-visual studies indicate that loss design, adversarial invariance, and local alignment remain open optimization variables rather than settled choices (Mei et al., 2022, Jung et al., 22 May 2025). More broadly, the literature suggests an ongoing shift from simple shared-space distance minimization toward structured comparison operators, distribution-aware regularization, and multi-granular alignment mechanisms that explicitly manage heterogeneity rather than assuming a single embedding alone will remove it.