Customized Similarity-Based Representation

• Customized similarity-based representation is an approach that adapts similarity metrics based on task conditions and dynamic criteria rather than using a fixed, universal metric.
• It employs adaptive neighbor bootstrapping, multi-metric contrastive learning, and explicit criterion conditioning to tailor embeddings and similarity computations for diverse applications.
• This approach enhances representational robustness, improves transfer performance across domains, and mitigates limitations inherent in universal embedding methods.

Customized similarity-based representation denotes a family of representation-learning and similarity-scoring methods in which the relation “similar” is not treated as a fixed, universal primitive, but is explicitly designed, conditioned, or adapted to the task, modality, supervision regime, or training dynamics. In the literature, this appears as adaptive neighbor bootstrapping in self-distillation, multi-metric contrastive learning with uncertainty weighting, criterion-conditioned projections built from LLM-generated semantic descriptors, expert-guided prototype scoring in vision-language systems, and task-aware similarity structures for regions, sets, graphs, recommendation, binary code, and speech (Lebailly et al., 2023, Mu et al., 2023, Safdar et al., 27 Aug 2025, Liu et al., 6 Oct 2025).

1. Conceptual basis

A standard starting point is self-supervised or contrastive representation learning with a fixed similarity relation. In cross-view self-supervision, positives are typically two augmentations of the same image, and contrastive learning optimizes an InfoNCE objective of the form

$$\mathcal{L}_{\text{contra}} = -\log\left(\frac{\exp(s^+/\tau)}{\exp(s^+/\tau)+\sum_{s^-}\exp(s^-/\tau)}\right),$$

with

$$s^+ = f(x)^\top f(x^+), \qquad s^- = f(x)^\top f(x^-).$$

Self-distillation instead uses only positives, with asymmetry between encoders, for example

$$\mathcal{L}_{\text{distil}} = d\big(f(x), f'(x^+)\big).$$

This fixed “same image, different view” notion is effective, but it is only one possible definition of similarity (Lebailly et al., 2023).

A parallel limitation appears in supervised contrastive learning. Standard SupCon defines positives solely from one label space, even though many datasets admit several meaningful similarity relations simultaneously: shoes may be similar by category, closure, or gender; medical images by disease label, severity, or anatomical region; disaster images by damage severity, disaster type, humanitarian category, or informativeness. This means a single learned embedding can collapse alternative semantic organizations that matter downstream (Mu et al., 2023).

The same problem appears in foundation-model reuse. Universal embeddings often emphasize dominant semantics, while customized tasks require different semantics. In animal habitat analysis, for example, universal embeddings may privilege category semantics while a downstream task prioritizes scene-related features; in industrial microstructure qualification, generic CLIP or FLAVA embeddings do not directly encode notions such as “fusion line”, “heat-affected zone (HAZ) depth ≤ 1 mm”, or “uniform carbide distribution”; in image-text matching, binary annotations compress an underlying continuous similarity space and introduce false negatives (Liu et al., 6 Oct 2025, Safdar et al., 27 Aug 2025, Wei et al., 29 May 2026).

This suggests that customized similarity-based representation is best understood as a shift from a universal metric to a task-conditioned similarity operator: the representation is learned or transformed so that distances, neighbors, or scores reflect the criterion actually required by the application.

2. Mechanisms for defining and adapting similarity

One line of work customizes similarity by adapting it online during training. In adaptive similarity bootstrapping, an encoder maintains a cache $Z \in \mathbb{R}^{N \times d}$ with one latent vector per dataset image. For image $x_i$, with current representation $z_i=f(t(x_i))$, raw similarity to image $x_j$ is

$m_i(j)=z_i^\top Z_j,$

which is converted into a distribution

$$s_i(j)=\frac{\exp(m_i(j)/\tau)}{\sum_k \exp(m_i(k)/\tau)}, \qquad p(x_j\mid x_i)=s_i(j).$$

The method then uses a windowed similarity estimate over recent epochs and trusts bootstrapped neighbors only when the most probable neighbor of $x_i$ is $$s^+ = f(x)^\top f(x^+), \qquad s^- = f(x)^\top f(x^-).$$0 itself. If this self-consistency test fails, training falls back to the standard same-image positive. Here, similarity is customized by temperature, temporal averaging, and a latent-quality gate rather than being fixed once and for all (Lebailly et al., 2023).

A second line of work customizes similarity by explicitly modeling several similarity relations at once. In Multi-Similarity Contrastive Learning, each example carries multiple categorical attributes

$$s^+ = f(x)^\top f(x^+), \qquad s^- = f(x)^\top f(x^-).$$1

a shared encoder produces $$s^+ = f(x)^\top f(x^+), \qquad s^- = f(x)^\top f(x^-).$$2, and each metric $$s^+ = f(x)^\top f(x^+), \qquad s^- = f(x)^\top f(x^-).$$3 has its own projection head

$$s^+ = f(x)^\top f(x^+), \qquad s^- = f(x)^\top f(x^-).$$4

For each metric,

$$s^+ = f(x)^\top f(x^+), \qquad s^- = f(x)^\top f(x^-).$$5

and the per-metric supervised contrastive loss is

$$s^+ = f(x)^\top f(x^+), \qquad s^- = f(x)^\top f(x^-).$$6

The full objective learns uncertainty parameters $$s^+ = f(x)^\top f(x^+), \qquad s^- = f(x)^\top f(x^-).$$7 and weights each similarity metric as

$$s^+ = f(x)^\top f(x^+), \qquad s^- = f(x)^\top f(x^-).$$8

Thus the representation is jointly shaped by multiple notions of similarity, with noisy metrics down-weighted automatically (Mu et al., 2023).

A third mechanism is explicit criterion conditioning. Conditional Representation Learning starts from a user-specified criterion $$s^+ = f(x)^\top f(x^+), \qquad s^- = f(x)^\top f(x^-).$$9, uses an LLM to generate descriptive texts

$$\mathcal{L}_{\text{distil}} = d\big(f(x), f'(x^+)\big).$$0

encodes them with a VLM text encoder to obtain a semantic basis

$$\mathcal{L}_{\text{distil}} = d\big(f(x), f'(x^+)\big).$$1

and projects image embeddings $$\mathcal{L}_{\text{distil}} = d\big(f(x), f'(x^+)\big).$$2 into a conditional feature space by

$$\mathcal{L}_{\text{distil}} = d\big(f(x), f'(x^+)\big).$$3

Each coordinate of $$\mathcal{L}_{\text{distil}} = d\big(f(x), f'(x^+)\big).$$4 is an image–descriptor similarity score, so the induced representation is explicitly aligned with the criterion rather than with dominant universal semantics (Liu et al., 6 Oct 2025).

A fourth mechanism is probabilistic similarity modeling. VACSR constructs a pairwise similarity vector

$$\mathcal{L}_{\text{distil}} = d\big(f(x), f'(x^+)\big).$$5

and feeds it into a variational adapter that models latent cross-modal similarity with a Gaussian mixture posterior. The objective is an ELBO,

$$\mathcal{L}_{\text{distil}} = d\big(f(x), f'(x^+)\big).$$6

with variance-optimization terms that increase uncertainty on incompatible or noisy binary labels. In this formulation, similarity is not a point score but a latent distribution with calibrated uncertainty (Wei et al., 29 May 2026).

3. Spatial, topological, and set-theoretic formulations

Customization is not limited to pair sampling or conditioning; it can also be realized by changing the structural object over which similarity is defined. ReSim replaces image-level similarity with region-level similarity. Given two augmented views, it computes aligned overlapping windows $$\mathcal{L}_{\text{distil}} = d\big(f(x), f'(x^+)\big).$$7 and $$\mathcal{L}_{\text{distil}} = d\big(f(x), f'(x^+)\big).$$8, extracts region features by Precise RoI Pooling, and optimizes a region contrastive loss

$$\mathcal{L}_{\text{distil}} = d\big(f(x), f'(x^+)\big).$$9

combined with a global image-level objective

$Z \in \mathbb{R}^{N \times d}$0

Here the representation is customized to localization: corresponding regions, not just whole images, are forced to be similar across views (Xiao et al., 2021).

Set2Box customizes similarity for set-valued data by representing each set as an axis-aligned box $Z \in \mathbb{R}^{N \times d}$1 in latent space. Volumes approximate set cardinalities,

$Z \in \mathbb{R}^{N \times d}$2

so a single learned representation supports multiple set similarities, including Jaccard, Dice, cosine, and overlap coefficient, using only box volumes and intersections. The representation is therefore customized to overlap structure rather than to a single downstream metric (Lee et al., 2022).

Similarity Learning via Kernel Preserving Embedding customizes similarity by learning a sample-wise representation $Z \in \mathbb{R}^{N \times d}$3 that preserves a chosen kernel matrix $Z \in \mathbb{R}^{N \times d}$4. Instead of reconstructing the data matrix $Z \in \mathbb{R}^{N \times d}$5, it solves

$Z \in \mathbb{R}^{N \times d}$6

The choice of kernel $Z \in \mathbb{R}^{N \times d}$7 defines the similarity notion, while the regularizer $Z \in \mathbb{R}^{N \times d}$8 determines whether the learned representation emphasizes low-rank global structure or sparse local neighborhoods (Kang et al., 2019).

These formulations share a common pattern: customized similarity is realized by redefining the primitive object whose geometry must be preserved—regions, sets, or kernel relations—rather than by merely changing a downstream classifier.

4. Domain-specific instantiations

In industrial qualification of heterogeneous microstructures, customized similarity-based representation is implemented with pre-trained CLIP and FLAVA encoders plus expert-curated positive and negative references. For each criterion, the method computes a text-guided delta

$Z \in \mathbb{R}^{N \times d}$9

and an image-guided delta

$x_i$0

After z-score normalization,

$x_i$1

the hybrid score is

$x_i$2

and the decision rule is $x_i$3 for “Positive” and $x_i$4 for “Negative”. Similarity is therefore customized by explicit expert criteria, curated prototypes, and local score normalization rather than by model retraining (Safdar et al., 27 Aug 2025).

In recommendation and similarity search, customization often appears as task-aware binary codes or graph proximities. One example is projected Hamming dissimilarity for collaborative filtering, where the user code itself selects which bits are active: $x_i$5 This retains XOR-and-bitcount efficiency while making bit importance query-dependent. A second example is Collaborative Similarity Embedding, which jointly learns direct user–item proximity and $x_i$6-th order neighborhood proximity in a bipartite graph with objective

$x_i$7

In both cases, similarity is explicitly tailored to recommendation structure rather than left to a fixed Euclidean or Hamming metric (Hansen, 2021, Chen et al., 2019).

Outside vision and recommendation, customized similarity also appears in binary code and speech. StrTune slices binaries by data dependence, forms slice graphs with flow types, and uses a Siamese loss so slices implementing the same value computation under different compilation settings become close in feature space. Retraining-free customized ASR for enharmonic named entities represents each target named entity by its phoneme sequence and compares it to a user dictionary using

$x_i$8

Here, customization is realized by data-dependence semantics in one case and by a phoneme-level dictionary in the other (He et al., 2024, Sudo et al., 2023).

5. Empirical behavior and common misconceptions

A recurring empirical result is that similarity is not plug-and-play. In self-distillation, naively replacing same-image positives with nearest-neighbor positives can degrade performance or collapse altogether. On ImageNet-1k with ResNet-50 and 100 epochs, SimSiam + NN drops from k-NN top-1 57.1 to 56.2 and linear accuracy from 68.0 to 65.9, while DINO-2 + NN collapses; by contrast, AdaSim restores stability and improves over both the naive NN variant and the original baselines (Lebailly et al., 2023).

Another recurring result is that multiple similarity relations and learned weighting improve transfer. On Zappos50k, out-of-domain brand classification rises from 32.10% top-1 for multi-task cross-entropy to 42.62% for MSCon, and corruption experiments show that learned uncertainty $x_i$9 increases as task noise increases, thereby reducing the weight of unreliable similarity sources (Mu et al., 2023).

Customized representations can also improve compactness and robustness. Set2Box+ reports up to 40.8X smaller estimation error while requiring 60% fewer bits to encode sets, and up to 96.8X more concise representations with similar estimation error. In noisy heterogeneous graphs, NoiseHGNN reports +5$z_i=f(t(x_i))$06% improvements on several noised datasets compared with previous SOTA methods, which the paper attributes to the use of a synthesized similarity graph as a noise-robust auxiliary view rather than as a replacement for heterogeneous structure (Lee et al., 2022, Zhang et al., 2024).

Several misconceptions follow directly from these results. One is that nearest neighbors are always safe positives; self-distillation results show the opposite when attraction is not balanced by repulsion. A second is that generic embeddings are sufficient if enough data were used in pretraining; industrial qualification work argues that generic CLIP and FLAVA embeddings do not directly encode the expert criteria needed for qualification. A third is that binary match labels adequately define similarity; VACSR argues that they compress a continuous similarity space and create false negatives that actively damage generalization (Safdar et al., 27 Aug 2025, Wei et al., 29 May 2026).

6. Limitations and open directions

Current methods remain constrained by hyperparameters, memory, annotation structure, and scale. AdaSim requires a warm-up phase, a full latent cache $z_i=f(t(x_i))$1, and tuning of temperature $z_i=f(t(x_i))$2, window size $z_i=f(t(x_i))$3, and support size $z_i=f(t(x_i))$4; the paper also avoids multi-crop for simplicity. MSCon scales linearly in the number of similarity metrics because each metric requires its own projection head and uncertainty parameter, and it assumes multiple categorical similarity labels rather than continuous or soft relations (Lebailly et al., 2023, Mu et al., 2023).

Domain-specific systems inherit domain-specific bottlenecks. The industrial vision-language framework depends on manual prompt design, reference coverage, segmentation quality, and a single Ni–WC MMC case study. SLKE is sensitive to kernel choice, uses $z_i=f(t(x_i))$5-scale matrix operations in its basic form, and optimizes a non-convex objective. CRL is not optimal when the desired semantics already coincide with the dominant semantics of the universal space, and its basis construction remains an approximation whose descriptor filtering has no universally best strategy (Safdar et al., 27 Aug 2025, Kang et al., 2019, Liu et al., 6 Oct 2025).

The main open directions already identified in the literature are convergent. AdaSim suggests better latent-quality estimators and extension to other modalities; MSCon suggests data-dependent uncertainty, continuous similarity metrics, meta-learned similarity functions, and conditional similarity weights; SLKE suggests multiple-kernel learning; CRL suggests improved basis construction and broader modal coverage. Taken together, these directions imply a broader research program in which representation learning no longer assumes a single immutable metric, but instead learns, conditions, or regularizes similarity as a first-class object aligned with the operational semantics of the task (Lebailly et al., 2023, Mu et al., 2023, Kang et al., 2019, Liu et al., 6 Oct 2025).