ID-Sim: An Identity-Focused Similarity Metric

Abstract: Humans have remarkable selective sensitivity to identities -- easily distinguishing between highly similar identities, even across significantly different contexts such as diverse viewpoints or lighting. Vision models have struggled to match this capability, and progress toward identity-focused tasks such as personalized image generation is slowed by a lack of identity-focused evaluation metrics. To help facilitate progress, we propose ID-Sim, a feed-forward metric designed to faithfully reflect human selective sensitivity. To build ID-Sim, we curate a high-quality training set of images spanning diverse real-world domains, augmented with generative synthetic data that provides controlled, fine-grained identity and contextual variations. We evaluate our metric on a new unified evaluation benchmark for assessing consistency with human annotations across identity-focused recognition, retrieval, and generative tasks.

• The paper introduces ID-Sim, which robustly measures unique visual identity by decoupling intrinsic object properties from contextual variations.
• It utilizes dual-supervision with InfoNCE for global alignment and Sinkhorn OT for local patch matching, leveraging synthetic augmentation for balanced training.
• The metric achieves state-of-the-art performance in instance retrieval, personalized segmentation, and re-identification while significantly reducing labeled data requirements.

ID-Sim: An Identity-Focused Similarity Metric

Motivation and Definition of Visual Identity

ID-Sim addresses the problem of selective sensitivity in visual recognition—a property of human vision wherein individuals distinguish minute, identity-altering changes while generalizing over extensive contextual variation (e.g., pose, lighting, background). This capability remains challenging for deep visual representation models, with failures particularly acute in personalized generation, instance retrieval, and re-identification tasks. Existing metrics typically conflate semantic similarity with true identity preservation, undermining evaluation and optimization for applications that demand fine-grained, instance-level discrimination.

The authors define “visual identity” as an object’s unique set of intrinsic visual properties (shape, texture, color), such that an instance encompasses all images depicting indistinguishable objects under any context. ID-Sim is proposed as a deterministic, feed-forward metric to operationalize this property and robustly reflect human-selective sensitivity.

Dataset Curation and Synthetic Augmentation

A critical barrier for identity-based metrics is that no existing dataset offers sufficient context, domain, and visual identity diversity simultaneously. The ID-Sim pipeline integrates real instance-level datasets with targeted synthetic augmentation.

Figure 1: The dataset curation pipeline combines real-world and synthetic data to ensure high diversity in context, domain, and visual identity for ID-Sim training.

Real data (“Subset 1”) provides broad domain coverage using datasets spanning everyday objects, artwork, fashion, and animals, subject to aggressive filtering for consistent instance granularity (e.g., excluding ambiguous logo categories or multi-site landmark classes). Balanced sampling strategies are used to avoid domain bias, validated through ablation studies showing a strong impact on downstream discrimination accuracy.

To address the under-representation of contextual and identity-altering changes, synthetic data generation is performed:

• Contextual edits (Subset 2a) utilize generative models for controlled modification of backgrounds and illumination, preserving foreground identity while diversifying non-intrinsic properties.
• Identity-altering edits (Subset 2b) use inpainting and Flux-based methods to produce realistic, hard-negative samples by modifying identity-defining cues without altering class semantics.

A mixture of real and synthetic triplets is constructed, with positive pairs sampled from contextually diversified views and negatives explicitly incorporating both real distractors and identity-modified edits. Extensive ablation reveals the necessity of balanced mixtures and data quality for robust generalization.

Dual-Supervision Training: Global and Local Objectives

ID-Sim employs a ViT-L/16 backbone, pretrained with DINOv3, and adapts it using dual MLP projection heads and LoRA adapters. The training objective explicitly balances global and local alignment:

Figure 2: The training pipeline uses supervised contrastive objectives: InfoNCE for global CLS tokens and Sinkhorn-regularized OT for patch-level local descriptors.

• Global supervision: InfoNCE is applied to CLS-token projections for triplet anchor/positive/negative groupings, encouraging discrimination at the image level.
• Local supervision: Sinkhorn OT aligns sets of patch embeddings between positive pairs, handling spatial misalignment due to viewpoint/context. Patch-level supervision proves critical for downstream tasks requiring spatial localization or region correspondence.

Contrastive learning is performed on batches curated to maximize both easy and hard negative diversity, with negative mining in the pretrained embedding space. The overall loss is a weighted sum of global and patch objectives, as ablation demonstrates joint supervision outperforms either individually.

Benchmarks and Experimental Protocols

Evaluation covers 7 benchmarks spanning instance retrieval (PODS, DeepFashion2), animal/person re-identification (AerialCattle2017, PetFace), fine-grained triplet discrimination (CUTE), and identity preservation in generative models (DreamBench++, Subjects2k).

• Instance retrieval and re-ID: Standard metrics (mAP, ROC AUC, nDCG) are employed, with queries required to retrieve the true instance amidst substantial context and identity variation.
• Generative evaluation: Subjects2k provides a high-fidelity, human-annotated benchmark to replace the noisy rubric-based scoring of DreamBench++, correcting for evaluation inconsistencies (see Figures 4, 23).

  Figure 3: Across all benchmarks, ID-Sim outperforms prior metrics—including universal embedding and large foundation models—using significantly less labeled data and smaller models.

Strong Claims and Empirical Findings

ID-Sim achieves consistent state-of-the-art performance across 49 out of 50 evaluation settings, substantially outperforming both large visual foundation models (e.g., DINOv3, CLIP) and metrics optimized for perceptual or semantic similarity. Notably:

• On PODS and DeepFashion2, ID-Sim yields $+0.11$ and $+0.30$ mAP over the nearest competitor, despite using $>$100$\times$ less labeled data and a smaller backbone.
• On Subjects2k, which features challenging negative pairs differing only by fine-grained details, ID-Sim outperforms all baselines by $+0.05$ AP.
• Patch-level embeddings trained with Sinkhorn OT grant strong localization and are essential for tasks like personalized segmentation, where ID-Sim more than doubles the segmentation mAP of DINOv3.

ID-Sim matches or exceeds the performance of MLLM-based evaluators (GPT-4v, Gemini, etc.) on concept preservation yet achieves this deterministically, at substantially lower computational cost and without prompt variance.

Selective Sensitivity Analysis

A key technical contribution is a joint sensitivity analysis isolating the effect of variation along four axes: identity, background, viewpoint, and lighting. Generative edit grids constructed on out-of-distribution instances (MVImgNet) quantify how similarity scores shift under each perturbation.

Figure 4: ID-Sim demonstrates pronounced sensitivity only to identity changes, maintaining near-complete invariance to background, viewpoint, and lighting transformations, outperforming other models that confound appearance and identity.

ID-Sim is uniquely selective: it penalizes only intrinsic identity alterations while maintaining robust invariance to all extrinsic factors. Competing metrics (DreamSim, DINOv3, CLIP, UNED, LPIPS) display elevated sensitivity to non-identity context, demonstrating a decoupling of appearance and identity in ID-Sim not present elsewhere. Theoretical implications are significant: these results suggest that dual-level contrastive supervision with synthetic context augmentation is sufficient to operationalize selective sensitivity in deep models.

Practical and Theoretical Implications

Practically, ID-Sim establishes a new standard for scalable, deterministic, and computationally efficient identity evaluation, with direct applicability to:

• Personalized generative model benchmarking (enabling robust monitoring of identity drift).
• Open-world re-identification and retrieval, especially in low-label or out-of-distribution domains.
• Personalized segmentation or downstream dense perception tasks, via patch-level localization.

Theoretically, the decoupling of context and identity under contrastive, synthetic-augmentation-based supervision provides a blueprint for future metric learning research. The selective sensitivity principle can potentially inform work in robust representation learning, generalization under domain shift, and perhaps provide model-based priors for human-level identity perception.

Limitations and Future Work

The definition of visual identity remains strictly property-based (shape/texture/color) and does not capture complex social or abstract identity notions (e.g., aging, accessories, or style), nor does it handle user-specified invariances. ID-Sim is prompt-free and operates globally; instance disambiguation in multi-object scenes requires external conditioning, though the learned patch features naturally support such extensions.

Future directions include integrating ID-Sim into generative feedback loops for improved personalization, and extending the metric to operate under user-controlled invariances or across more abstract identity spaces.

Conclusion

ID-Sim delivers a technically principled and empirically substantiated advance in identity-focused similarity metrics, operationalizing selective sensitivity through dataset curation and dual-headed contrastive learning. The resulting metric demonstrates robust alignment with human notions of visual identity across wide contexts, outperforming all baseline approaches on a diverse suite of benchmarks and enabling new applications in both evaluation and generation. The methodological innovations in data composition, synthetic augmentation, and supervision structure point toward future advances in robust, controllable vision representation learning.

(2604.05039)