NearID: Identity Representation Learning via Near-identity Distractors

Abstract: When evaluating identity-focused tasks such as personalized generation and image editing, existing vision encoders entangle object identity with background context, leading to unreliable representations and metrics. We introduce the first principled framework to address this vulnerability using Near-identity (NearID) distractors, where semantically similar but distinct instances are placed on the exact same background as a reference image, eliminating contextual shortcuts and isolating identity as the sole discriminative signal. Based on this principle, we present the NearID dataset (19K identities, 316K matched-context distractors) together with a strict margin-based evaluation protocol. Under this setting, pre-trained encoders perform poorly, achieving Sample Success Rates (SSR), a strict margin-based identity discrimination metric, as low as 30.7% and often ranking distractors above true cross-view matches. We address this by learning identity-aware representations on a frozen backbone using a two-tier contrastive objective enforcing the hierarchy: same identity > NearID distractor > random negative. This improves SSR to 99.2%, enhances part-level discrimination by 28.0%, and yields stronger alignment with human judgments on DreamBench++, a human-aligned benchmark for personalization. Project page: https://gorluxor.github.io/NearID/

• The paper introduces a novel framework that decouples object identity from background context using near-identity distractors.
• It employs a lightweight Multi-head Attention Pooling head on a frozen backbone with a structured two-tier contrastive loss and ranking regularization.
• Empirical results reveal an SSR improvement from 30.7% to 99.2% and enhanced alignment with human and oracle judgments on identity-sensitive tasks.

NearID: Identity Representation Learning via Near-identity Distractors

Introduction and Problem Motivation

The paper addresses a fundamental limitation in contemporary visual foundation models (e.g., CLIP, DINOv2, SigLIP2, Qwen3-VL) regarding instance-level identity discrimination in images. Specifically, these pre-trained encoders frequently conflate intrinsic object identity with extraneous background context, undermining their reliability for downstream personalized generation and image editing tasks. The authors rigorously demonstrate that when visually similar but distinct objects (Near-identity distractors) are inserted into the same background as a reference, standard models are unable to reliably isolate genuine identity cues, often assigning higher correspondence to the distractor than to true cross-view matches. This failure mode extends to automated evaluation protocols for identity preservation, invalidating the widespread usage of simple embedding similarity (e.g., CLIP-I, DINO scores).

Figure 1: NearID constructs matched-context distractors to eliminate background-based shortcuts, allowing measurement and isolation of true identity discrimination capacity.

NearID Framework and Methodology

To directly address context-entanglement, the authors introduce NearID, a unified framework with three core components: (1) a large-scale dataset of curated object identities with meticulously synthesized NearID distractors; (2) a hierarchical, two-tier contrastive learning objective that implements an explicit similarity order (same identity > NearID distractor > random negative); and (3) a strict, margin-based evaluation protocol quantifying identity discrimination and its alignment with human and oracle judgments.

Dataset Synthesis

The NearID dataset is constructed from a rigorously filtered subset of SynCD, yielding 19,386 unique identities with 2–3 high-quality, multi-view images per identity and an extensive set (316,505) of NearID distractors generated via diverse diffusion models (SDXL, FLUX.1, Qwen-Image, PowerPaint). This ensures broad coverage of artifact and generative priors, preventing shortcut learning.

Structured Contrastive Objective

The model adapts only a lightweight Multi-head Attention Pooling (MAP) head (∼3.6% of parameters) on top of a frozen SigLIP2 backbone. This head is optimized by a two-component loss: (i) a discrimination term that softmaxes over all positives and explicit NearID distractors, and (ii) a ranking regularizer encouraging each NearID distractor to be closer to the anchor than generic batch negatives, using a softplus listwise formulation. This approach avoids the excessive collapse of semantic structure seen in aggressive metric mining and obviates manual margin selection.

Figure 2: NearID reconfigures only the MAP head on a frozen backbone, reshaping embedding space to privilege identity above contextual similarity.

Evaluation Protocol and Analysis

The evaluation design exclusively considers matched-context scenarios: for each anchor, the model assesses whether cross-background positive views receive higher similarity scores than visually confusable distractors in the same background. Two primary metrics are introduced: Sample Success Rate (SSR), reporting the fraction of identities where all positive-vs-distractor margins are strictly positive, and Pairwise Accuracy (PA), the ratio across all margin test pairs.

Strong baselines—frozen CLIP, DINOv2, SigLIP2, Qwen3-VL—perform poorly under this strict protocol (e.g., SigLIP2 SSR: 30.7%).

Figure 3: Left: NearID MAP head's attention focuses on identity-salient regions, suppressing context; right: only NearID achieves high discrimination, mask-free inference, and strong alignment with human/part-level oracles.

Empirical Results

NearID achieves profound gains across all axes. Under matched-context discrimination, SSR rises from 30.7% (frozen SigLIP2) to 99.2%. On the Mind-the-Glitch (MTG) part-level edit benchmark, NearID reaches 35.0% SSR (vs. 0% for all frozen encoders; best alternative, VSM, achieves 7.0%). Alignment with human preference scores on DreamBench++ is boosted as well (Pearson corr. up to 0.545 vs. 0.516 for SigLIP2), generalizing to previously unseen domains (animals, humans).

Qualitative analyses underline these results: NearID distinctly separates true positives from near-identity distractors, while frozen backbone and VLM judges either collapse positives and distractors or default to category-level granularity.

Figure 4: NearID assigns higher scores to true instance positives across multiple categories, whereas existing encoders and VLMs are frequently confounded by distractors inserted into matching backgrounds.

The robustness of the approach is further evidenced by cross-generator generalization (e.g., SSR ≥ 97.8% on all 9 distractor-generation pipelines) and insensitivity to explicit masking (foreground-only evaluation yields negligible SSR degradation).

Ablation Studies

Ablations establish that (i) training on NearID distractors is essential—removing them yields a 40–60% drop in SSR; (ii) part-level edits (MTG) alone cannot teach object-level decoupling; (iii) explicit oracle-based ranking objectives or aggressive margin/enhanced contrastive mining induce representation collapse or poor human/oracle alignment; and (iv) the NearID loss with moderate ranking regularization ($\alpha=0.5$) provides optimal trade-off between sharp identity discrimination and maintenance of graded, semantically meaningful similarity (crucial for generalization and perceptual fidelity).

Impact and Insights

The research establishes that evaluation and learning paradigms in subject-driven and personalization pipelines must rigorously decouple object identity from background scene, as current practice leads to substantial misestimation of model fidelity. NearID provides foundational tools (dataset, benchmark, methodology) for the diagnosis and remediation of such failure modes.

Pragmatically, the lightweight head adaptation framework enables reliable identity representation learning with minimal compute and without sacrificing the strong prior knowledge in frozen backbones. The approach readily integrates with any encoder architecture supporting patchwise or spatial embedding extraction.

Theoretically, the strict matched-context distractor protocol surfaces compositional entanglements in current visual foundation models, guiding future research in architectural design and metric learning regimes sensitive to this class of confounds. Furthermore, the structured regularization approach introduced here provides a data-driven alternative to laborious margin tuning or oracle ranking, and can transfer to related tasks requiring hierarchical or listwise control over negative space granularity.

Figure 5: KernelPCA projections indicate that NearID facilitates clean separation of instance positives and near-identity distractors, unlike the baseline embedding space where clusters are entangled.

Conclusion

NearID advances the state of the art in open-domain identity-aware representation learning by systematically dismantling contextual shortcuts. It delivers nearly perfect identity discrimination under attack from highly confusable, matched-context distractors with strong calibration to human and oracle judgments, all while maintaining general semantic integrity. The NearID dataset and evaluation protocol should become standard for future identity-focused representation and generation research, ensuring claims of identity fidelity are genuinely grounded in discriminative capacity rather than incidental background correlation.

Figure 6: On part-level edits (MTG), NearID detects true positives even when distractors differ by subtle, localized modifications, surpassing both holistic and region-focused baselines.