LCMem: Latent Contrastive Memorization Network

• LCMem is a universal model for robust image memorization detection that reformulates the task as binary classification combining re-identification and copy detection.
• It employs a latent-space backbone with a Siamese ConvNeXt encoder and a two-stage training strategy to optimize both identity consistency and augmentation robustness.
• LCMem achieves significant improvements with a mean AUC of 0.96 and strong copy-detection recall, ensuring scalability and cross-domain performance.

Latent Contrastive Memorization Network (LCMem) is a universal model for robust image memorization detection, designed to address privacy auditing challenges in generative image modeling. LCMem recasts memorization detection as a unified binary classification problem: given two images, the model predicts whether they originate from the same underlying training instance. This approach integrates both re-identification (identity consistency) and copy detection (augmentation-robust duplication), enabling cross-domain generalizable memorization detection for diverse datasets and domains (Dombrowski et al., 16 Dec 2025).

1. Problem Formulation

Memorization detection is defined as a binary classification task. For a given pair of images $(x, x')$, the task is to predict whether both are derived from the same training image, capturing two core phenomena:

• Re-identification (identity consistency): determining if $x$ and $x'$ are different real images of the same subject or content.
• Copy detection (augmentation-robust duplication): determining if $x'$ is a transformed or noisy version of $x$.

Formally, the dataset of real images $D_r = \{x_i, y_i\}$ contains images $x_i$ with identity labels $y_i$. The following binary pairs are defined:

• $S_{id}^+ = \{ (x_i, x_j) \mid y_i = y_j \}$: positive re-identification pairs
• $S_{id}^- = \{ (x_i, x_j) \mid y_i \neq y_j \}$: negative re-identification pairs

For copy detection, a family of strong augmentations $\mathcal{A}$ (e.g., rotation, blur) is applied:

• $$S_{copy}^+ = \{ (x, A(x)) \mid x \in D_r, A \in \mathcal{A} \}$$: positive copy pairs
• $S_{copy}^-$: negative pairs drawn at random

The target classifier is $P_\theta(x, x') = \sigma(h_\theta(u, u'))$, where $u, u'$ are learned image embeddings, $\sigma$ is the sigmoid, and $h_\theta$ is the prediction head. The objective is $P_\theta \approx 1$ for all $S_{id}^+ \cup S_{copy}^+$ and $P_\theta \approx 0$ for all negatives.

2. Model Architecture

LCMem employs a latent-space representation and a Siamese neural encoder within a unified prediction and contrastive learning framework:

• Latent-space Backbone: Each input image $x$ is passed through a pretrained Stable Diffusion v2 VAE encoder $E$, yielding a compressed latent $z = E(x) \in \mathbb{R}^{4 \times 64 \times 64}$. This process reduces high-frequency image noise while preserving semantic identification cues.
• Siamese ConvNeXt Encoder: Both $z$ and $z'$ are processed in parallel by a weight-sharing ConvNeXt-Tiny backbone $S_\theta$, yielding embeddings $u, u' \in \mathbb{R}^d$, $d \approx 768$.
• Projection Heads: Two outputs:
  • A contrastive projection for embedding similarity (dot-product, $\ell_2$-normalized as needed).
  • A classification head (MLP) takes $|u - u'|$, producing the logit input to $\sigma$ for $P_\theta(x, x')$.

Inference Flow:

x → E → z \ ↘ S_θ → u →──\ x′ → E → z′ / ↘ S_θ → u′ → |u–u′| → MLP → σ → P_θ(x, x′)

3. Two-Stage Training Strategy

LCMem is trained in two distinct phases to disentangle identity learning from robust augmentation matching:

• Stage I: Identity-Consistency Learning
  • Data: $(x_i, x_j)$ pairs from $S_{id}^+ \cup S_{id}^-$, no augmentation.
  • Loss: Contrastive NT-Xent loss over batch of size $K$:

  $$\mathcal{L}_{id} = -\log \frac{\exp(u \cdot u^+ / \tau)}{\sum_{k=1}^K \exp(u \cdot u_k / \tau)}$$

  with $\tau = 0.07$, encouraging intra-identity attraction and inter-identity repulsion.

• Stage II: Augmentation-Robust Copy Detection
  • Data: For each $x$, a strong augmentation $A \sim \mathcal{A}$ is applied; $z = E(x)$, $z' = E(A(x))$.
  • Loss: Same NT-Xent loss, now comparing $u, u'$:

  $$\mathcal{L}_{copy} = -\log \frac{\exp(u \cdot u'/\tau)}{\sum_k \exp(u \cdot u_k'/\tau)}$$

• Combined Objective:

$$\mathcal{L}_{LCMem} = \lambda_{id} \mathcal{L}_{id} + \lambda_{copy} \mathcal{L}_{copy}$$

Typically, $\lambda_{id} = \lambda_{copy} = 1$.

Ablation analysis shows that omitting Stage II collapses augmentation robustness and harms cross-domain generalization.

4. Data and Evaluation Protocols

LCMem is evaluated jointly on multiple public datasets spanning varied domains:

Dataset	Domain	Size/Labels
CelebA	Faces	200K / 40 attributes
ImageNet-LT	Natural images	185K / 1,000 classes
MIMIC-CXR	Chest X-rays	120K / patient IDs
NIH-CXR-LT	Chest X-rays	84K / 20 disease labels
ISIC-2020	Dermoscopic	33K / binary skin lesion
CTRate	CT slices	27K / pulmonary disease

Re-identification Evaluation:

• Area under ROC curve (AUC-ROC)
• Specificity at 99% recall

Copy-detection Evaluation:

• Threshold $T$ calibrated to 95% recall under mild augmentations.
• On 1,000 test images, six strong perturbations (Gaussian blur, noise, rotation, brightness, contrast, JPEG) at varied levels are applied.
• Recall at strongest perturbation, averaged across datasets.

5. Experimental Results

5.1 Re-identification Performance

• LCMem trained across all datasets achieves mean AUC $\approx 0.96$ and specificity $\approx 0.65$ at 99% recall.
• The strongest prior foundation model (CLIP) reaches only specificity $\approx 0.54$; other task-specific methods yield less than 10% specificity, underperforming LCMem by over 16 percentage points at the same recall constraint.

5.2 Copy-Detection Robustness

• LCMem demonstrates recall $\approx 0.85$ (macro-averaged across datasets) under strongest perturbations. BYOL attains $\approx 0.55$.
• This reflects a 30 percentage-point gain in augmentation robustness relative to previous methods.

5.3 Ablation Analysis

• Omitting Stage II reduces copy recall from $\approx 0.85$ to $\approx 0.40$.
• Introduction of mild augmentations in Stage I degrades re-ID AUC by 5–10 percentage points.
• Adjusting the contrastive-classification loss balance ($\alpha \approx 0.8$ for contrastive) optimizes AUC.
• Switching from raw pixel inputs to VAE latents accelerates training $4\times$ and gives +0.03 AUC.

6. Cross-Domain Generalization and Failure Modes

Joint training on all six datasets reveals significant overfitting among baseline filters (SSIM, Inception-features, CLIP, BYOL, prior Siamese nets), which exhibit domain-specific collapse. LCMem consistently maintains $>0.90$ AUC for re-identification and copy detection on all evaluated datasets. The most prominent failure case for LCMem involves images subjected to extreme brightness shifts, wherein BYOL shows a transient 0.05 recall advantage. On the large-scale MIMIC-CXR one-vs-all benchmark (5,000 patients, 220M negative pairs), LCMem registers only a single misclassification among 16,000 positives.

7. Privacy and Practical Implementation Considerations

7.1 Motivation for Reliable Memorization Detection

Widespread deployment of generative models in sensitive domains (e.g., healthcare, digital art) escalates privacy and copyright risks if training data are inadvertently reproduced. LCMem functions as a defense against such memorization-driven leaks.

7.2 Comparison to Prior Filters

Existing privacy filters—including MAE, SSIM, foundation models (CLIP, DINO, BYOL), and task-specific Siamese architectures—lack simultaneous identity and augmentation robustness. These methods underperform LCMem in cross-domain scenarios, with either low specificity or diminished transform-invariance, and exhibit pronounced failures when scaling to heterogeneous data.

7.3 Practical Deployment

LCMem enables real-time large-scale evaluation, processing approximately 10,000 image pairs per second on a single GPU. The typical deployment pipeline for synthetic data involves precomputing an “atlas” of embeddings for reference images; each new candidate is matched in latent space, with any generated image satisfying $P_\theta(x_\text{real}, x_\text{gen}) > 0.5$ flagged and rejected. Empirical findings indicate this protocol bounds memorization rates below 5% under a 1% false-positive budget.

In summary, LCMem unifies re-identification and copy-detection via a two-stage, contrastive paradigm in generative latent space, achieving up to 16 percentage-point gains in re-identification and 30 percentage-point improvements in copy detection over previous methods, maintaining robustness across diverse domains, and providing a scalable privacy safeguard for synthetic data generation workflows (Dombrowski et al., 16 Dec 2025).