Visual Memorization Audits

• Visual memorization audits are systematic protocols that quantify and attribute training data memorization, distinguishing genuine generalization from overfitting.
• They employ two-model and one-model estimation strategies to evaluate recall and precision gaps using metrics like Population Precision Gap and AUCG.
• Audit methodologies inform regularization strategies such as text masking and weight decay to balance model utility while mitigating privacy risks.

Visual memorization audits refer to systematic protocols and quantitative metrics designed to identify, quantify, and attribute training-set memorization in visual and vision–LLMs. They distinguish genuine generalization—inferring properties from correlations or priors—from the model’s recall of idiosyncratic, non-generalizable features of training data. Precise audit methodologies enable robust privacy risk assessment, benchmarking, and the principled tuning of regularization strategies for large-scale encoders including contrastive, self-supervised, and multi-modal (image–text) architectures.

1. Formal Definitions and Conceptual Foundations

Visual memorization in the context of representation learning is operationalized as the model’s ability to infer target labels or recover detailed annotations about an input, given only partial or proxy information, to a greater extent than can be justified by dataset-level correlations. For vision–LLMs (VLMs) trained on image–caption pairs, rigorous definitions such as Déjà Vu memorization and variants precisely isolate memorization from generalization by leveraging disjoint training splits, public image/text galleries, and auxiliary predictors.

Déjà Vu Memorization (Definition 1):

Given a VLM $f$ trained on dataset $D_{\mathrm{tr}}$, a paired example $z = (z_{\mathrm{img}}, z_{\mathrm{txt}})$ exhibits memorization if the set of object labels recovered by using $f(z_{\mathrm{txt}})$ to retrieve similar public images has a significantly higher overlap with the ground-truth objects of $z_{\mathrm{img}}$ when $z \in D_{\mathrm{tr}}$ (seen during training) as compared to $z \not\in D_{\mathrm{tr}}$. Practically, this requires comparing model $f_A$ (trained on set $A$) against $f_B$ (trained on disjoint set $D_{\mathrm{tr}}$0) (Jayaraman et al., 2024).

Single-Model Estimation (One-Model Déjà Vu):

For pre-trained models where retraining a reference model is infeasible, dataset-level correlation is estimated using auxiliary models (e.g., ResNet-50 classifier, Naive Bayes on detected objects, or pre-trained text embedder for captions), providing a baseline version of what should be inferable from correlations alone (Kokhlikyan et al., 8 Apr 2025).

2. Audit Methodologies and Quantitative Metrics

Two-Model Audit Protocols:

1. Partition the dataset $D_{\mathrm{tr}}$1 into disjoint sets $D_{\mathrm{tr}}$2 and $D_{\mathrm{tr}}$3. Train $D_{\mathrm{tr}}$4 and $D_{\mathrm{tr}}$5 respectively.
2. For each $D_{\mathrm{tr}}$6, compute "recovered" objects using k-nearest neighbors under $D_{\mathrm{tr}}$7 and $D_{\mathrm{tr}}$8.
3. For each $D_{\mathrm{tr}}$9, define:
   • $z = (z_{\mathrm{img}}, z_{\mathrm{txt}})$0
   • $z = (z_{\mathrm{img}}, z_{\mathrm{txt}})$1
   • Memorization gap: $z = (z_{\mathrm{img}}, z_{\mathrm{txt}})$2, likewise for precision.
4. Aggregate metrics:
   • Population Precision Gap (PPG):

   $z = (z_{\mathrm{img}}, z_{\mathrm{txt}})$3

• Population Recall Gap (PRG): defined analogously.

• AUC Gap (AUCG):

  $z = (z_{\mathrm{img}}, z_{\mathrm{txt}})$4 where $z = (z_{\mathrm{img}}, z_{\mathrm{txt}})$5 are the CDFs over recall values. (Jayaraman et al., 2024)

One-Model Estimation (Efficient Audit):

1. Build a reference model $z = (z_{\mathrm{img}}, z_{\mathrm{txt}})$6 using a held-out set to estimate $z = (z_{\mathrm{img}}, z_{\mathrm{txt}})$7.

2. For each training point $z = (z_{\mathrm{img}}, z_{\mathrm{txt}})$8, compute:

   • $z = (z_{\mathrm{img}}, z_{\mathrm{txt}})$9 via k-NN or linear probe (memorization signal).
   • $f(z_{\mathrm{txt}})$0 for correlation baseline.
   • Declare $f(z_{\mathrm{txt}})$1 memorized if $f(z_{\mathrm{txt}})$2 and $f(z_{\mathrm{txt}})$3.
3. Aggregate over the set to obtain a population memorization rate.
4. Report both aggregate and top-$f(z_{\mathrm{txt}})$4 (highest-confidence) memorization (Kokhlikyan et al., 8 Apr 2025).

Sample-Level Analysis and Benchmarks:

• Both approaches report aggregate and percentile scores; e.g., top-20% Déjà Vu (DV$f(z_{\mathrm{txt}})$5).
• For vision–language audits, both cross-modal (text→image) and unimodal (text→text) NNs serve as baselines; e.g., OpenCLIP YFCC15M gives cross-modal PPG$f(z_{\mathrm{txt}})$60.16, PRG$f(z_{\mathrm{txt}})$70.17.

3. Practical Implications and Empirical Findings

Population-level memorization is nontrivial even at large scales: for OpenCLIP ViT-B-32 trained on up to $f(z_{\mathrm{txt}})$8M LAION pairs, memorization metrics such as AUCG stabilize at nonzero values—AUCG$f(z_{\mathrm{txt}})$90.023 at $z_{\mathrm{img}}$0M, with higher values at smaller scales (AUCG$z_{\mathrm{img}}$10.074 at $z_{\mathrm{img}}$2M) (Jayaraman et al., 2024). Sample-level gaps in recall/precision can exceed $z_{\mathrm{img}}$3–$z_{\mathrm{img}}$4 for the most “vulnerable” records. Efficient one-model audits yield nearly identical population rates to two-model baselines, validating their suitability for open-source and pre-trained models (Kokhlikyan et al., 8 Apr 2025).

Memorization in subset-trained encoders is markedly higher than in off-the-shelf (OSS) models, with VICReg, Barlow Twins, and DINO showing substantial reduction of the DV metric when trained on full ImageNet rather than a 300k subset (e.g., VICReg OSS DV(ResNet)$z_{\mathrm{img}}$5 vs $z_{\mathrm{img}}$6 on subset) (Kokhlikyan et al., 8 Apr 2025).

4. Mitigation Strategies and Regularization Effects

Text-domain regularization is the most effective strategy for reducing memorization with minimal utility cost. Empirically:

• Randomly masking a fraction $z_{\mathrm{img}}$7 of caption tokens reduces AUCG by more than half, from $z_{\mathrm{img}}$8 to $z_{\mathrm{img}}$9 at $z \in D_{\mathrm{tr}}$0, with only a modest drop in zero-shot accuracy (from $z \in D_{\mathrm{tr}}$1 to $z \in D_{\mathrm{tr}}$2) (Jayaraman et al., 2024).
• Increasing weight decay and decreasing training duration provide intermediate benefits, but trade utility more aggressively.
• For self-supervised representation learners, larger and more diverse training sets dilute memorization (lower DV), consistent with the observed reduced gap between what the model can infer via correlation vs. the training-sample-specific signal.

Regularization must be guided by explicit memorization metrics, rather than utility alone, to achieve robust privacy-preserving performance.

5. Limitations, Best Practices, and Interpretability

There are key caveats in both defining and measuring visual memorization:

• Overfitting the reference correlation estimator $z \in D_{\mathrm{tr}}$3 results in underestimated memorization.
• Simple conditional independence assumptions (as in Naive Bayes on detected objects) can yield noisy approximations, especially with imperfect object detectors.
• For vision–language audits, the coverage and stylistic match of public text sets crucially affect the baseline.

Best practices for practitioners include:

• Train correlation baselines on strictly held-out data.
• Cross-check with diverse estimators to control false positives.
• Report both population-level and top-p% metrics to characterize the long-tail of memorization risks.
• Use entropy-based confidence to triage candidate high-risk samples for review.
• For vision–language encoders, combine cross-modal and unimodal baselines for comprehensive coverage (Kokhlikyan et al., 8 Apr 2025).

6. Broader Impacts and Future Directions

Systematic visual memorization audits reveal persistent privacy and attribute-inference risks in large-scale encoders, despite increasing training set size. Integrating memorization audits and trade-off metrics into the development and deployment pipeline—particularly for high-capacity VLMs and self-supervised learners—enables both privacy risk assessment and utility-preserving mitigation strategies. Ongoing research seeks to further refine sample-level attributions, extend audits to generative models, and develop more formally grounded, scalable methodologies for auditing and bounding visual memorization in open-source and commercial systems.

---

Table: Major Quantitative Metrics for Visual Memorization Auditing

Metric	Definition Summary	Interpretation/Usage
DV (Déjà Vu)	Accuracy diff between same-data and hold-out encoder	Sample- and population-level memorization signal
PPG/PRG	Net count of sample recall/precision advantages	Aggregate bias in recovery of ground-truth objects
AUCG	Area between recall CDF curves of paired encoders	Population memorization distribution shift
Top-p% DV	Highest-confidence memorization effect	Long-tail risk characterization

---

All claims, definitions, and workflow details in this entry are found in (Jayaraman et al., 2024) and (Kokhlikyan et al., 8 Apr 2025).