Sample-Level Explorability Metric

• Sample-Level Explorability Metric is a quantitative measure that evaluates individual data samples on fidelity, diversity, and authenticity in machine learning systems.
• It leverages per-sample classifiers and hypothesis tests to provide granular insights for model auditing, sample curation, and anomaly detection.
• The metric supports quality improvements in synthetic data generation by identifying memorization, bias, and privacy risks through precise pointwise evaluation.

A sample-level explorability metric denotes any quantitative measure designed to characterize the distinct properties or vulnerabilities of individual data samples or predictions within machine learning models. Such metrics capture per-sample fidelity, diversity, explainability, adversarial sensitivity, privacy, or contribution, enabling detailed diagnosis, auditing, and refinement of generative, discriminative, or multimodal learning systems. Unlike dataset-level or global metrics, sample-level measures provide granular insight on a pointwise basis, supporting post-hoc interventions and guiding both application-specific quality improvements and compliance monitoring.

1. Core Principles and Definitions

Sample-level explorability in modern machine learning refers to the ability to systematically evaluate and interpret the attributes or fate of individual samples—whether generated or consumed—by a model. Central to this are quantitative metrics capable of assigning explicit scores, decisions, or classifications to each instance. Primary requirements include:

• Granularity: Each score applies to a single sample, not only to model- or distribution-level aggregates.
• Interpretability: The metric's output can be mapped to actionable properties (e.g., fidelity, diversity, “forgettability,” vulnerability).
• Domain-agnosticism: Applicable, in principle, to synthetic data (images, text), adversarial robustness, privacy engineering, or explainability.

For generative modeling, the canonical framework establishes three dimensions, as introduced in the $(\alpha$-Precision, $\beta$-Recall, Authenticity) paradigm (Alaa et al., 2021):

• $(\alpha)$-Precision: Fraction of synthetic samples lying inside the $\alpha$-support of the real data—quantifies sample-level fidelity.
• $(\beta)$-Recall: Fraction of real samples covered by the $\beta$-support of the generative density—measures diversity at the sample level.
• Authenticity: Probability that a generated sample is novel (not a copy/memorization of the training set)—indexes generalization or privacy risk per sample.

Mathematically, the core definitions are as follows:

• $$\mathcal{S}^\alpha \triangleq \arg\min_{s}\{ V(s): P(s) = \alpha \}$$, where $V(\cdot)$ is volume and $P$ is the distribution.
• $$P_\alpha = \mathrm{Pr}(\tilde{X}_g \in \mathcal{S}^α_r)$$, $$R_\beta = \mathrm{Pr}(\tilde{X}_r \in \mathcal{S}^β_g)$$, $$A = \mathrm{Pr}(\text{generated sample is not a noisy copy})$$.

2. Computation via Sample-wise Classification

Explorability metrics leverage pointwise classifiers or hypothesis tests to generate per-sample binary or real-valued signals. For the sample-level precision, recall, and authenticity, as operationalized in (Alaa et al., 2021):

• $f_P(\cdot)$: Classifies a synthetic sample as high-fidelity if within the $\alpha$-support ball of real embeddings.
• $f_R(\cdot)$: Flags a real sample as covered by the generator’s $\beta$-support.
• $f_A(\cdot)$: Tests if a generated sample is “authentic” (i.e., not a memorized instance) based on its distance to the nearest real training point relative to intra-real pairwise distances.

Each classifier assigns a 0/1 decision per sample, and scores are aggregated:

• $P_\alpha = \frac{1}{m}\sum_j f_P(\tilde{X}_{g,j})$
• $R_\beta = \frac{1}{n}\sum_i f_R(\tilde{X}_{r,i})$
• $A = \frac{1}{m}\sum_j f_A(\tilde{X}_{g,j})$

Algorithms embed the data using trainable encoders (e.g., one-class networks), estimate quantile-based radii, and deploy non-parametric proximity checks or likelihood-ratio tests tailored for the statistical properties of the embedded space.

3. Application: Model Auditing and Post-hoc Sample Curation

Sample-level metrics enable a post-hoc auditing workflow that extends beyond global score comparisons. The process, as detailed in (Alaa et al., 2021), involves assigning individual quality and authenticity scores to generated data. Downstream, samples with low fidelity (outside the real-support ball) or low authenticity (e.g., found to be memorized) can be filtered from synthetic datasets.

Two concrete use cases are prominent:

• Curation: After model sampling, remove outlier or memorized samples, yielding “cleaned” synthetic datasets for downstream statistical or learning tasks.
• Rejection Sampling: During generation (if the model interface permits), iteratively accept samples passing $f_P$ and $f_A$ and reject others.

This auditing strategy demonstrably improves performance in application tasks such as synthetic data-based predictive modeling and enhances privacy compliance—particularly where minimizing information leakage from memorized samples is mandated.

4. Generalization Dimension and Privacy Implications

The authenticity metric extends explorability to model generalization and privacy. It disambiguates two operational regimes in synthetic models:

• Generalizing: The generator invents new, plausible samples, as reflected by high $A$.
• Memorizing: The generator outputs (possibly perturbed) near-duplicates of training data, lowering $A$.

The metric formalizes this via a probabilistic mixture:

$$P_g = A \cdot P_g' + (1 - A) \cdot \delta_{g,\epsilon}$$

where $\delta_{g,\epsilon}$ denotes a noisy copy component. Authenticity is estimated by a comparative test of a synthetic sample’s proximity to its nearest training neighbor versus the distribution of distances between real samples.

Such operationalization is critical when evaluating models tasked with sensitive data synthesis (e.g., clinical or financial datasets), ensuring that privacy risks due to overfitting and unauthorized memorization are systematically monitored.

5. Diagnostic Power and Practical Interventions

Sample-level explorability metrics provide practitioners with fine-grained diagnostic and remediation tools that surpass those based on distributional distances (e.g., FID, MMD):

• Failure Mode Identification: Visualization of $P_\alpha$ and $R_\beta$ as functions of $\alpha$ and $\beta$ surfaces distributional weaknesses—such as mode collapse or coverage gaps—in generative models.
• Hyper-parameter and Utility-Privacy Tradeoff Tuning: In privacy-preserving generation (e.g., using ADS-GAN for medical synthesis), balancing fidelity/diversity vs. authenticity via sample-level metrics informs optimal model selection and calibration.
• Quality Assurance for Heterogeneous Data: Applicability across image, timeseries, and tabular modes supports domain-agnostic evaluation pipelines.

Notably, sample-level metrics enable class-wise and subgroup analysis, supporting fairness auditing and targeted enhancement of generators.

6. Comparative Summary and Broader Implications

Component	Property Assessed	Key Method
$\alpha$-Precision	Fidelity	Support set inclusion test (synthetic in real)
$\beta$-Recall	Diversity	Support set inclusion test (real in synthetic)
Authenticity	Generalization	Local proximity-based copy detection

The rigorous, three-dimensional framework for sample-level explorability captures complementary and independent aspects of generative quality, yielding an actionable, interpretable, and robust toolkit for synthetic data evaluation. Its universality and granularity mark a shift from reliance on aggregate scores—improving practical model selection, risk management, privacy surveillance, and detailed post-hoc data curation (Alaa et al., 2021).