Sample-Level Explorability Metric

• Sample-level explorability metrics are quantitative tools that assess individual sample behavior, including fidelity, diversity, and vulnerability.
• They employ techniques like α-Precision, β-Recall, and trust scores to guide improvements in synthetic data, explainability, and robustness.
• These metrics enable fine-grained evaluation and dynamic filtering, optimizing performance in diverse machine learning workflows.

A sample-level explorability metric is a quantitative tool designed to assess model behavior, output quality, or vulnerability with respect to individual samples in a dataset. Unlike aggregate metrics, which average performance across data or focus on overall distributions, sample-level explorability metrics enable fine-grained analysis, guiding both evaluation and improvement of machine learning systems. These metrics are now central in domains ranging from synthetic data generation and explainability to adversarial robustness, modality cooperation in multimodal models, privacy unlearning, and @@@@1@@@@ efficiency.

1. Core Definitions and Conceptual Dimensions

Sample-level explorability metrics serve to measure distinct qualities at the single-sample granularity, including fidelity, diversity, contribution strength, vulnerability, interpretability, privacy risk, and reasoning potential. Examples include:

• $\alpha$-Precision: Probability a synthetic sample lies in the high-density $\alpha$-support of real data (Alaa et al., 2021).
• $\beta$-Recall: Fraction of real samples covered by the $\beta$-support of synthetic data (Alaa et al., 2021).
• Authenticity: Likelihood a synthetic sample is not a near-duplicate of a training sample (Alaa et al., 2021).
• Degree of Explainability (DoX): Aggregated pertinence of explanation details to various question types, indicating how "explorable" an AI-generated explanation is (Sovrano et al., 2021).
• Trust Score $T(x^i)$: Combines distance to decision boundary and high-frequency feature reliance to estimate adversarial vulnerability (Nayak et al., 2022).
• Informative Sample-Aware Weighting: Dynamic adjustment of gradient contributions based on sample informativeness (Li et al., 2022).
• Modality Shapley Value $\varphi^i$: Average marginal contribution of each modality to the prediction for a sample (Wei et al., 2023).
• Unlearning Completeness (UnleScore): Average of likelihood and MAD-derived scores quantifying how fully a sample has been "forgotten" by a model (Wang et al., 19 Mar 2024).
• Memory Removal Difficulty (MRD): Expected normalized change in token log-likelihood under parameter perturbation to measure unlearning difficulty per sample (Feng et al., 9 Apr 2025).
• Explorability Metric $\mathcal{E}(q,a,\{O^t\})$: Combines normalized reward and entropy over rollouts to identify samples yielding diverse reasoning in RL (Tang et al., 1 Sep 2025).
• Diversity and Distance Composite Score (DDCS): Reciprocal of reconstruction distance(s) between reconstructed and target samples under model inversion attacks (Li et al., 26 Feb 2025).

These metrics provide the substrate for per-sample auditing, dynamic selection, filtering, or improvement processes.

2. Methodological Foundations

The design and computation of sample-level explorability metrics typically follow principled methodologies, such as:

• Minimum-volume set estimation: Embedding data into hyperspherical or low-dimensional spaces and computing quantiles, common for $\alpha$-Precision and $\beta$-Recall (Alaa et al., 2021).
• Binary classification of sample quality: Using classifiers $f_P$, $f_R$, or $f_A$ to assign "high-quality," "recalled," or "authentic" labels (Alaa et al., 2021).
• Knowledge graph extraction and semantic similarity: Parsing explanations into triplet details, constructing graphs, and estimating pertinence via cosine similarity in embedding space for DoX (Sovrano et al., 2021).
• Harmonic mean aggregation: Merging normalized distance and frequency-based robustness components in trust scores (Nayak et al., 2022).
• Dynamic thresholding and memory queues: Automatically tuning weighting windows and gradient decays for informative sample selection (Li et al., 2022).
• Game-theoretic Shapley value computation: Averaging marginal gains across all possible modality insertions in multimodal cooperation (Wei et al., 2023).
• Statistical comparison and normalization: Measuring changes in logits or likelihoods, evaluating distributions (Gaussian, MAD), and normalizing for scoring (Wang et al., 19 Mar 2024).
• Second-order sensitivity estimation: Perturbing model parameters and averaging normalized log-likelihood changes for MRD (Feng et al., 9 Apr 2025).
• Advantage-entropy rollouts with windowing: Aggregating exploration potential through sliding epoch windows, filtering by reward/entropy (Tang et al., 1 Sep 2025).
• Reciprocal distance-based coverage and diversity scoring: Matching reconstructions to dataset samples and combining with coverage metrics for DDCS (Li et al., 26 Feb 2025).

Techniques may involve embedding, statistical inference, deep model outputs, or combinatorial optimization, depending on the metric's domain.

3. Practical Applications and Auditing Workflows

Sample-level explorability metrics enable several practical functionalities, including:

• Auditing synthetic data: Post-hoc rejection or curation of individual samples by assessing whether they are typical, diverse, and authentic, improving downstream predictive performance (e.g., curation boosting AUC from 0.76 to 0.78) (Alaa et al., 2021).
• Filtering explanations in XAI: Evaluating sample-level faithfulness or robustness in explanation maps, guiding the selection or refinement of interpretable outputs, with statistical validation (e.g., DoX correlating with effectiveness scores; $p < 0.01$) (Sovrano et al., 2021, Stassin et al., 2023).
• Flagging adversarial vulnerability: Alerting to test samples with low trust scores, prompting human review and increasing reliability in prediction pipelines (Nayak et al., 2022).
• Guiding knowledge distillation: Selecting the most robust ("trustworthy") samples for student model training, outperforming random selection across clean and adversarial benchmarks (Nayak et al., 2022).
• Balancing multimodal cooperation: Identifying weak modalities at the sample or modality level, triggering targeted re-sampling, improving discriminability and system performance (Wei et al., 2023).
• Monitoring privacy unlearning: Continuous audit of sample “forgottenness,” identifying under- or over-unlearning events, and exposing resilience and equity risks in approximate algorithms (Wang et al., 19 Mar 2024, Feng et al., 9 Apr 2025).
• Data-efficient RL training: Dynamically pruning rollouts based on per-sample explorability, achieving up to 1.85× speed-up with similar accuracy (Tang et al., 1 Sep 2025).
• Evaluating model inversion attacks: Quantifying coverage/diversity/accuracy of reconstructed samples, allowing detection of privacy-vulnerable data points (Li et al., 26 Feb 2025).

Implementation often involves maintaining sample-level scores throughout the training or evaluation lifecycle and integrating thresholding, filtering, or sampling policies driven by these metrics.

4. Mathematical Formulation and Statistical Properties

Sample-level metrics are typically grounded in formal mathematical definitions. Notable cases include:

• Minimum-volume set:

$$S^\alpha \equiv \underset{s \subseteq S}{\arg\min} \{\text{volume}(s) \mid P(s) = \alpha\}$$

$P_{(\alpha)} = P(\Phi(X_g) \in S_r^\alpha)$

• Integrated precision/recall:

$$IP_{(\alpha)} = 1 - 2\int_0^1 |P_{(\alpha)} - \alpha|\, d\alpha$$

• Shapley value for modality valuation:

$$\varphi^i = \frac{1}{n!}\sum_{\pi \in \Pi_N} [v(S_\pi(x^i) \cup \{x^i\}) - v(S_\pi(x^i))]$$

• Unlearning completeness (UnleScore):

$$\text{UnleScore}(z) = \frac{L\text{-Diff}(z) + D\text{-Liks}(z)}{2}$$

• MRD for LLM unlearning:

$$\text{MRD}(x^i; \theta) = \left|\mathbb{E}_{\delta\sim\mathcal{N}(0, \sigma^2 I)} \sum_{t=1}^{n^i} \frac{P_t(\theta) - P_t(\theta+\delta)}{P_t(\theta)}\right|$$

• Explorability metric in RLVR:

$$\mathcal{E}(q,a,\{O^t\}_{t=e-w+1}^{e}) = \frac{1}{w} \sum_{t=e-w+1}^{e} \frac{1}{G} \sum_{i=1}^{G} \hat{A}_{i} \cdot E(o^t_i) \cdot \mathbb{I}[\ldots]$$

These formulations enable theoretical analysis, e.g., convergence properties, diagnostic curve generation, and threshold tuning.

5. Comparative Evaluation and Diagnostic Strengths

Sample-level metrics surpass aggregate metrics in their diagnostic power:

• Detection of mode collapse/invention: Precision-recall curves ($P_{(\alpha)}$, $R_{(\beta)}$) reveal subtleties missed by FID/MMD (Alaa et al., 2021).
• Privacy leakage localization: DDCS pinpoints samples susceptible to inversion, while aggregate metrics (e.g., label-level FID) can obscure individual risks (Li et al., 26 Feb 2025).
• Efficiency and effectiveness: UnleScore achieves high correlation ($r \approx 0.83$) with true unlearning completeness and is computationally efficient (10× faster than membership inference) (Wang et al., 19 Mar 2024).
• Redundancy and robustness: Comparative studies show metric families can exhibit high or low correlations (e.g., Sparseness $\tau=1.0$ vs Pixel Flipping, which is baseline-sensitive), calling for the use of multiple baselines and dummy checks (Stassin et al., 2023, Barkan et al., 23 Dec 2024).
• Domain-Agnosticism: Metrics based on embeddings or abstracted feature spaces can generalize across domains (images, time-series, tabular, NLP, multimodal) (Alaa et al., 2021, Wei et al., 2023).

Balanced, sample-level analysis also exposes fairness/equity concerns, e.g., in unlearning, where algorithms variably fulfill commitments across data groups (Wang et al., 19 Mar 2024).

6. Limitations and Implementation Considerations

Despite their strengths, sample-level metrics entail several challenges:

• High-dimensional support estimation: Calculating minimum-volume sets and supports is computationally intensive; embedding tricks (one-class networks) mitigate but do not eliminate scaling issues (Alaa et al., 2021).
• Hyperparameter tuning: Metrics often depend on quantile parameters ($\alpha$, $\beta$), threshold windows, kernel radii, or decay rates that demand careful cross-validation (Alaa et al., 2021, Li et al., 2022).
• Robustness to baseline and metric choice: Especially in explainability, metric selection and baseline representation critically affect sample-level ranking (Stassin et al., 2023, Barkan et al., 23 Dec 2024).
• Scalability and resource consumption: Advanced approaches such as NGD or multi-modal Shapley computation may present excessive runtime for large datasets (Li et al., 26 Feb 2025, Wei et al., 2023).
• Overlap and ambiguity: Metrics can redundantly capture similar phenomena (e.g., Sparseness vs Complexity), requiring deliberate choice and occasional ablation studies (Stassin et al., 2023).

Mitigation may involve embedding optimization, adaptive methods, parallelization, or selective sampling strategies.

7. Future Directions

Research around sample-level explorability metrics will likely focus on:

• Even finer granularity: Moving from sample to sub-feature or token-level, increasing diagnostic interpretability.
• Unified multi-property frameworks: Metrics capturing exploration, vulnerability, fidelity, fairness, and privacy simultaneously (possibly through combined curve analysis).
• Auto-tuned and domain-adapted metrics: Leveraging meta-learning to adjust parameters or adapt to novel modalities/datasets (Li et al., 2022).
• Open benchmarking resources: Standardized evaluation platforms for metrics and algorithms, as announced for unlearning workflows (Wang et al., 19 Mar 2024).
• Scalable deployment: Integration in auditing, monitoring, or defense pipelines for real-time risk assessment at roster-scale.

These developments reflect the fundamental role of sample-level explorability metrics in evolving high-performance, interpretable, and secure machine learning systems.