In-Distribution Success Rate (IDSR)

• In-Distribution Success Rate (IDSR) is a family of metrics that condition performance on explicitly defined in-distribution subsets based on criteria like energy thresholds, variant distributions, or feature congruence.
• It spans methodologies from energy-based pseudo-labeling in semi-supervised learning to distributional attack evaluations and data-centric reliability analysis, each offering concrete operational definitions.
• Empirical results indicate that applying defined in-distribution criteria can enhance model accuracy and robustness, with improvements such as a 6.9% accuracy boost and better tail-class recall.

Searching arXiv for the cited papers and closely related usage of “In-Distribution Success Rate / distributional success” to ground the article. In-Distribution Success Rate (IDSR) is a conditional or distributional notion of success in which performance is measured not over an undifferentiated evaluation set, but over samples, variants, or regions that are explicitly treated as “in-distribution.” Taken together, recent work suggests that IDSR is best understood not as a single standardized metric, but as a family of operational definitions whose meaning depends on how “in-distribution” is specified: by energy-based inlier selection in imbalanced semi-supervised learning, by an explicit distribution over attack variants in jailbreak evaluation, or by congruent regions of the feature space identified through feature-wise conformal intervals (Yu et al., 2023, Maple et al., 9 May 2026, Seedat et al., 2022).

1. Conceptual scope

Across the cited literature, “in-distribution” is not a universal primitive. In "InPL: Pseudo-labeling the Inliers First for Imbalanced Semi-supervised Learning" (Yu et al., 2023), it denotes unlabeled samples that are close to the current training distribution and whose energy is below a threshold. In "Single-Configuration Attack Success Rate Is Not Enough: Jailbreak Evaluations Should Report Distributional Attack Success" (Maple et al., 9 May 2026), the relevant object is a distribution over attack variants, so “in-distribution success” becomes success under an explicit variant distribution rather than under a single hand-picked configuration. In "Data-SUITE: Data-centric identification of in-distribution incongruous examples" (Seedat et al., 2022), the emphasis is on regions inside the training distribution that are congruent with the training set and therefore likely to be predicted reliably.

Setting	In-distribution criterion	Success quantity
InPL	$$E(\omega(\mathbf{x}), f(\omega(\mathbf{x}))) < \tau_e$$	Correct pseudo-label or correct final prediction
Jailbreak evaluation	Variant drawn from an explicit tested variant space	Mean ASR, contextualized by VSM and UC
Data-SUITE	Congruent or certain region defined by feature-wise CIs	Conditional downstream accuracy

This suggests that IDSR is a structurally recurring idea: first define the subset or distribution that counts as in-distribution, then compute success conditional on that definition. The technical content therefore lies less in the name “IDSR” than in the mechanism used to determine in-distribution membership.

2. Energy-defined IDSR in imbalanced semi-supervised learning

InPL reinterprets pseudo-labeling as an evolving in-distribution versus out-of-distribution classification problem (Yu et al., 2023). Initially, only the ground-truth human-labeled samples are considered in-distribution. During training, unlabeled samples are admitted into the in-distribution set if they are close to the current training distribution formed by the labeled and previously pseudo-labeled data. The paper is explicit that this use of “out-of-distribution” differs from standard OOD detection: a sample can be OOD for the current iteration even if it belongs to a known semantic class, provided that it is far from the current training data.

The operational mechanism is an energy score: $$E(\mathbf{x}, f(\mathbf{x})) = -T \cdot \log \Big( \sum_{i=1}^K e^{f_i(\mathbf{x}) / T} \Big),$$ with $K$ classes and temperature $T > 0$, typically $T=1$. Lower energy indicates that a sample is more likely to be in-distribution; higher energy indicates that it is more likely to be out-of-distribution. For an unlabeled sample $\mathbf{x}_b$, InPL computes the energy on the weakly augmented view $\omega(\mathbf{x}_b)$ and applies the indicator

$$\mathbb{1}\big[ E(\omega(\mathbf{x}_b), f(\omega(\mathbf{x}_b))) < \tau_e \big].$$

This indicator is the exact operational definition of an in-distribution unlabeled sample in the algorithm.

The SSL objective retains the standard supervised term

$$\mathcal{L}_s = \frac{1}{B_s} \sum_{b=1}^{B_s} \mathcal{H} \big( \mathbf{y}_b, \, p(\mathbf{y} \mid \omega(\mathbf{x}_b)) \big),$$

and uses an unsupervised term restricted to energy-selected inliers: $$\mathcal{L}_u = \frac{1}{B_u} \sum_{b=1}^{B_u} \mathbb{1} \big[ E(\omega(\mathbf{x}_b), f(\omega(\mathbf{x}_b))) < \tau_e \big]\, \mathcal{H} \big( \hat{p}(\mathbf{y} \mid \omega(\mathbf{x}_b)), \, p(\mathbf{y} \mid \Omega(\mathbf{x}_b)) \big),$$ with overall loss

$$E(\mathbf{x}, f(\mathbf{x})) = -T \cdot \log \Big( \sum_{i=1}^K e^{f_i(\mathbf{x}) / T} \Big),$$0

An optional Adaptive Margin Loss (AML) can replace the cross-entropy in the unsupervised term: $$E(\mathbf{x}, f(\mathbf{x})) = -T \cdot \log \Big( \sum_{i=1}^K e^{f_i(\mathbf{x}) / T} \Big),$$1 where $$E(\mathbf{x}, f(\mathbf{x})) = -T \cdot \log \Big( \sum_{i=1}^K e^{f_i(\mathbf{x}) / T} \Big),$$2 is a class-specific margin based on an EMA of predicted probabilities.

Within this framework, a natural training-time IDSR is the fraction of correctly pseudo-labeled samples among those deemed in-distribution by the energy rule: $$E(\mathbf{x}, f(\mathbf{x})) = -T \cdot \log \Big( \sum_{i=1}^K e^{f_i(\mathbf{x}) / T} \Big),$$3

$$E(\mathbf{x}, f(\mathbf{x})) = -T \cdot \log \Big( \sum_{i=1}^K e^{f_i(\mathbf{x}) / T} \Big),$$4

The paper’s own pseudo-label precision is exactly this quantity restricted to the energy-selected inliers, while pseudo-label recall measures how many ground-truth examples, including head, body, and tail subsets, are both admitted into the in-distribution set and correctly labeled.

The empirical significance is that energy-based inlier selection improved pseudo-label quality and long-tailed performance. The paper reports around $$E(\mathbf{x}, f(\mathbf{x})) = -T \cdot \log \Big( \sum_{i=1}^K e^{f_i(\mathbf{x}) / T} \Big),$$5 absolute accuracy improvement on CIFAR10-LT, and in one challenging scenario InPL achieves a $$E(\mathbf{x}, f(\mathbf{x})) = -T \cdot \log \Big( \sum_{i=1}^K e^{f_i(\mathbf{x}) / T} \Big),$$6 accuracy improvement over the best competitor. It also reports higher overall pseudo-label precision and substantially higher tail-class recall, interpreted in the paper as improved reliability among the low-energy inliers (Yu et al., 2023).

3. Distributional IDSR in jailbreak evaluation

The jailbreak evaluation paper argues that reporting ASR for a single attack configuration is insufficient because parameterized attacks exhibit substantial variation over internal templates, rounds, dispersion, or teaching shots (Maple et al., 9 May 2026). The core proposal is distributional reporting over a tested variant space $$E(\mathbf{x}, f(\mathbf{x})) = -T \cdot \log \Big( \sum_{i=1}^K e^{f_i(\mathbf{x}) / T} \Big),$$7, rather than exclusive reliance on a single best-case configuration.

For each attack variant $$E(\mathbf{x}, f(\mathbf{x})) = -T \cdot \log \Big( \sum_{i=1}^K e^{f_i(\mathbf{x}) / T} \Big),$$8, let $$E(\mathbf{x}, f(\mathbf{x})) = -T \cdot \log \Big( \sum_{i=1}^K e^{f_i(\mathbf{x}) / T} \Big),$$9 denote the set of prompts successfully jailbroken from a prompt set $K$0. The per-variant ASR is

$K$1

The paper then defines the Variant Sensitivity Measure: $K$2 where Best ASR is the maximum ASR across the tested variant space and Mean ASR is the average ASR across the same space. Union Coverage is

$K$3

VSM measures how unrepresentative the best configuration is relative to typical performance; UC measures the fraction of prompts that can be jailbroken by at least one tested variant.

A plausible IDSR interpretation in this setting is the expected success rate under an explicit variant distribution. The paper effectively instantiates this as Mean ASR under a uniform distribution over the enumerated variant grid. It also presents a more general formalization: $K$4 where $K$5 is a distribution over variants and $K$6 is a distribution over prompts. Under the paper’s protocol, $K$7 is induced by 100 HarmBench prompts chosen by stratified sampling across 7 harm categories, and $K$8 is uniform over the tested configurations.

The experimental spaces are finite grids. For PAIR, the paper evaluates 3 system prompt templates and 3 conversation depths, yielding 9 variants per target model. For bijection, it evaluates a $K$9 grid over fixed-size dispersion $T > 0$0 and teaching shots $T > 0$1, yielding 36 variants per target model. These enumerations allow direct computation of Best ASR, Mean ASR, VSM, and UC.

The numerical results show pronounced gaps between best-case and distributional success (Maple et al., 9 May 2026). For PAIR on Mistral-7B, the best template at $T > 0$2 reaches $T > 0$3 ASR, Mean ASR across all 9 configurations is $T > 0$4, VSM is $T > 0$5, and UC is $T > 0$6. For PAIR on Qwen3-0.6B, the corresponding values are $T > 0$7, $T > 0$8, $T > 0$9, and $T=1$0. For bijection on Mistral-7B, Best ASR is $T=1$1, Mean ASR across 36 variants is $T=1$2, VSM is $T=1$3, and UC is $T=1$4. On Qwen3-0.6B, Best ASR is $T=1$5, Mean ASR $T=1$6, VSM $T=1$7, and UC $T=1$8; on Llama-3.1-8B-Instruct, Best ASR is $T=1$9, Mean ASR $\mathbf{x}_b$0, VSM $\mathbf{x}_b$1, and UC $\mathbf{x}_b$2. These results motivate a distributional notion of in-distribution success in which the relevant quantity is typical success over variants, not merely the extremal configuration.

4. Congruence-conditioned IDSR in data-centric reliability analysis

Data-SUITE addresses heterogeneous regions inside the training distribution rather than standard OOD data (Seedat et al., 2022). The paper calls these regions “in-distribution incongruous examples” and explicitly notes that their flagged instances are not standard OOD points: overlap with OOD detectors is reported as $\mathbf{x}_b$3–$\mathbf{x}_b$4, and OOD scores are low. The objective is to identify regions where a downstream model trained on the training set will or will not be reliably predictive, independently of any task-specific model used to define the regions.

The formal setting assumes feature space

$\mathbf{x}_b$5

with training data

$\mathbf{x}_b$6

and future points in the same feature space. Data-SUITE constructs feature-wise confidence intervals via a three-stage pipeline: a copula-based generator that augments the training data using vine copulas, a PCA-based representer that maps $\mathbf{x}_b$7 to a latent space $\mathbf{x}_b$8, and a conformal predictor that produces adaptive feature-wise confidence intervals for each test instance.

For each feature $\mathbf{x}_b$9, a regressor $\omega(\mathbf{x}_b)$0 predicts $\omega(\mathbf{x}_b)$1 from the latent representation, and a normalizer $\omega(\mathbf{x}_b)$2 estimates residual scale. The normalized nonconformity score is

$\omega(\mathbf{x}_b)$3

Given the calibration quantile $\omega(\mathbf{x}_b)$4, the adaptive interval is

$\omega(\mathbf{x}_b)$5

Data-SUITE then defines feature-level and instance-level inconsistency: $\omega(\mathbf{x}_b)$6 with an instance called inconsistent if $\omega(\mathbf{x}_b)$7, and the paper uses $\omega(\mathbf{x}_b)$8. It also defines normalized uncertainty: $\omega(\mathbf{x}_b)$9 Instances are sorted by $$\mathbb{1}\big[ E(\omega(\mathbf{x}_b), f(\omega(\mathbf{x}_b))) < \tau_e \big].$$0; the most certain region consists of low-$$\mathbb{1}\big[ E(\omega(\mathbf{x}_b), f(\omega(\mathbf{x}_b))) < \tau_e \big].$$1 points, while high-$$\mathbb{1}\big[ E(\omega(\mathbf{x}_b), f(\omega(\mathbf{x}_b))) < \tau_e \big].$$2 or high-$$\mathbb{1}\big[ E(\omega(\mathbf{x}_b), f(\omega(\mathbf{x}_b))) < \tau_e \big].$$3 points form incongruous regions.

Within this framework, a candidate IDSR is conditional downstream accuracy on a congruent region: $$\mathbb{1}\big[ E(\omega(\mathbf{x}_b), f(\omega(\mathbf{x}_b))) < \tau_e \big].$$4 for example with

$$\mathbb{1}\big[ E(\omega(\mathbf{x}_b), f(\omega(\mathbf{x}_b))) < \tau_e \big].$$5

or with a combined condition using $$\mathbb{1}\big[ E(\omega(\mathbf{x}_b), f(\omega(\mathbf{x}_b))) < \tau_e \big].$$6 and $$\mathbb{1}\big[ E(\omega(\mathbf{x}_b), f(\omega(\mathbf{x}_b))) < \tau_e \big].$$7. The paper does not name this quantity IDSR, but it computes the required ingredients directly. On real datasets, it reports

$$\mathbb{1}\big[ E(\omega(\mathbf{x}_b), f(\omega(\mathbf{x}_b))) < \tau_e \big].$$8

where $$\mathbb{1}\big[ E(\omega(\mathbf{x}_b), f(\omega(\mathbf{x}_b))) < \tau_e \big].$$9 is the most certain $$\mathcal{L}_s = \frac{1}{B_s} \sum_{b=1}^{B_s} \mathcal{H} \big( \mathbf{y}_b, \, p(\mathbf{y} \mid \omega(\mathbf{x}_b)) \big),$$0 fraction and $$\mathcal{L}_s = \frac{1}{B_s} \sum_{b=1}^{B_s} \mathcal{H} \big( \mathbf{y}_b, \, p(\mathbf{y} \mid \omega(\mathbf{x}_b)) \big),$$1 the most uncertain $$\mathcal{L}_s = \frac{1}{B_s} \sum_{b=1}^{B_s} \mathcal{H} \big( \mathbf{y}_b, \, p(\mathbf{y} \mid \omega(\mathbf{x}_b)) \big),$$2 fraction. The reported MPI values are $$\mathcal{L}_s = \frac{1}{B_s} \sum_{b=1}^{B_s} \mathcal{H} \big( \mathbf{y}_b, \, p(\mathbf{y} \mid \omega(\mathbf{x}_b)) \big),$$3 on SEER–CUTRACT, $$\mathcal{L}_s = \frac{1}{B_s} \sum_{b=1}^{B_s} \mathcal{H} \big( \mathbf{y}_b, \, p(\mathbf{y} \mid \omega(\mathbf{x}_b)) \big),$$4 on Adult, and $$\mathcal{L}_s = \frac{1}{B_s} \sum_{b=1}^{B_s} \mathcal{H} \big( \mathbf{y}_b, \, p(\mathbf{y} \mid \omega(\mathbf{x}_b)) \big),$$5 on Electricity, all higher than the baselines in the study. In synthetic experiments, the top-100 most certain instances identified by Data-SUITE produce MSE values close to the clean training MSE, while the explicitly inconsistent subset has much higher MSE. This empirically supports the interpretation of congruent-region conditional accuracy as an in-distribution success rate (Seedat et al., 2022).

5. Formal variants of IDSR

The literature therefore supports several mathematically distinct but structurally parallel forms of IDSR.

In subset-conditioned pseudo-labeling, success is correctness among energy-selected inliers: $$\mathcal{L}_s = \frac{1}{B_s} \sum_{b=1}^{B_s} \mathcal{H} \big( \mathbf{y}_b, \, p(\mathbf{y} \mid \omega(\mathbf{x}_b)) \big),$$6 In distributional attack evaluation, success is expectation over a prompt distribution and a variant distribution: $$\mathcal{L}_s = \frac{1}{B_s} \sum_{b=1}^{B_s} \mathcal{H} \big( \mathbf{y}_b, \, p(\mathbf{y} \mid \omega(\mathbf{x}_b)) \big),$$7 In data-centric reliability analysis, success is conditional model accuracy over a congruent region: $$\mathcal{L}_s = \frac{1}{B_s} \sum_{b=1}^{B_s} \mathcal{H} \big( \mathbf{y}_b, \, p(\mathbf{y} \mid \omega(\mathbf{x}_b)) \big),$$8

These forms differ in what is being conditioned on. InPL conditions on low-energy inliers admitted during training. The jailbreak framework conditions on a tested distribution over attack configurations. Data-SUITE conditions on feature-space congruence relative to the training set. The common pattern is that raw aggregate performance is replaced by success restricted to a subset, region, or distribution that has been formally designated as in-distribution.

This suggests that IDSR is most useful when the evaluation question is not merely “how often does the system succeed?” but rather “how often does it succeed on the subset that the methodology itself judges to be within scope?” In pseudo-labeling, that scope is the inlier set. In jailbreak evaluation, it is the attack family under an explicit distribution over variants. In data-centric reliability analysis, it is the congruent region where feature-wise conformal structure matches the training distribution.

6. Interpretation, misconceptions, and limitations

A recurrent misconception is that “in-distribution” always means membership in known semantic classes or conventional train-test support. The cited work shows otherwise. In InPL, an unlabeled point may be treated as OOD for the current iteration even if it belongs to a known class, because “OOD” there means far from the current training data rather than semantically novel (Yu et al., 2023). In Data-SUITE, the focus is on incongruous regions inside the global training distribution rather than on classic OOD detection, and the paper reports only $$\mathcal{L}_s = \frac{1}{B_s} \sum_{b=1}^{B_s} \mathcal{H} \big( \mathbf{y}_b, \, p(\mathbf{y} \mid \omega(\mathbf{x}_b)) \big),$$9–$$\mathcal{L}_u = \frac{1}{B_u} \sum_{b=1}^{B_u} \mathbb{1} \big[ E(\omega(\mathbf{x}_b), f(\omega(\mathbf{x}_b))) < \tau_e \big]\, \mathcal{H} \big( \hat{p}(\mathbf{y} \mid \omega(\mathbf{x}_b)), \, p(\mathbf{y} \mid \Omega(\mathbf{x}_b)) \big),$$0 overlap with OOD detectors (Seedat et al., 2022). In jailbreak evaluation, the relevant distribution is not over data points alone but over parameterized attack variants, so “in-distribution success” concerns typical performance across a variant space rather than success of a single tuned configuration (Maple et al., 9 May 2026).

Another misconception is that a single headline score is sufficient. The jailbreak paper explicitly rejects this position, arguing that Best ASR alone hides both representativeness and missed attack surface. VSM quantifies the gap between best and mean performance, and UC quantifies the total fraction of prompts reachable by any tested configuration. High VSM indicates that the reported best-case result is not representative of typical behavior; high UC indicates that single-configuration evaluation understates total coverage of the vulnerable prompt set (Maple et al., 9 May 2026).

The methodologies also impose distinct limitations. In InPL, the authors note that energy scores are unbounded and their scale depends on architecture and number of classes, making thresholds less interpretable; energy-based selection can still misclassify some OOD points as inliers or delay inclusion of difficult inliers (Yu et al., 2023). In the jailbreak setting, the evaluated variant space is only a finite approximation to the true space of attack configurations, and Mean ASR depends on the chosen distribution over variants, which the paper takes to be uniform over the tested grid (Maple et al., 9 May 2026). In Data-SUITE, conformal validity requires exchangeability, proper training and calibration splits, and adequate representation quality; strong violations of these assumptions can degrade the relation between congruence and downstream reliability (Seedat et al., 2022).

Taken together, these limitations imply that IDSR is not an absolute property of a model or method. It is a conditional statistic whose value depends on the choice of in-distribution criterion, thresholding rule, variant distribution, calibration assumptions, and evaluation subset. Its technical usefulness lies precisely in making those conditions explicit.