Repeatability and Sensitivity Metrics

• Repeatability and sensitivity metrics are quantitative measures assessing consistency in repeated tests and the influence of parameter variations on outcomes.
• They utilize statistical tools like the coefficient of variation, intraclass correlation, and rank-based methods to ensure precision and reliable reproducibility across domains.
• These metrics guide experimental design improvements, enabling calibration of models and robust benchmarking in fields from imaging to nuclear astrophysics.

Repeatability and sensitivity metrics are quantitative constructs central to experimental design, model validation, measurement assurance, and reproducibility assessment across scientific domains. Repeatability expresses the degree to which repeated measurements under controlled, invariant conditions yield consistent results, while sensitivity gauges the impact of deliberate or incidental variations on measurement outcomes—effectively quantifying a system's susceptibility to perturbation, parameter change, or noise. These metrics are mathematically formalized in diverse contexts spanning metrology, machine learning, imaging sciences, nuclear astrophysics, acoustics, and beyond. Their rigorous definition, proper computation, and context-appropriate interpretation are essential for reliable scientific inference and robust system development.

1. Fundamental Definitions and Mathematical Formalism

Repeatability, in the metrological sense, is the precision attained by repeating measurements under invariant conditions—identical equipment, samples, operators, environmental states, computational random seeds, etc. In quantitative terms, it is typically expressed via the coefficient of variation (CV), intraclass correlation coefficient (ICC), or related variance partitioning statistics. Formally, for a series of $n$ measurements $v_1, \ldots, v_n$:

$$\mu = \frac{1}{n} \sum_{i=1}^n v_i,\quad s^* = \sqrt{\frac{1}{n-1} \sum_{i=1}^n (v_i-\mu)^2},\quad \mathrm{CV}^* = \left(1+\frac1{4n}\right) \frac{s^*}{\mu}$$

where $CV^*$ corrects for small sample bias (Belz, 2021).

Sensitivity, variously termed reproducibility under altered conditions, quantifies the variation in primary measurements when one or more contextual or experimental parameters change. The mathematical form often mirrors that of repeatability, but the set $\{v_i'\}$ is obtained under distinct, intentionally perturbed conditions (e.g., changed data split, hardware, prompt phrasing, quantization scheme, or laboratory):

$$\mu' = \frac{1}{n} \sum_{i=1}^n v_i',\quad s'^* = \sqrt{\frac{1}{n-1} \sum_{i=1}^n (v_i'-\mu')^2},\quad \mathrm{CV'}^* = \left(1+\frac1{4n}\right) \frac{s'^*}{\mu'}$$

In binary settings with hierarchical variance sources (e.g., inter- and intra-laboratory), the beta-binomial model provides closed-form decompositions for repeatability and reproducibility variances (Takeshita et al., 2020).

2. Statistical Metrics for Repeatability and Sensitivity

A variety of statistical metrics are utilized to characterize repeatability and sensitivity in both univariate and multivariate contexts:

• Coefficient of Variation ($\mathrm{CV}^*$): Normalized measure of dispersion, robust to mean scaling, directly interpretable as percent variation.
• Intraclass Correlation Coefficient (ICC): Fraction of total variance attributable to between-entity (e.g., subject, lab, metric) versus within-entity sources. For repeated measures $x_{it}$,

$$\mathrm{ICC} = \frac{\sigma^2_{\text{between}}}{\sigma^2_{\text{between}} + \sigma^2_{\text{error}}}$$

ICC is central in imaging (Schwier et al., 2018), biomechanics (Reynard et al., 2013), and large-scale ML benchmarking (Gonzalez et al., 28 Sep 2025).

• Bland–Altman SD of differences ($s_d$): Characterizes the “noise floor” between test–retest pairs in longitudinal or imaging studies, yielding limits of agreement that directly quantify tolerable measurement error (Desseroit et al., 2016).
• Discriminability ($v_1, \ldots, v_n$0), Rank Sums, and Fingerprinting: Multivariate and nonparametric extensions (e.g., for high-dimensional, non-Gaussian data). Discriminability is strictly monotonic with ICC under univariate Gaussian models but provides robust power and invariance under broader settings (Wang et al., 2020).
• Custom Measures: Domain-specific constructs include Fractal Dimension repeatability noise parameter $v_1, \ldots, v_n$1 in retinal imaging (Engelmann et al., 2024), log-ratio sensitivity metrics in $v_1, \ldots, v_n$2-process nucleosynthesis (Shand et al., 2017), and prompted-LLM sensitivity/consistency coefficients (Errica et al., 2024).

3. Contextual Applications and Domain-Specific Implementations

Machine Learning and NLP

Repeatability quantifies run-to-run metric stability under fixed seeds, splits, and hardware, supporting trustworthy model comparison and leaderboard construction. Sensitivity/reproducibility is measured as the increase in metric variance when perturbing seeds, hardware, or input formulations. Empirically, CV$v_1, \ldots, v_n$3 values $v_1, \ldots, v_n$42% indicate high repeatability, with sensitivity analyses revealing metric inflation as more sources of randomness or uncertainty are introduced (Belz, 2021, Gonzalez et al., 28 Sep 2025).

Medical and Scientific Imaging

Test–retest designs undergird assessments of radiomic feature stability (Desseroit et al., 2016, Schwier et al., 2018, Aggarwal et al., 2023) and deep-learning–derived biomarkers (Engelmann et al., 2024). ICC and CV provide population-level benchmarks; per-sample repeatability is often operationalized as the ratio between within-unit and between-unit SD, e.g., median $v_1, \ldots, v_n$5 of 3.55% for a robust retinal FD estimation algorithm (Engelmann et al., 2024). Processing parameter sensitivity is critical: small changes in normalization, bin width, or filtering can shift ICC by $v_1, \ldots, v_n$6, underlining pipeline transparency demands (Schwier et al., 2018).

Physical Sciences and Metrology

Acoustic standards (e.g., ISO 3382-3) employ within- and between-path SDs, coefficients of variation, repeatability coefficients (limits for 95% agreement), and mixed-effects ICC for reliability; directional and path-to-path sensitivity is made explicit by contrasting Type 1 (repeatability) and Type 2 (sensitivity) protocols (Yadav et al., 2023). Nuclear astrophysics leverages normalized, scale-free metrics and explicit minimization procedures to enable cross-study comparison of sensitivity factors (Shand et al., 2017).

Binary Classification in Collaborative Laboratories

The beta-binomial model decomposes variance into within- and between-lab components for sensitivity and repeatability, yielding analytical estimators robust to sample size and permitting exact testing for laboratory (source) effects (Takeshita et al., 2020).

4. Best Practices and Methodological Recommendations

The literature converges on several domain-agnostic principles:

• Always report both repeatability and sensitivity/reproducibility values. This includes providing point estimates (e.g., mean, ICC, CV) as well as confidence limits.
• Perform protocol-specific normalization and minimization. For metrics with arbitrary scales, such as sum-of-abundance shifts, minimize over scale ratios to yield baseline-independent, comparably interpretable factors (Shand et al., 2017).
• Quantify and report all relevant variance sources. This spans run-to-run noise, hardware or algorithmic changes, sample-path effects, and processing configuration shifts (Gonzalez et al., 28 Sep 2025, Belz, 2021, Yadav et al., 2023).
• Adopt nonparametric or robust estimators in high-dimensional or non-Gaussian contexts. Discriminability or rank-sum–based statistics maintain consistency and test power where parametric models may mislead (Wang et al., 2020).
• Provide detailed documentation of all experimental/processing variables. Variations in data handling, filtering, quantization, or prompt engineering can obfuscate true repeatability and sensitivity without cautious transparency (Schwier et al., 2018, Errica et al., 2024).
• Use simulation or permutation testing to directly assess test power and batch effect robustness. Explicit statistical power analysis is critical for determining whether a metric can reliably detect the effect sizes of interest (Wang et al., 2020).

5. Comparison of Metric Properties, Strengths, and Limitations

Metric	Scale-Free	Handles Multivariate	Robust to Outliers/Batch	Domain Applicability
CV / CV*	✓	✓ (per-feature)	–	General (ML, imaging, physics)
ICC	✓	I2C2 variant	–	Imaging, ML, collaborative studies
Bland–Altman SD	✓	–	–	Imaging, clinical measurement
Discriminability ($v_1, \ldots, v_n$7)	✓	✓	Partial (robust $v_1, \ldots, v_n$8)	fMRI, genomics, non-Gaussian data
Rank Sums/Fingerprint	Partial	✓	Fingerprint: no	Nonparametric inference
$v_1, \ldots, v_n$9 (FD repeat.)	✓	–	–	Retinal oculomics
Prompt Sensitivity/Consistency	✓	N/A	–	LLMs, prompt engineering

Limitations include sensitivity of ICC to violation of ANOVA assumptions, loss of interpretability of nonparametric indices, and—particularly in the case of pixel-based metrics—potential inflation by spurious “repeatable” events unless ground-truthing is strictly enforced (e.g., via 3D virtual pre-selection (Lang et al., 2019)).

6. Domain-Specific Sensitivity Assessments and Calibration

Robust application of repeatability and sensitivity metrics requires context-specific calibration. In radiomics, maintaining feature SD$$\mu = \frac{1}{n} \sum_{i=1}^n v_i,\quad s^* = \sqrt{\frac{1}{n-1} \sum_{i=1}^n (v_i-\mu)^2},\quad \mathrm{CV}^* = \left(1+\frac1{4n}\right) \frac{s^*}{\mu}$$0 the volume repeatability SD is deemed “very reliable” (Desseroit et al., 2016). In low-field MRI, SNR CV$$\mu = \frac{1}{n} \sum_{i=1}^n v_i,\quad s^* = \sqrt{\frac{1}{n-1} \sum_{i=1}^n (v_i-\mu)^2},\quad \mathrm{CV}^* = \left(1+\frac1{4n}\right) \frac{s^*}{\mu}$$1 is considered “high repeatability,” while geometric distortion CV$$\mu = \frac{1}{n} \sum_{i=1}^n v_i,\quad s^* = \sqrt{\frac{1}{n-1} \sum_{i=1}^n (v_i-\mu)^2},\quad \mathrm{CV}^* = \left(1+\frac1{4n}\right) \frac{s^*}{\mu}$$2 after phase correction is the target for field homogeneity (Aggarwal et al., 2023). For LLM benchmarks, empirical evidence shows that two stochastic runs are required to suppress $$\mu = \frac{1}{n} \sum_{i=1}^n v_i,\quad s^* = \sqrt{\frac{1}{n-1} \sum_{i=1}^n (v_i-\mu)^2},\quad \mathrm{CV}^* = \left(1+\frac1{4n}\right) \frac{s^*}{\mu}$$383% of pairwise leaderboard rank flips compared to single-run evaluation; averaging three runs offers marginal SE shrinkage but full stability (Gonzalez et al., 28 Sep 2025).

7. Emerging Tools, Future Directions, and Open Challenges

Recent literature emphasizes the need for:

• Automated, experiment-style evaluation protocols in ML and scientific workflows—elevating statistical variance decomposition and reporting to first-class status in benchmark design (Gonzalez et al., 28 Sep 2025).
• Robust, scale-invariant metrics for comparative studies—enabling cross-experiment and cross-center inferences through explicit normalization/calibration methodologies (Shand et al., 2017, Yadav et al., 2023).
• Transparency and code/data availability in imaging and radiomics, with an insistence on full documentation of preprocessing and feature extraction pipelines given high parameter sensitivity (Schwier et al., 2018).
• Sensitivity diagnostics for model development in LLM and CV fields, with entropy- and distributional-metrics tracing model robustness to prompt and pipeline perturbations (Errica et al., 2024, Lang et al., 2019).
• Targeted sample sizing and power analysis to ensure that repeatability and sensitivity metrics attain desired confidence and detection thresholds—even for moderate sample sizes (Yadav et al., 2023, Wang et al., 2020).

Persistent challenges include the harmonization of repeatability and sensitivity standards across fields, selection of task-appropriate metric thresholds, and development of easily interpretable yet generalizable performance indices under real-world nonideality.

---

In summary, repeatability and sensitivity metrics constitute foundational statistical tools indispensable for robust, transparent, and reproducible scientific practice. Their rigorous application mandates thoughtful attention to domain idiosyncrasies, benchmarking protocol, and multifactor variance decomposition, supported by explicit reporting and transparent data/method sharing. As measurement complexity and data heterogeneity increase, these metrics—and the experimental sophistication underlying them—are central to meaningful inference and scientific progress (Shand et al., 2017, Reynard et al., 2013, Belz, 2021, Yadav et al., 2023, Wang et al., 2020, Lang et al., 2019, Schwier et al., 2018, Engelmann et al., 2024, Errica et al., 2024, Gonzalez et al., 28 Sep 2025, Desseroit et al., 2016, Takeshita et al., 2020, Aggarwal et al., 2023).