Data Reliability Index (DRI)

• Data Reliability Index (DRI) is a data-centric measure that evaluates a model’s per-prediction trustworthiness based on local representation and stability.
• DRI computes reliability by combining k-nearest neighbor distances and local uncertainty metrics into SRU and WRU indices for each query point.
• The framework offers scalable, inference-time risk-of-error estimation and outperforms traditional model-centric uncertainty methods.

Searching arXiv for papers on “Data Reliability Index” and closely related DRI usages. The Data Reliability Index (DRI), in the machine-learning sense formalized by "Reliability Evaluation of Individual Predictions: A Data-centric Approach" (Shahbazi et al., 2022), is a family of data-centric reliability measures for assessing whether a model trained on a given dataset is fit for making a specific prediction. Its central premise is that expected predictive performance over a data distribution does not by itself establish reliability for an individual query point. The framework therefore evaluates reliability from the training data rather than from the model’s internal confidence, using two criteria: whether the query is sufficiently represented by similar training instances, and whether the target behavior in the query’s neighborhood is locally stable or highly fluctuating (Shahbazi et al., 2022). The acronym is not uniform across research areas: in one distribution-grid paper the “Data Reliability Index” is not specifically defined, despite extensive use of reliability indices (Karngala et al., 2023), while another literature uses DRI to denote the Deliberative Reason Index (Veri, 18 Apr 2026).

1. Core definition and scope

The DRI framework addresses a specific problem in predictive inference: a model with high accuracy may or may not be reliable for predicting an individual query point. Instead of treating reliability as a property fully determined by the predictor, the framework asks whether the training data place the query in a region that is both well represented and locally coherent. The paper states that a model’s prediction is not reliable if (i) there were not enough similar instances in the training set to the query point, and (ii) if there is a high fluctuation (uncertainty) in the vicinity of the query point in the training set (Shahbazi et al., 2022).

This definition makes DRI explicitly data-centric. It is therefore distinct from model-centric approaches such as conformal predictions, probabilistic predictions, and prediction intervals, which rely on the model’s own notion of certainty, and distinct again from explainability methods that seek post hoc explanations of individual predictions. The DRI formulation is instead anchored in neighborhood structure and local target variability in the training set.

A consequential feature of this framing is that DRI is not a single global model score. It is computed per query point. A plausible implication is that DRI is best understood as a per-prediction reliability or distrust measure that complements, rather than replaces, standard aggregate evaluation.

2. Two constituent criteria: representation and local uncertainty

The first constituent of DRI is Instance Similarity (Representation), formalized through a measure of lack of representation denoted $P_o(q)$. For a query $q$, the framework computes the radius of the $k$-nearest-neighbor ball,

$\rho_q = \Delta_k(q, D),$

where $\Delta$ is typically Euclidean distance. A large $\rho_q$ indicates that $q$ lies in a sparse region of the training set and is therefore comparatively outlying (Shahbazi et al., 2022).

To quantify how extreme that sparsity is, the radius is ranked against the multiset of $k$-NN radii of all training points: $$r_q = \frac{|\{r \in \Gamma_D: r \leq \Delta_k(q,D)\}|}{n},$$ with $n = |D|$. Given an expected outlier ratio $q$0 and standard deviation $q$1, the rank is mapped through the standard Normal CDF $q$2: $q$3 The resulting quantity increases with how extreme the query’s neighborhood radius is relative to the training distribution.

The second constituent is Local Uncertainty, denoted $q$4. Its operationalization depends on task type. For classification, the framework uses Shannon entropy in the $q$5-NN neighborhood: $q$6 where $q$7 is the proportion of label $q$8 among the $q$9-nearest neighbors. For regression, it uses residual sum of squares within the neighborhood: $k$0 where $k$1 is the mean target value in the neighborhood (Shahbazi et al., 2022).

As with representation, local uncertainty is ranked against the training set: $k$2 and mapped via

$k$3

where $k$4 is an expected uncertainty ratio and $k$5 is either estimated from data or set by experts. This constructs a probability-like quantity expressing how unusual the query’s local uncertainty is relative to the training data.

3. Aggregation into the DRI family: SRU and WRU

The paper synthesizes $k$6 and $k$7 into two indices: the Strong Reliability/Unreliability Index (SRU) and the Weak Reliability/Unreliability Index (WRU) (Shahbazi et al., 2022). Their definitions are

$k$8

and

$k$9

The distinction is logical as well as numerical. SRU signals unreliability only when both lack of representation and high local uncertainty are substantial. WRU is more permissive: it signals unreliability when either condition is substantial. Both indices take values in $\rho_q = \Delta_k(q, D),$0, and the paper states that they can be interpreted as the probability that the prediction at $\rho_q = \Delta_k(q, D),$1 is unreliable due to training data limitations (Shahbazi et al., 2022).

This construction makes explicit that DRI is not merely an outlier detector. Sparse regions and unstable regions are treated separately and then combined. A query may be well represented but locally inconsistent, or underrepresented but locally homogeneous. The SRU/WRU pair preserves that distinction and offers different operating points for conservative versus inclusive screening of risky predictions.

4. Computational procedure and inference-time deployment

The framework is designed for scalable inference. Its computational core begins with $\rho_q = \Delta_k(q, D),$2-NN index construction, using a fast data structure such as a KD-tree, ball tree, or approximate indices for efficient nearest-neighbor queries. It then performs preprocessing by precomputing and sorting arrays of $\rho_q = \Delta_k(q, D),$3-NN radii $\rho_q = \Delta_k(q, D),$4 and local uncertainty values $\rho_q = \Delta_k(q, D),$5, so that rank-based probability mapping can be carried out by binary search (Shahbazi et al., 2022).

At query time, the workflow is straightforward. For a query $\rho_q = \Delta_k(q, D),$6, the method finds the $\rho_q = \Delta_k(q, D),$7-nearest neighbors, computes the neighborhood radius and the entropy or RSS, locates the corresponding quantiles in the precomputed arrays, maps them through the standard Normal CDF, and combines the two probabilities into SRU or WRU. The reported complexities are preprocessing time $\rho_q = \Delta_k(q, D),$8, which can be done offline, and query time $\rho_q = \Delta_k(q, D),$9 (Shahbazi et al., 2022).

The paper also introduces a no-data-access estimator for settings in which the original data cannot be consulted at inference time, for example because of privacy or size constraints. In this variant, synthetic samples are drawn from the query space; for each sample, the true $\Delta$0-NN radius and local uncertainty are computed using the original data; and two regression models, $\Delta$1 and $\Delta$2, are trained to predict these quantities for new queries. At inference, the regression estimates replace direct $\Delta$3-NN search. The estimator is trained with an exponential search strategy that increases training sample size until the generalization error falls below a user-specified threshold on a held-out set (Shahbazi et al., 2022).

5. Empirical behavior and methodological position

The empirical claim made for DRI is that distrust values correlate strongly and consistently with realized predictive error across multiple real and synthetic datasets, multiple learning algorithms, and both classification and regression tasks (Shahbazi et al., 2022). Test points placed into bins with higher DRI or distrust scores exhibit higher error, including lower accuracy or $\Delta$4 and higher MSE, FPR, and FNR. The paper reports that this pattern holds under different values of $\Delta$5, different distance metrics including Euclidean, Manhattan, and Chebyshev, and different outlier-detection methodologies.

The reported quantitative summary is that the correlation between DRI bins and model error measures is high, with Pearson’s $\Delta$6 on the order of $\Delta$7–$\Delta$8 depending on the setting (Shahbazi et al., 2022). Within the framework’s intended interpretation, this supports the use of DRI as a per-prediction risk-of-error indicator.

The comparison class in the paper is also revealing. Model-centric methods such as conformal prediction, prediction probability, and data coverage are described as either failing to detect unrepresented regions or failing to handle local uncertainty as effectively as DRI (Shahbazi et al., 2022). This does not imply that DRI subsumes all uncertainty quantification, but it does place the framework in a specific methodological niche: it evaluates whether the data support a prediction, rather than whether the model reports high internal confidence.

6. Terminological ambiguity and adjacent research uses

The term Data Reliability Index is not used uniformly across the supplied literature, and disambiguation is often necessary.

Usage	Definition or status	Paper
Data-centric reliability measures for individual predictions	Formalized through $\Delta$9, $\rho_q$0, SRU, and WRU	(Shahbazi et al., 2022)
Distribution-grid reliability study	“The Data Reliability Index is not specifically defined”; indices are SAIFI, SAIDI, AIF, AID, and AENS	(Karngala et al., 2023)
Deliberative Reason Index	Unrelated use of the acronym DRI in deliberative studies	(Veri, 18 Apr 2026)

A related but distinct line of work is dataset-level reliability without ground truth. "Data Reliability Scoring" introduces the Gram determinant score (GDS) for datasets collected from potentially strategic sources, defines ground-truth-based orderings such as exact match ordering, Blackwell dominant ordering, and distance-based (Hamming, $\rho_q$1, etc.) orderings, and proves that GDS is uniquely, up to scaling, experiment agnostic under the stated conditions (Chen et al., 20 Oct 2025). This is a different object of analysis: the unit is the dataset, not the individual prediction.

Another adjacent notion appears in the data integration framework DIRA, where a DRI-like metric is constructed from Fact-Completeness, Validity, Accuracy, and Timeliness, and used for source selection, pruning, and top-$\rho_q$2 ranking via the Threshold Algorithm (TA) (Monem et al., 2016). Here, reliability is tied to source quality and query answering in integration systems, not to the reliability of a single predictive inference.

The most precise usage of Data Reliability Index in the supplied material is therefore the data-centric, per-prediction formulation of (Shahbazi et al., 2022). In that formulation, DRI denotes a principled mechanism for quantifying when a query is underrepresented, locally unstable, or both, and for turning those training-data properties into an operational reliability or distrust score for inference.