Confidence Interval Index

• The Confidence Interval Index is a scalar measure that integrates empirical coverage probability and average interval length to assess estimator performance.
• It rewards methods that achieve near-nominal coverage with short intervals while penalizing deviations using a defined loss function.
• The metric enables quick comparative evaluations in simulation studies, reducing the need for computationally intensive recalibration.

The confidence interval index is a scalar criterion for comparing confidence interval estimators when several interval procedures are available for the same parameter. It was proposed to combine the two standard performance attributes of an interval estimator—empirical coverage probability and average interval length—into a single quantity that is easy to compute and interpret. Its motivating premise is that high coverage probability close to the nominal level and short interval length are simultaneously desirable but structurally opposed, so estimator choice cannot be based on either attribute alone. In the proposed formulation, the index takes values in the neighborhood of the range of the coverage probability, $[0,1]$; values approaching $1$ indicate a good interval estimator, and values approaching $0$ indicate a poor one (Minkah et al., 2017).

1. Motivation and problem setting

In many statistical problems, multiple interval estimators are available for the same inferential target. The conventional criteria for comparison are empirical coverage probability and average interval length. Coverage close to the nominal level is required for inferential validity, whereas shorter intervals are preferred for precision. The conflict is direct: high coverage is often associated with longer intervals, while short intervals often under-cover. Some methods, such as bootstrap calibration, attempt to modify the nominal level so that final selection can be based on interval length alone, but these methods are computationally expensive (Minkah et al., 2017).

The confidence interval index was introduced as an alternative to such calibration-heavy comparisons. Its role is not to replace coverage studies or interval-length reporting, but to compress both dimensions into a standalone metric that can rank candidate procedures without additional resampling or optimization. This suggests a pragmatic use case in simulation studies and methodological reviews, where empirical coverage and average interval length are already available and a single comparative summary is desirable (Minkah et al., 2017).

2. Mathematical definition

For an estimator $j$ among $R$ candidate confidence interval procedures, let $L_j$ denote the average empirical confidence interval length, let $\eta_j$ denote the empirical coverage probability, and let $1-\alpha$ denote the nominal coverage level. The confidence interval index is defined by

$$I(L_j, \eta_j; \alpha) = k_\alpha \left(1 - \frac{1}{2} \left( \frac{1 + H(\eta_j; \alpha)} {1 + \left(\frac{\eta_j}{1 + L_j}\right)} \right)\right)$$

with $L_j \geq 0$, $1$0, loss function

$1$1

and scaling constant

$1$2

The loss term $1$3 penalizes deviation from nominal coverage, while the ratio $1$4 rewards the combination of higher coverage and shorter length. The construction is intended to make the index monotone increasing in coverage when interval length is fixed, and monotone decreasing in interval length when coverage is fixed. The paper also notes that alternative loss functions, such as squared error, may be used, with the corresponding scaling adjusted accordingly (Minkah et al., 2017).

3. Range, limiting behavior, and interpretation

The index was designed so that its values remain close to the familiar $1$5 scale of coverage probabilities. Under the absolute-loss choice above, the limiting behavior is explicit.

Limit	Behaviour	Index value
$1$6	Short interval, poor coverage	$1$7
$1$8	Very wide interval, poor coverage	$1$9
$0$0	Wide interval, correct coverage	$0$1
$0$2	Very short, perfect coverage	$0$3

For a typical choice such as $0$4, $0$5, which yields an approximate range of $0$6. A value near $0$7 corresponds to an interval procedure whose empirical coverage is close to nominal and whose average length is short. Low values indicate a poor compromise, arising either from substantial undercoverage, excessive length, or both. The paper also notes that the index can be linearly transformed to the exact range $0$8 for easier interpretation (Minkah et al., 2017).

The interpretation is intentionally comparative rather than absolute. A high index does not certify optimality in any decision-theoretic sense; rather, it indicates that, among the candidate estimators under study, the procedure achieves a favorable coverage–length compromise. Conversely, a low index can expose methods whose short intervals are obtained at the cost of poor coverage, or whose valid coverage is achieved only through excessive width (Minkah et al., 2017).

4. Theoretical properties and comparative logic

The theoretical rationale of the index is tied to three properties. First, it penalizes deviation from the nominal level through the loss function $0$9. Second, it incorporates interval length in a form that discourages overly wide intervals even when coverage is nominal. Third, it avoids distribution-specific assumptions: the index can be computed for any parameter, estimator, or confidence interval construction method, provided empirical coverage probabilities and average interval lengths are available (Minkah et al., 2017).

This construction places the index in a distinct position relative to calibration-based approaches. Bootstrap calibration attempts to equalize coverage first and then compare methods chiefly by length. The confidence interval index instead accepts the empirical coverage and length as jointly relevant outputs and summarizes them directly. A plausible implication is that the index is especially useful when many methods must be screened quickly, or when computationally expensive recalibration is undesirable.

The proposal is deliberately modular. Because the loss term can be altered, the framework is not tied to the absolute-deviation penalty alone. That flexibility allows different emphases on undercoverage or overcoverage, although the paper’s main development uses the absolute loss because it yields an interpretable scale and tractable limiting behavior (Minkah et al., 2017).

5. Simulation evidence and applied use

A simulation study was conducted to assess the finite-sample performance of the index in three settings: mean estimation from normal and lognormal distributions, binomial proportion estimation, and a practical comparison of coefficient-of-variation interval estimators. For symmetric data such as the normal mean, the indexes of all interval estimators approach $j$0 as sample size increases; for small $j$1, Johnson $j$2 and normal-based intervals perform best. For skewed data such as the lognormal mean, normal-based intervals perform poorly, bootstrap bias-corrected intervals perform best when skewness is high, and Johnson $j$3 intervals perform well when skewness is moderate. In the binomial case, the index effectively distinguishes poor from good procedures: the classic Wald interval is poor, whereas Wilson, Agresti, and mid-$j$4 intervals perform better; as sample size increases, the indexes for all estimators increase, and the best estimators’ indexes approach $j$5 (Minkah et al., 2017).

The paper also emphasizes that bootstrap calibration is computationally intensive and not always necessary. In the reported experiments, the index values for calibrated and non-calibrated intervals often led to the same conclusion. This suggests that the index can reproduce the substantive ranking of procedures without the extra cost of calibration.

In the application to confidence intervals for the coefficient of variation, empirical coverage probabilities and average interval lengths from Gulhar et al. (2012) were used as inputs. The index was computed for each estimator and scenario, and the procedures were then compared through their index values. The practical interpretation is direct: the highest index identifies the preferred estimator, while the lowest index indicates an inferior one. The S.K estimator had small interval lengths but poor coverage and therefore a low index, leading to its rejection. Bootstrap-based and certain analytic interval estimators had high indexes and were recommended. The paper further notes that index values can be superimposed on coverage plots or plotted against sample size or the coefficient of variation, allowing simultaneous visual and numerical assessment (Minkah et al., 2017).

6. Relation to later coverage–width summaries and diagnostics

Subsequent work on interval comparison has developed related ways to combine calibration and sharpness. One example is the interval score for binomial proportion intervals, defined by

$j$6

which combines interval width with an explicit penalty for non-coverage. Under uniform weighting over the true proportion, the expected interval score recommends the Wilson confidence interval or Bayesian credible intervals with a uniform prior; when extremely low or high proportions receive more weight, it recommends Bayesian credible intervals based on Jeffreys’ prior (Hofer et al., 2022). Unlike the confidence interval index, the interval score is formulated as a proper scoring rule for central intervals and is evaluated through expectation over the sampling distribution.

A complementary development is the Singh plot, which visualizes whether a confidence structure achieves frequentist coverage across all confidence levels. For a precise confidence distribution, ideal behavior corresponds to a cumulative unit uniform distribution; deviations below the diagonal indicate undercoverage, while deviations above it indicate conservatism. For imprecise confidence structures, the plot becomes a band bounded by lower and upper confidence distributions. The paper explicitly notes that the area between Singh plot curves and the diagonal, or their vertical deviation, can serve as a quantitative measure of coverage deviation and can guide the definition of a confidence interval index or other coverage quality metrics (Wimbush et al., 2021).

These related methods clarify the place of the confidence interval index within a broader literature on uncertainty quantification. The common theme is that interval procedures should be judged jointly by calibration and informativeness. The confidence interval index provides a direct scalar compromise based on empirical coverage and length; interval scores formalize the trade-off through a penalty-based scoring rule; Singh plots provide a visual diagnostic of coverage behavior over the full confidence spectrum (Minkah et al., 2017).