Objective Evaluation Criteria

• Objective evaluation criteria are quantitative measures defined without free parameters, relying solely on empirical data from confusion matrices and Shannon entropy.
• They employ normalized information-theoretic measures—such as mutual information, divergence, and cross-entropy—to clearly differentiate between misclassification and reject types.
• The framework is governed by three meta-measures (diagonal monotonicity, reject rate sensitivity, and cost hierarchies) to ensure both practical relevance and theoretical rigor.

Objective evaluation criteria, in the context of information-theoretic classification assessment, refer to quantitative evaluation measures that do not contain any free parameters or user-set preferences—thus providing parameter-free, cost-free, and data-driven assessment of classifier performance. This approach, as detailed by the framework of twenty-four normalized information-theoretic measures (ITMs), is predicated on quantities derived solely from the confusion matrix and fundamental information-theoretic constructs, such as Shannon entropy, mutual information, divergence, and cross-entropy. These measures enable the rigorous, non-arbitrary distinction between misclassification error types and reject types, and are critically evaluated using three essential meta-measures that shape their practical and theoretical utility.

1. Formal Notion of Objective Measures

Objective evaluation in classification is defined by the absence of free parameters. Under this strict definition, an objective measure is uniquely determined by the data—typically, the empirical joint and marginal distributions extracted from the confusion matrix—without recourse to user-imposed weights, costs, or thresholds. The adoption of standard information-theoretic quantities such as the Shannon entropy,

$H(Y) = -\sum_{y} p(y)\log_2 p(y),$

ensures that objectivity is maintained and that the evaluation is unimpeachably parameter-free.

Subjective measures, by contrast, depend on external or subjective cost terms, tunable weights, or domain-specific parameters, and are thus inherently less generalizable.

2. Families of Information-Theoretic Measures (ITMs)

The twenty-four ITMs are systematically derived and categorized as follows:

A. Mutual-Information Based Measures:

• Use the mutual information between target %%%%1%%%% and prediction %%%%2%%%%:

$$I(T, Y) = \sum_{t}\sum_{y}p(t, y)\log_2\left(\frac{p(t, y)}{p(t)p(y)}\right).$$

• Different normalization strategies produce measures such as

$$NI_1(T, Y) = \frac{I(T, Y)}{H(T)}, \qquad NI_2(T, Y) = \frac{I_M(T, Y)}{H(T)},$$

where $I_M$ sums over the “intersection” (non-reject outcomes) to isolate correct assignments.

• Other variants normalize by $H(Y)$, the arithmetic mean $(I/H(T) + I/H(Y))/2$, or the geometric mean $I/\sqrt{H(T)H(Y)}$, among others.

B. Divergence-Based Measures:

• Use standard divergences $D_k$ to quantify dissimilarity between true and predicted distributions.
• Normalized as

$NI_k = \exp(-D_k),$

e.g., for Kullback-Leibler, Bhattacharyya, $\chi^2$, Euclidean, and Cauchy-Schwartz divergences. This yields $NI_k=1$ if $T$ and $Y$ are identical.

C. Cross-Entropy Based Measures:

• Based on the cross-entropy:

$H(T; Y) = -\sum_z p_t(z)\log_2 p_y(z),$

with normalization providing metrics reflecting distributional similarity. The relationship $H(T;Y) = H(T) + KL(T,Y)$ allows connection to the divergence group.

Each of these measures is constructed without tuning parameters, leveraging only empirical probabilities and information-theoretic identities.

3. Error and Reject Types: Augmented Confusion Matrix Formalism

The ITM framework extends the traditional confusion matrix by appending a “reject” column, producing an $(m \times (m + 1))$ matrix for $m$ classes. The augmented matrix $C = [c_{ij}]$ with $i=1,\ldots,m$, $j=1,\ldots,m+1$ enables precise attribution of off-diagonal elements to either misclassification (Type I/II errors) or rejection (Type I/II rejects):

• Type I Error: misclassification from class 1 to class 2 ($c_{12}$)
• Type II Error: misclassification from class 2 to class 1 ($c_{21}$)
• Type I Reject: rejection from first/large class ($c_{1, m+1}$)
• Type II Reject: rejection from second/minority class ($c_{2, m+1}$)

This partitioning permits ITMs to spatially and quantitatively distinguish between error types and reject types without introducing cost penalties, as all contributions are inferred directly from observed frequencies.

4. Three Essential Meta-Measures for Assessment

Given the proliferation of theoretically valid ITMs, the suitability of a measure for practical assessment is determined using three higher-order criteria:

1. Monotonicity with Respect to Diagonal Terms: A measure must be monotonic with increasing correct classifications (i.e., as diagonal entries in the confusion matrix rise). This monotonicity is analytically validated for select ITMs (notably $NI_2$) and is pivotal for ensuring that increases (or decreases) in accuracy are faithfully reflected in the score.
2. Variation with Reject Rate: The measure’s value must reflect both the accuracy and the proportion of rejected classifications. Scalar sensitivity to the reject rate is essential since practical classifiers often abstain rather than risk low-confidence misclassification decisions.
3. Intuitively Consistent Cost Hierarchies: The measure must impose heavier penalties on errors in minority classes and on misclassifications compared to rejections within the same class. This empirically rooted criterion enforces alignment with common-sense cost implications, notably without explicit cost terms.

The paper demonstrates via theoretical arguments and constructed confusion matrices that several conventional measures fail at least one meta-measure, often non-monotonicity or insensitivity to reject rates.

5. Analytical and Numerical Validation of ITMs

Extensive numerical experiments and analytic derivations demonstrate the discriminatory power and validity of the ITMs:

• Binary and three-class confusion matrices are assembled with subtle variations in error and reject terms while holding overall accuracy or correct recognition rate fixed.
• Conventional performance metrics (accuracy, precision, recall, ROC-based) are unable to differentiate cases with the same overall rates but distinct error/reject patterns.
• The $NI_2$ measure,

$NI_2(T, Y) = \frac{I_M(T, Y)}{H(T)},$

exhibits robust monotonicity, high sensitivity to both errors and reject rate, and aligns model rankings with the meta-measures. For example, $NI_2$ penalizes major-class errors less than minor-class errors and penalizes rejection less than outright misclassification, consistent with practical cost expectations.

Analytical derivations (Theorems 1–5) further confirm local extremum properties and boundary cases, such as $NI(T,Y)=1$ not necessarily implying perfect classification, but also label-exchange symmetries, and $NI=0$ corresponding precisely to minimum similarity scenarios.

6. Implications and Advantages of Objective ITMs

The rigorous, parameter-free ITM framework yields several significant advantages:

• Objectivity and Reproducibility: All scores are determined solely by the observed confusion matrix; no subjective parameterization or calibration is required.
• Granularity and Discrimination Power: ITMs, especially $NI_2$, resolve distinctions between error and reject distributions that are entirely missed by accuracy or cost-weighted metrics.
• Analytical Tractability and Interpretability: The closed-form information-theoretic definitions facilitate mathematical analysis of properties, limits, and sensitivity.
• Application to Abstention/Reject Scenarios: Unlike conventional metrics, ITMs robustly accommodate models that include abstain/reject decisions, common in safety-critical or low-confidence settings.

7. Practical Application and Guidance

For application, compute the required empirical probabilities from the augmented confusion matrix, implement the desired ITM variant (e.g., $NI_2$), and rank models or choose thresholds accordingly. No parameter search or calibration is necessary. For multi-class and reject-inclusive problems, $NI_2$ provides monotonic, reject-sensitive, and intuitively calibrated objective evaluation, as numerically validated by the test cases in the paper.

Summary Table: ITMs and Meta-Measure Compliance

ITM Group	Example Formula	Satisfies Meta-Measures? (See text)
Mutual-Information	$NI_1=I(T,Y)/H(T)$	Not always
Modified Mutual-Inf.	$NI_2=I_M(T,Y)/H(T)$	Yes (see text)
Divergence-Based	$NI_k = \exp(-D_k)$	Variable
Cross-Entropy	$H(T;Y)/H(T)$, etc.	Variable

$NI_2$ is highlighted in the analysis as most robust to meta-measure requirements in both binary and multi-class scenarios with reject options.

Conclusion

Objective evaluation criteria for classification, as formalized by normalized information-theoretic measures, provide a principled, mathematically rigorous, and empirically validated alternative to subjective, parameter-dependent metrics. With a full treatment of error and reject types, support for configuration-free deployment, and explicit guidance via meta-measures, these criteria define a standard for objective, nuanced classification assessment, especially in contexts where cost terms are unavailable or abstention is essential.