Class-Conditional Entropy

• Class-Conditional Entropy is a metric that measures the residual uncertainty in a classifier's output when the true label is known, particularly in binary classification.
• It provides a closed-form analytical framework that relates prior probabilities and error rates to quantify the information cost of classification errors.
• Enhanced analytical bounds, including a tighter upper bound than Kovalevskij’s, offer precise limits on error probabilities and guide optimal classifier design.

Class-conditional entropy quantifies the residual uncertainty in the classifier’s predicted output, given full knowledge of the true class label. In the context of binary classification, this metric rigorously encapsulates the informational cost of errors, and its relation to error probability is deeply intertwined with both classical and recently derived analytical bounds. The precise characterization of this relationship is central to understanding the theoretical limits of classification algorithms, especially under specified priors and error patterns.

1. Formal Definition of Class-Conditional Entropy

In binary classification, class-conditional entropy $H(Y|X)$ measures the average uncertainty in the classifier output $Y$ when the true label $X$ is known. For random variables $X, Y \in \{1, 2\}$ with joint distribution $p(x, y)$, the conditional entropy is given by

$$H(Y|X) = -\sum_{i=1}^2 \sum_{j=1}^2 p(x_i, y_j) \log_2 p(y_j | x_i).$$

An equivalent formulation in terms of marginal and joint entropy is

$$H(Y|X) = H(X, Y) - H(X) = - \sum_{i,j} p_{ij} \log_2 p_{ij} + \sum_i p(x_i) \log_2 p(x_i),$$

where $p_{ij} = p(X = x_i, Y = y_j)$, $p_i = p(X = x_i)$, and $e_i$ denotes the error rate from class $Y$0. In this framework, the parameters relate as follows:

• $Y$1, $Y$2,
• $Y$3, $Y$4,
• with $Y$5, $Y$6, $Y$7.

This structure enables a closed-form expression of conditional entropy in terms of priors and error components (Hu et al., 2012).

2. Exact Closed-Form Expression in Binary Classification

The closed-form solution for the class-conditional entropy, utilizing the above parametrization, is

$Y$8

This mapping $Y$9 is exact for all admissible values. In practice, it is common to define the total error $X$0 and the minimal prior $X$1, and then to compute extremal values of $X$2 over the domain $X$3, which are fundamental for establishing entropy–error bounds (Hu et al., 2012).

3. Analytical Bounds: Fano’s Inequality and New Upper Bounds

The relationship between class-conditional entropy and classification error is classically expressed by Fano’s inequality. For binary classification, the simplified Fano bound is

$X$4

where $X$5 and $X$6 is the binary entropy function. Inverting this yields the exact lower bound on error in terms of conditional entropy:

$X$7

with $X$8 standing for the functional inverse of the binary entropy on $X$9.

A novel analytical upper bound for $X, Y \in \{1, 2\}$0 as a function of $X, Y \in \{1, 2\}$1 and $X, Y \in \{1, 2\}$2 is derived via minimization of $X, Y \in \{1, 2\}$3 for a fixed $X, Y \in \{1, 2\}$4 and known priors. For the configuration with error concentrated on the minority class (e.g., $X, Y \in \{1, 2\}$5 if $X, Y \in \{1, 2\}$6), the mapping is

$X, Y \in \{1, 2\}$7

Inverting, the explicit upper bound is

$X, Y \in \{1, 2\}$8

which is strictly tighter than the classical Kovalevskij upper bound $X, Y \in \{1, 2\}$9, except at $p(x, y)$0 or $p(x, y)$1 (Hu et al., 2012).

4. Geometry of the Error–Entropy Relationship

The analytical bounds describe the admissible region of $p(x, y)$2:

• Lower boundary: Described by Fano’s inverse curve $p(x, y)$3.
• Upper boundary (Bayesian errors, priors known): A smooth, concave segment governed by $p(x, y)$4 for $p(x, y)$5, then a vertical segment at $p(x, y)$6 for $p(x, y)$7.
• Non-Bayesian errors (priors unknown): The region expands, bounded by the Fano lower bound and its reflection about $p(x, y)$8. When $p(x, y)$9 is specified, the region is further constrained by the analytical bounds and their mirror curves for $$H(Y|X) = -\sum_{i=1}^2 \sum_{j=1}^2 p(x_i, y_j) \log_2 p(y_j | x_i).$$0.

As $$H(Y|X) = -\sum_{i=1}^2 \sum_{j=1}^2 p(x_i, y_j) \log_2 p(y_j | x_i).$$1 increases, $$H(Y|X) = -\sum_{i=1}^2 \sum_{j=1}^2 p(x_i, y_j) \log_2 p(y_j | x_i).$$2 follows the concave Fano curve, saturates at $$H(Y|X) = -\sum_{i=1}^2 \sum_{j=1}^2 p(x_i, y_j) \log_2 p(y_j | x_i).$$3 at $$H(Y|X) = -\sum_{i=1}^2 \sum_{j=1}^2 p(x_i, y_j) \log_2 p(y_j | x_i).$$4, and, in certain settings, decreases for $$H(Y|X) = -\sum_{i=1}^2 \sum_{j=1}^2 p(x_i, y_j) \log_2 p(y_j | x_i).$$5 along the mirrored boundary (Figure 1 and 2 in (Hu et al., 2012)).

5. Numerical Illustration and Practical Usage

For $$H(Y|X) = -\sum_{i=1}^2 \sum_{j=1}^2 p(x_i, y_j) \log_2 p(y_j | x_i).$$6, $$H(Y|X) = -\sum_{i=1}^2 \sum_{j=1}^2 p(x_i, y_j) \log_2 p(y_j | x_i).$$7 (i.e., $$H(Y|X) = -\sum_{i=1}^2 \sum_{j=1}^2 p(x_i, y_j) \log_2 p(y_j | x_i).$$8), and $$H(Y|X) = -\sum_{i=1}^2 \sum_{j=1}^2 p(x_i, y_j) \log_2 p(y_j | x_i).$$9:

• Fano’s lower bound: $$H(Y|X) = H(X, Y) - H(X) = - \sum_{i,j} p_{ij} \log_2 p_{ij} + \sum_i p(x_i) \log_2 p(x_i),$$0 bits.
• Analytical upper bound:

$$H(Y|X) = H(X, Y) - H(X) = - \sum_{i,j} p_{ij} \log_2 p_{ij} + \sum_i p(x_i) \log_2 p(x_i),$$1

• Upper bound on $$H(Y|X) = H(X, Y) - H(X) = - \sum_{i,j} p_{ij} \log_2 p_{ij} + \sum_i p(x_i) \log_2 p(x_i),$$2: $$H(Y|X) = H(X, Y) - H(X) = - \sum_{i,j} p_{ij} \log_2 p_{ij} + \sum_i p(x_i) \log_2 p(x_i),$$3, which is tighter than Kovalevskij’s $$H(Y|X) = H(X, Y) - H(X) = - \sum_{i,j} p_{ij} \log_2 p_{ij} + \sum_i p(x_i) \log_2 p(x_i),$$4.
• Lower bound on $$H(Y|X) = H(X, Y) - H(X) = - \sum_{i,j} p_{ij} \log_2 p_{ij} + \sum_i p(x_i) \log_2 p(x_i),$$5: $$H(Y|X) = H(X, Y) - H(X) = - \sum_{i,j} p_{ij} \log_2 p_{ij} + \sum_i p(x_i) \log_2 p(x_i),$$6.

Consequently, for $$H(Y|X) = H(X, Y) - H(X) = - \sum_{i,j} p_{ij} \log_2 p_{ij} + \sum_i p(x_i) \log_2 p(x_i),$$7 bits, $$H(Y|X) = H(X, Y) - H(X) = - \sum_{i,j} p_{ij} \log_2 p_{ij} + \sum_i p(x_i) \log_2 p(x_i),$$8 is confined to $$H(Y|X) = H(X, Y) - H(X) = - \sum_{i,j} p_{ij} \log_2 p_{ij} + \sum_i p(x_i) \log_2 p(x_i),$$9 (Hu et al., 2012).

6. Significance for Classification Theory

The closed-form expression for class-conditional entropy in binary classification, the exact re-derivation of Fano’s lower bound, and the new upper bound (parameterized by $p_{ij} = p(X = x_i, Y = y_j)$0 and $p_{ij} = p(X = x_i, Y = y_j)$1) form a comprehensive analytical toolkit. These results precisely bracket the $p_{ij} = p(X = x_i, Y = y_j)$2 region for binary classification, allowing explicit quantification of how residual uncertainty grows and then saturates as classification errors increase. The upper bound derived is provably tighter than previously known results, such as Kovalevskij’s bound.

These advances elucidate the role of prior information and error structure in determining informational limits. The characterization of admissible regions is instrumental for theoretical investigations of classifier design and for benchmarking achievable error–entropy trade-offs (Hu et al., 2012).