---
title: Class-Conditional Entropy
url: https://www.emergentmind.com/topics/class-conditional-entropy
type: topic
---

# Class-Conditional Entropy

Class-conditional entropy quantifies the residual uncertainty in the classifier’s predicted output, given full knowledge of the true class label. In the context of binary classification, this metric rigorously encapsulates the informational cost of errors, and its relation to error probability is deeply intertwined with both classical and recently derived analytical bounds. The precise characterization of this relationship is central to understanding the theoretical limits of classification algorithms, especially under specified priors and error patterns.

## 1. Formal Definition of Class-Conditional Entropy

In binary classification, class-conditional entropy $H(Y|X)$ measures the average uncertainty in the classifier output $Y$ when the true label $X$ is known. For random variables $X, Y \in \{1, 2\}$ with joint distribution $p(x, y)$, the conditional entropy is given by

$$
H(Y|X) = -\sum_{i=1}^2 \sum_{j=1}^2 p(x_i, y_j) \log_2 p(y_j | x_i).
$$

An equivalent formulation in terms of marginal and joint entropy is

$$
H(Y|X) = H(X, Y) - H(X) = - \sum_{i,j} p_{ij} \log_2 p_{ij} + \sum_i p(x_i) \log_2 p(x_i),
$$

where $p_{ij} = p(X = x_i, Y = y_j)$, $p_i = p(X = x_i)$, and $e_i$ denotes the error rate from class $i$. In this framework, the parameters relate as follows:

- $p_{11} = p_1 - e_1$, $p_{12} = e_1$,
- $p_{21} = e_2$,     $p_{22} = p_2 - e_2$,
- with $p_1 + p_2 = 1$, $e_1 \in [0,p_1]$, $e_2 \in [0,p_2]$.

This structure enables a closed-form expression of conditional entropy in terms of priors and error components [1205.6602].

## 2. Exact Closed-Form Expression in Binary Classification

The closed-form solution for the class-conditional entropy, utilizing the above parametrization, is

\[
\begin{aligned}
H(T|Y) = {} & -p_1 \log_2 p_1 - p_2 \log_2 p_2 \\
            & - e_1 \log_2 \left( \frac{e_1}{(p_2 + e_1 - e_2)p_1} \right) \\
            & - e_2 \log_2 \left( \frac{e_2}{(p_1 - e_1 + e_2)p_2} \right) \\
            & - (p_1 - e_1) \log_2 \left( \frac{p_1 - e_1}{(p_1 - e_1 + e_2)p_1} \right) \\
            & - (p_2 - e_2) \log_2 \left( \frac{p_2 - e_2}{(p_2 + e_1 - e_2)p_2} \right).
\end{aligned}
\]

This mapping $(p_1, p_2, e_1, e_2) \mapsto H(T|Y)$ is exact for all admissible values. In practice, it is common to define the total error $e = e_1 + e_2$ and the minimal prior $p_{min} = \min\{p_1, p_2\}$, and then to compute extremal values of $H(T|Y)$ over the domain $\{e_1 + e_2 = e\}$, which are fundamental for establishing entropy–error bounds [1205.6602].

## 3. Analytical Bounds: Fano’s Inequality and New Upper Bounds

The relationship between class-conditional entropy and classification error is classically expressed by Fano’s inequality. For binary classification, the simplified Fano bound is

$$
H(X|Y) \leq H_b(P_e),
$$

where $P_e = P(Y \neq X)$ and $H_b(P_e) = -P_e \log_2 P_e - (1 - P_e) \log_2 (1 - P_e)$ is the binary entropy function. Inverting this yields the exact lower bound on error in terms of conditional entropy:

$$
P_e \geq H_b^{-1}(H(X|Y)),
$$

with $H_b^{-1}$ standing for the functional inverse of the binary entropy on $[0, 1]$.

A novel analytical upper bound for $P_e$ as a function of $H(T|Y)$ and $p_{min}$ is derived via minimization of $H(T|Y)$ for a fixed $e$ and known priors. For the configuration with error concentrated on the minority class (e.g., $e_2 = 0$ if $p_2 = p_{min} < p_1$), the mapping is

$$
H(T|Y) = -p_{min} \log_2 \left( \frac{p_{min}}{e + p_{min}} \right) - e \log_2 \left( \frac{e}{e + p_{min}} \right).
$$

Inverting, the explicit upper bound is

$$
P_e \leq G_2(H(T|Y), p_{min}), \quad P_e \leq \min\{p_{min}, G_2(H(T|Y), p_{min})\},
$$

which is strictly tighter than the classical Kovalevskij upper bound $P_e \leq H(T|Y)/2$, except at $P_e = 0$ or $P_e = p_{min}$ [1205.6602].

## 4. Geometry of the Error–Entropy Relationship

The analytical bounds describe the admissible region of $(P_e, H(T|Y))$:

- **Lower boundary**: Described by Fano’s inverse curve $P_e = H_b^{-1}(H)$.
- **Upper boundary (Bayesian errors, priors known)**: A smooth, concave segment governed by $G_2$ for $0 \leq H \leq H_b(p_{min})$, then a vertical segment at $P_e = p_{min}$ for $H = H_b(p_{min})$.
- **Non-Bayesian errors (priors unknown)**: The region expands, bounded by the Fano lower bound and its reflection about $P_e = 0.5$. When $p_{min}$ is specified, the region is further constrained by the analytical bounds and their mirror curves for $P_e > 0.5$.

As $P_e$ increases, $H(T|Y)$ follows the concave Fano curve, saturates at $H_b(p_{min})$ at $P_e = p_{min}$, and, in certain settings, decreases for $P_e > p_{min}$ along the mirrored boundary (Figure 1 and 2 in [1205.6602]).

## 5. Numerical Illustration and Practical Usage

For $p_1 = 0.8$, $p_2 = 0.2$ (i.e., $p_{min} = 0.2$), and $P_e = 0.10$:

- **Fano’s lower bound**: $H(T|Y) \leq H_b(0.10) \approx 0.469$ bits.
- **Analytical upper bound**:

  $$
  H(T|Y) = -0.2\log_2(0.2/0.3) - 0.1\log_2(0.1/0.3) \approx 0.306
  $$

- **Upper bound on $P_e$**: $P_e \leq G_2(0.306, 0.2) = 0.15$, which is tighter than Kovalevskij’s $P_e \leq 0.306/2 = 0.153$.
- **Lower bound on $P_e$**: $P_e \geq H_b^{-1}(0.306) \approx 0.05$.

Consequently, for $H(T|Y) = 0.306$ bits, $P_e$ is confined to $[0.05, 0.15]$ [1205.6602].

## 6. Significance for Classification Theory

The closed-form expression for class-conditional entropy in binary classification, the exact re-derivation of Fano’s lower bound, and the new upper bound (parameterized by $p_{min}$ and $H(T|Y)$) form a comprehensive analytical toolkit. These results precisely bracket the $(P_e, H(T|Y))$ region for binary classification, allowing explicit quantification of how residual uncertainty grows and then saturates as classification errors increase. The upper bound derived is provably tighter than previously known results, such as Kovalevskij’s bound.

These advances elucidate the role of prior information and error structure in determining informational limits. The characterization of admissible regions is instrumental for theoretical investigations of classifier design and for benchmarking achievable error–entropy trade-offs [1205.6602].

Source: https://www.emergentmind.com/topics/class-conditional-entropy