Defining over- and under-confidence in multiclass classification

Determine a mathematically precise and generally applicable definition of over-confidence and under-confidence for multiclass probabilistic classifiers f: X -> Delta_k, beyond the binary case, so that these notions can be evaluated separately for multiclass predictions.

Background

The paper introduces a variational estimator for L_p calibration errors and shows how to separately assess over- and under-confidence in the binary setting by modifying proper losses. While the binary case admits a clear separation between over- and under-confidence, extending this decomposition to multiclass predictions raises conceptual issues.

In the appendix, the authors note that it is not clear how to define over- and under-confidence for multiclass classifiers. As a pragmatic workaround, they suggest a one-versus-rest approach that evaluates over- and under-confidence for the top class, but this does not resolve the underlying definitional uncertainty.