Multilabel Confusion Matrix Overview

• Multilabel confusion matrix is a family of evaluation frameworks that extends traditional binary contingency tables to handle multiple, non-mutually-exclusive labels.
• It employs per-label binary counts, probability distributions, tensors, and local fuzzy corrections to incorporate thresholding, label dependence, and ambiguity.
• A significant insight is that traditional confusion matrices must be rethought for multilabel tasks, prompting advanced normalization, semantic alignment, and differentiable metric improvements.

Searching arXiv for recent and foundational papers on multilabel confusion matrices, multi-output confusion representations, and related evaluation frameworks. The literature suggests that a multilabel confusion matrix is not a single universally fixed object, but a family of representations for comparing predicted and expected outputs when each instance may carry multiple, non-mutually-exclusive labels. In the simplest construction, multilabel evaluation is reduced to one binary contingency table per label; in broader formulations, the matrix becomes a multioutput confusion tensor, a probability distribution over actual and predicted label variables, a local fuzzy correction device for pairwise classifiers, or a semantics-aware alignment matrix. Across these formulations, the common objective is to preserve the diagnostic role of the conventional confusion matrix while accommodating thresholding, label dependence, hierarchy, soft supervision, and ambiguity (Görtler et al., 2021, Wang et al., 2019).

1. Why the conventional confusion matrix is insufficient

The conventional confusion matrix is a tabular layout that compares predicted class labels against actual class labels over all data instances. That representation is well matched to flat single-label classification, but it does not directly support hierarchical and multi-output labels, and it assumes a flat, one-dimensional label space (Görtler et al., 2021). In multilabel classification, by contrast, a single example may have multiple labels, and labels are not mutually exclusive (Bénédict et al., 2021).

This mismatch appears in several distinct ways. First, the usual two-dimensional matrix cannot directly represent multiple simultaneously active labels per instance without ad hoc preprocessing or custom scripts. Second, the diagonal can dominate the display, especially in imbalanced datasets, so off-diagonal confusions are hidden. Third, raw counts often do not provide enough analysis context, because precision-, recall-, and count-oriented summaries are needed alongside the matrix. Fourth, in annotation settings with set-valued outputs, ordinary confusion-matrix logic for mutually exclusive classes is not directly appropriate unless one first projects the multilabel output into a single dominant label (Görtler et al., 2021, Öhman et al., 19 Jun 2026).

A plausible implication is that the phrase “multilabel confusion matrix” should be read operationally rather than literally. In some settings it means a family of per-label binary matrices; in others it means a generalized object that preserves the confusion-matrix metaphor while expanding its algebraic scope.

2. Per-label binary contingency tables and averaging schemes

A standard multilabel construction is to treat each label as a binary decision problem and compute confusion counts separately for each label. Using a threshold $t$, the binary counts are written as

$\begin{array}{ll} \mathit{tp} = \sum \mathds{1}_{\hat{\mathbf{y}} \geq t \;\odot\; \mathbf{y}} & \mathit{fp} = \sum \mathds{1}_{\hat{\mathbf{y}} \geq t \;\odot\; (\mathds{1} - \mathbf{y})} \ \mathit{fn} = \sum \mathds{1}_{\hat{\mathbf{y}} < t \;\odot\; \mathbf{y}} & \mathit{tn} = \sum \mathds{1}_{\hat{\mathbf{y}} < t \;\odot\; (\mathds{1} - \mathbf{y})} \end{array}$

and, for multilabel classification, the same formulas apply per class $j$ by substituting $\mathbf{y}$ with $\mathbf{y}^j$ (Bénédict et al., 2021).

This per-label binary view is also used for annotation agreement in multilabel emotion analysis. For annotator pair $(a,b)$, emotion $e$, and item $i$, the decisions are reduced to binary presence or absence, and agreement is summarized by per-emotion $F_1$, macro-$F_1$, and micro-$\begin{array}{ll} \mathit{tp} = \sum \mathds{1}_{\hat{\mathbf{y}} \geq t \;\odot\; \mathbf{y}} & \mathit{fp} = \sum \mathds{1}_{\hat{\mathbf{y}} \geq t \;\odot\; (\mathds{1} - \mathbf{y})} \ \mathit{fn} = \sum \mathds{1}_{\hat{\mathbf{y}} < t \;\odot\; \mathbf{y}} & \mathit{tn} = \sum \mathds{1}_{\hat{\mathbf{y}} < t \;\odot\; (\mathds{1} - \mathbf{y})} \end{array}$0 (Öhman et al., 19 Jun 2026). In that framework, macro-$\begin{array}{ll} \mathit{tp} = \sum \mathds{1}_{\hat{\mathbf{y}} \geq t \;\odot\; \mathbf{y}} & \mathit{fp} = \sum \mathds{1}_{\hat{\mathbf{y}} \geq t \;\odot\; (\mathds{1} - \mathbf{y})} \ \mathit{fn} = \sum \mathds{1}_{\hat{\mathbf{y}} < t \;\odot\; \mathbf{y}} & \mathit{tn} = \sum \mathds{1}_{\hat{\mathbf{y}} < t \;\odot\; (\mathds{1} - \mathbf{y})} \end{array}$1 is obtained by computing $\begin{array}{ll} \mathit{tp} = \sum \mathds{1}_{\hat{\mathbf{y}} \geq t \;\odot\; \mathbf{y}} & \mathit{fp} = \sum \mathds{1}_{\hat{\mathbf{y}} \geq t \;\odot\; (\mathds{1} - \mathbf{y})} \ \mathit{fn} = \sum \mathds{1}_{\hat{\mathbf{y}} < t \;\odot\; \mathbf{y}} & \mathit{tn} = \sum \mathds{1}_{\hat{\mathbf{y}} < t \;\odot\; (\mathds{1} - \mathbf{y})} \end{array}$2 separately for each emotion and averaging the resulting scores equally, whereas micro-$\begin{array}{ll} \mathit{tp} = \sum \mathds{1}_{\hat{\mathbf{y}} \geq t \;\odot\; \mathbf{y}} & \mathit{fp} = \sum \mathds{1}_{\hat{\mathbf{y}} \geq t \;\odot\; (\mathds{1} - \mathbf{y})} \ \mathit{fn} = \sum \mathds{1}_{\hat{\mathbf{y}} < t \;\odot\; \mathbf{y}} & \mathit{tn} = \sum \mathds{1}_{\hat{\mathbf{y}} < t \;\odot\; (\mathds{1} - \mathbf{y})} \end{array}$3 is obtained by flattening all binary decisions into one contingency table and computing a single pooled $\begin{array}{ll} \mathit{tp} = \sum \mathds{1}_{\hat{\mathbf{y}} \geq t \;\odot\; \mathbf{y}} & \mathit{fp} = \sum \mathds{1}_{\hat{\mathbf{y}} \geq t \;\odot\; (\mathds{1} - \mathbf{y})} \ \mathit{fn} = \sum \mathds{1}_{\hat{\mathbf{y}} < t \;\odot\; \mathbf{y}} & \mathit{tn} = \sum \mathds{1}_{\hat{\mathbf{y}} < t \;\odot\; (\mathds{1} - \mathbf{y})} \end{array}$4 (Öhman et al., 19 Jun 2026).

This decomposition is the most common strict multilabel interpretation of a confusion matrix. It preserves the familiar $\begin{array}{ll} \mathit{tp} = \sum \mathds{1}_{\hat{\mathbf{y}} \geq t \;\odot\; \mathbf{y}} & \mathit{fp} = \sum \mathds{1}_{\hat{\mathbf{y}} \geq t \;\odot\; (\mathds{1} - \mathbf{y})} \ \mathit{fn} = \sum \mathds{1}_{\hat{\mathbf{y}} < t \;\odot\; \mathbf{y}} & \mathit{tn} = \sum \mathds{1}_{\hat{\mathbf{y}} < t \;\odot\; (\mathds{1} - \mathbf{y})} \end{array}$5 semantics, but it does so label by label rather than through a single global multiclass table. Its principal limitation is that inter-label structure is external to the matrix itself: label dependence, co-occurrence, and hierarchical relations are not explicit in the representation.

3. From matrices to probability distributions and confusion tensors

A more general line of work treats the confusion matrix as a probabilistic object. In Neo, the matrix is redefined as a joint distribution over actual and predicted label variables, such as $\begin{array}{ll} \mathit{tp} = \sum \mathds{1}_{\hat{\mathbf{y}} \geq t \;\odot\; \mathbf{y}} & \mathit{fp} = \sum \mathds{1}_{\hat{\mathbf{y}} \geq t \;\odot\; (\mathds{1} - \mathbf{y})} \ \mathit{fn} = \sum \mathds{1}_{\hat{\mathbf{y}} < t \;\odot\; \mathbf{y}} & \mathit{tn} = \sum \mathds{1}_{\hat{\mathbf{y}} < t \;\odot\; (\mathds{1} - \mathbf{y})} \end{array}$6, with cell values derived from relative frequencies over the dataset (Görtler et al., 2021). The fruit example makes this explicit: $\begin{array}{ll} \mathit{tp} = \sum \mathds{1}_{\hat{\mathbf{y}} \geq t \;\odot\; \mathbf{y}} & \mathit{fp} = \sum \mathds{1}_{\hat{\mathbf{y}} \geq t \;\odot\; (\mathds{1} - \mathbf{y})} \ \mathit{fn} = \sum \mathds{1}_{\hat{\mathbf{y}} < t \;\odot\; \mathbf{y}} & \mathit{tn} = \sum \mathds{1}_{\hat{\mathbf{y}} < t \;\odot\; (\mathds{1} - \mathbf{y})} \end{array}$7 Once a confusion matrix is viewed as a multivariate distribution, conditioning, marginalization, and nesting become the basic transformation operators, and the algebra is closed because the result of each operation is still a probability distribution (Görtler et al., 2021).

For multi-output or multilabel settings, each label dimension is treated as another variable in the joint distribution. Neo’s example augments Fruit with Taste, producing $\begin{array}{ll} \mathit{tp} = \sum \mathds{1}_{\hat{\mathbf{y}} \geq t \;\odot\; \mathbf{y}} & \mathit{fp} = \sum \mathds{1}_{\hat{\mathbf{y}} \geq t \;\odot\; (\mathds{1} - \mathbf{y})} \ \mathit{fn} = \sum \mathds{1}_{\hat{\mathbf{y}} < t \;\odot\; \mathbf{y}} & \mathit{tn} = \sum \mathds{1}_{\hat{\mathbf{y}} < t \;\odot\; (\mathds{1} - \mathbf{y})} \end{array}$8, and the joint state space grows as

$\begin{array}{ll} \mathit{tp} = \sum \mathds{1}_{\hat{\mathbf{y}} \geq t \;\odot\; \mathbf{y}} & \mathit{fp} = \sum \mathds{1}_{\hat{\mathbf{y}} \geq t \;\odot\; (\mathds{1} - \mathbf{y})} \ \mathit{fn} = \sum \mathds{1}_{\hat{\mathbf{y}} < t \;\odot\; \mathbf{y}} & \mathit{tn} = \sum \mathds{1}_{\hat{\mathbf{y}} < t \;\odot\; (\mathds{1} - \mathbf{y})} \end{array}$9

Neo then returns this high-dimensional structure to a matrix form by three operations: conditioning on fixed label values, marginalizing labels not under analysis, and nesting labels so that multiple dimensions can be visualized simultaneously (Görtler et al., 2021). This is the basis for the paper’s practical claim that the “right” confusion matrix is often not one matrix, but a family of related views derived from the same probabilistic specification.

An adjacent theoretical generalization is the confusion tensor for multioutput classification. In that setting,

$j$0

so each output $j$1 has its own $j$2 confusion-matrix slice $j$3 (Wang et al., 2019). When $j$4, the confusion tensor reduces to the usual $j$5 confusion matrix; when $j$6, the framework recovers the multilabel case (Wang et al., 2019). The same paper studies metrics of the form

$j$7

and shows that micro-, macro-, and instance-averaging correspond to distinct ways of aggregating these output-specific confusion slices (Wang et al., 2019).

Normalization remains central in these generalized settings. Neo treats normalization as part of the algebra rather than a separate plotting concern: default normalization is by instance count, while row- and column-normalization expose recall-like and precision-like interpretations. The diagonal of a row-normalized matrix yields recall,

$j$8

and the diagonal of a column-normalized matrix yields precision,

$j$9

with additional derived metrics including accuracy, count observed, count predicted, true positives, false positives, false negatives, and true negatives (Görtler et al., 2021).

4. Differentiable, soft, and threshold-sensitive formulations

A major modern development is the replacement of hard confusion counts by smooth surrogates. The central observation in "sigmoidF1" is that standard confusion-matrix metrics depend on hard thresholding and therefore are not differentiable. The paper states the broader principle explicitly: “Any confusion matrix metric can be formulated with a smooth surrogate” (Bénédict et al., 2021). The hard step is replaced by

$\mathbf{y}$0

where $\mathbf{y}$1 controls the slope and $\mathbf{y}$2 controls the offset (Bénédict et al., 2021).

The resulting surrogate confusion entries are

$\mathbf{y}$3

$\mathbf{y}$4

and these smoothed counts induce differentiable surrogates for accuracy, precision, negative predictive value, recall, specificity, and $\mathbf{y}$5 (Bénédict et al., 2021). The specific loss

$\mathbf{y}$6

is presented as a smooth surrogate for multilabel macro $\mathbf{y}$7 and is claimed to estimate both label suitability and label counts (Bénédict et al., 2021).

A related shift appears when the reference labels themselves are soft. In multilabel emotion annotation, hard targets are defined by union over annotators,

$\mathbf{y}$8

whereas soft targets are empirical vote shares,

$\mathbf{y}$9

with an additional intensity-weighted variant based on rescaled annotation intensities (Öhman et al., 19 Jun 2026). In that setting, classical $\mathbf{y}^j$0 counts remain meaningful after thresholding, but they are no longer the full story because the reference is a probability-like quantity rather than a binary gold label. The paper therefore supplements thresholded macro-/micro-$\mathbf{y}^j$1 with

$\mathbf{y}^j$2

which measures probabilistic alignment between predicted probabilities and soft vote-share targets (Öhman et al., 19 Jun 2026).

This line of work makes threshold sensitivity explicit. In multilabel evaluation, confusion-matrix-derived metrics depend strongly on the decision threshold, and threshold-moving at inference time does not necessarily dominate fixed thresholds (Bénédict et al., 2021). The practical implication is that a multilabel confusion matrix is often inseparable from the thresholding protocol that generated it.

5. Local fuzzy matrices, pairwise correction, and semantic alignment

Another family of methods constructs confusion matrices locally rather than globally. In label-pairwise multilabel classification, one binary classifier is trained for each pair of labels,

$\mathbf{y}^j$3

and, for a test instance, each pairwise classifier is corrected using a local fuzzy confusion matrix estimated from a validation set and a Gaussian neighborhood (Trajdos et al., 2017). The core target is

$\mathbf{y}^j$4

the probability that the true class is $\mathbf{y}^j$5 given classifier output $\mathbf{y}^j$6 at location $\mathbf{y}^j$7, approximated by normalized local fuzzy confusion entries (Trajdos et al., 2017). The correction is combined with a Random Reference Classifier and then fused across label pairs by weighted averaging (Trajdos et al., 2017).

The weighted fuzzy-confusion-matrix scheme introduces an information-theoretic weight

$\mathbf{y}^j$8

intended to emphasize classifiers with higher correction ability and to reduce the bias of the original method toward majority labels (Trajdos et al., 2017). A related pairwise correction paper proposes two refinements of the confusion-matrix estimation procedure: an imbalance-handling scheme with class-specific weights $\mathbf{y}^j$9, and an overlap-aware scheme that reinserts double-labeled instances with membership $(a,b)$0 into the validation subset used for confusion estimation (Trajdos et al., 2017). The reported conclusion is that the basic FCM is not recommended because it is strongly affected by imbalance, while the improved variants matter most for zero-one loss and reduce sensitivity to label imbalance and label concurrence (Trajdos et al., 2017).

A closely related binary wrapper is the soft-confusion-matrix classifier (SCM), which estimates a local soft confusion matrix around each query point using validation data, Randomized Reference Classifier decision regions, and Gaussian neighborhood weighting (Trajdos et al., 2019). In multilabel use, SCM and the Bayes metaclassifier are not applied directly to multilabel outputs; instead they are embedded in problem-transformation pipelines such as Binary Relevance and Label Pairwise (Trajdos et al., 2019). Under LPW, both methods significantly outperformed the unmodified reference method, and no statistically significant differences were found between SCM and BMC (Trajdos et al., 2019).

A different extension replaces purely statistical alignment by ontology-driven semantic alignment. In "Knowledge-Based Construction of Confusion Matrices for Multi-Label Classification Algorithms using Semantic Similarity Measures," each predicted label is compared with each expected label, semantic similarities are computed, and four rules are applied to obtain unique correspondences (Turki et al., 2020). The method uses an adaptive threshold,

$(a,b)$1

to reject unrelated matches, and the resulting correspondences are used to populate a confusion matrix whose values may be simple counts or rates; in that paper, simple counts are used (Turki et al., 2020). The experimental study relies on the feature-based semantic similarity measure of Rodriguez et al. (2003) driven by WordNet 3.1, and the authors explicitly acknowledge dependence on a semantic resource, threshold sensitivity, and preliminary validation based on four fabricated examples (Turki et al., 2020).

6. Interpretation, theory, and recurrent misconceptions

One recurrent misconception is that any model involving a confusion matrix in a multi-entity setting is automatically a multilabel confusion-matrix model. "TrueLabel + Confusions" explicitly rejects that interpretation: its latent label space is single-valued, $(a,b)$2, each judge produces exactly one categorical rating per item, and the framework is therefore a single-label confusion model rather than a multilabel confusion matrix model in the standard machine-learning sense (Liu et al., 2012). The paper is better interpreted as a multi-rater multiclass confusion-matrix model with hierarchical Bayesian sharing.

A second misconception is that projected multiclass confusion matrices are themselves multilabel confusion matrices. In multilabel emotion analysis, the reported confusion matrix is a “row-normalized single-label confusion matrix, obtained by projecting multilabel outputs to a dominant label per instance” (Öhman et al., 19 Jun 2026). That device is diagnostically useful—it revealed strong diagonal dominance for sadness and trust, off-diagonal confusion between disgust and sadness, and structured confusions such as sadness vs. fear and trust vs. anticipation—but it is not a strict multilabel confusion matrix (Öhman et al., 19 Jun 2026).

A third misconception is that scalar summary metrics can substitute for confusion structure. The PAC-Bayesian confusion-matrix literature in multiclass classification motivates the confusion matrix precisely as an error measure richer than mere scalar criterion, and derives bounds on the operator norm of the true confusion matrix rather than on scalar misclassification alone (Morvant et al., 2012). Although that work is multiclass rather than multilabel, it provides a theoretical rationale for treating confusion structure as a first-class evaluation object. Neo reaches a similar practical conclusion from visual analytics: overall accuracy may look strong while normalization and per-class inspection reveal poor precision and recall for specific categories, hidden confusions, and class-imbalance effects (Görtler et al., 2021).

The current literature also distinguishes discrete error accounting from uncertainty-aware evaluation. Hard labels may maximize $(a,b)$3 metrics, while soft supervision yields predictions that better reflect empirical annotator variance and uncertainty (Öhman et al., 19 Jun 2026). This suggests that multilabel confusion matrices are most informative when interpreted together with thresholding policy, normalization regime, derived metrics, and, where applicable, probabilistic alignment measures.

In that sense, the multilabel confusion matrix is best understood as a spectrum of related constructions. At one end are per-label binary contingency tables; at the other are generalized probabilistic, tensor-valued, fuzzy, and semantic objects. What unifies them is not a single matrix shape, but the attempt to preserve the confusion-matrix role of diagnosing which outputs are mistaken for which others under the structural conditions unique to multilabel prediction.