Label Set Level Metrics: Evaluation & Optimization

• Label set level metrics are evaluation criteria that assess predicted label sets as unified entities rather than independent labels.
• They incorporate measures such as subset accuracy, precision, recall, F1-measure, and Jaccard, aggregated via instance, micro, or macro regimes.
• These metrics are applied in multi-label learning, ranking frameworks, and hierarchical settings to provide nuanced performance insights and guide metric-aware optimization.

Searching arXiv for the cited papers on label set level metrics and related evaluation frameworks. Label set level metrics are evaluation criteria that assess a predicted label set as a set, rather than treating labels independently. In multi-label learning, they are typically defined per instance from the ground-truth set $Y \subseteq [l]$ and the prediction set $\hat Y \subseteq [l]$, then aggregated by averaging over instances; common examples are subset accuracy, precision, recall, $F$-measure, and Jaccard. Closely related literatures use analogous set- or group-level metrics for synthesized multi-label examples, hierarchical ontologies, bags in learning from label proportions, and parser-produced template sets (Mohri et al., 27 May 2026, Park et al., 2022, Alfassy et al., 2019, Falis et al., 2021, Brahmbhatt et al., 2023, Qin et al., 26 Dec 2025).

1. Formal setting and aggregation regimes

In the standard multi-label setting, there are $l$ labels, a ground-truth label set $Y \subseteq [l]$, and a prediction set $\hat Y \subseteq [l]$. Using indicator encodings, the per-instance set counts are

$$\mathrm{TP} = |Y \cap \hat Y|,\qquad \mathrm{FP} = |\hat Y \setminus Y|,\qquad \mathrm{FN} = |Y \setminus \hat Y|,\qquad \mathrm{TN} = l - \mathrm{TP} - \mathrm{FP} - \mathrm{FN}.$$

Instance-averaged metrics compute a per-instance set score and then average across instances. By contrast, micro-averaged metrics pool $\mathrm{TP},\mathrm{FP},\mathrm{FN},\mathrm{TN}$ over all instance-label pairs before forming a ratio, and macro-averaged metrics compute a per-label metric over instances and then average across labels. The instance-averaged regime evaluates the quality of $\hat Y_i$ as a set and captures label dependencies within each instance (Mohri et al., 27 May 2026).

The same distinction reappears in ranking-oriented multi-label settings. In RLSEP, the ground truth is a ranked label set with ranks $r_j$, negatives tied at rank $\hat Y \subseteq [l]$0, and evaluation is defined on ordered label pairs with unequal ranks. This converts an ordered label set into per-instance pairwise decisions, from which precision, recall, $\hat Y \subseteq [l]$1, accuracy, exact match, and ranked mAP are computed (Dari et al., 2022).

A separate but important terminological distinction arises in metric labeling. There, a finite label set $\hat Y \subseteq [l]$2 is equipped with a metric $\hat Y \subseteq [l]$3 satisfying non-negativity, identity of indiscernibles, symmetry, and triangle inequality, and this metric shapes pairwise costs in an optimization objective on a graph. This is a metric on labels, not an evaluation score over predicted label sets (Felzenszwalb, 2018).

2. Canonical example-based metrics

The canonical example-based metrics are defined directly from $\hat Y \subseteq [l]$4 and $\hat Y \subseteq [l]$5 (Mohri et al., 27 May 2026, Park et al., 2022).

Metric	Definition	Characterization
Subset accuracy	$\hat Y \subseteq [l]$6	Exact set match
Precision	$\hat Y \subseteq [l]$7	Predicted-set purity
Recall	$\hat Y \subseteq [l]$8	Ground-truth coverage
$\hat Y \subseteq [l]$9 / $F$0	$F$1, $F$2	Precision–recall trade-off
Jaccard index	$F$3	Overlap relative to union

Exact match is the strictest set-level metric. Jaccard measures overlap relative to the union, with the convention $F$4 (Park et al., 2022). For contrast, Hamming similarity is label-level rather than label-set-level:

$F$5

It aggregates per-label correctness and does not evaluate the prediction as a set (Park et al., 2022).

Ranking-aware variants extend these ideas when the true label set is ordered. RLSEP constructs the set of comparable pairs

$F$6

assigns ground-truth pair labels by whether $F$7, predicted pair labels by whether $F$8, and then computes per-instance pairwise precision, recall, $F$9, and accuracy from $l$0 over pairs. Its exact-match score is

$l$1

and its ranked AP averages precision over successively retained relevance subsets of the positive labels (Dari et al., 2022).

3. Decision-theoretic formulations and metric-aware optimization

A major line of work formalizes label set level metrics as utilities over confusion counts. In the Empirical Utility Maximization framework, many multi-label metrics, including averaged accuracy $l$2, averaged precision, recall, $l$3-measure, and Jaccard, can be written as linear-fractional functions of $l$4. The paper on generalized metric optimization defines micro-, macro-, and instance-averaged empirical families:

$l$5

$l$6

$l$7

The same work derives the linearization $l$8, proves an equivalence between maximizing the population metric and minimizing the expected linearized cost at $l$9, and introduces MMO, a family of algorithms for optimizing generalized linear-fractional metrics with $Y \subseteq [l]$0-consistent surrogates and exact $Y \subseteq [l]$1 factorization (Mohri et al., 27 May 2026).

Confidence estimation can also be made metric-aware at the label-set level. Given a predictive posterior $Y \subseteq [l]$2 over labelsets, the expected accuracy of predicting a fixed set $Y \subseteq [l]$3 under a similarity $Y \subseteq [l]$4 is

$Y \subseteq [l]$5

For exact match, maximizing expected accuracy reduces to choosing the mode:

$Y \subseteq [l]$6

For Jaccard,

$Y \subseteq [l]$7

which is combinatorial for large $Y \subseteq [l]$8 (Park et al., 2022).

The same paper studies seven candidate confidence functions derived from the labelset distribution: High Probability, Top Gap, Shannon Entropy, Collision Entropy, Min-Entropy, Gini impurity, and a Chi-squared statistic. Its empirical conclusion is that three candidates stand out as robust correlates of expected accuracy: Shannon Entropy, Collision Entropy, and the Chi-squared statistic; any single candidate can estimate expected Hamming similarity well, while a learned combination performs best for expected Jaccard and exact match (Park et al., 2022).

In extreme multi-label classification, the Expected Test Utility framework generalizes metric-aware prediction under a per-instance budget $Y \subseteq [l]$9. Utilities decompose over labels as

$\hat Y \subseteq [l]$0

covering instance-wise weighted utilities, macro-averaged utilities, and coverage. The paper derives optimal top-$\hat Y \subseteq [l]$1 rules for utilities linear in prediction-dependent arguments, and a block coordinate ascent algorithm for general macro utilities. It argues that macro measures budgeted at $\hat Y \subseteq [l]$2 and coverage are substantially more sensitive to long-tail performance than standard instance-wise precision@k or propensity-scored variants (Schultheis et al., 2023).

4. Metrics for synthesized and manipulated label sets

Label set level evaluation is especially explicit in multi-label few-shot synthesis. LaSO synthesizes feature vectors whose target label sets are formed by union, intersection, or set difference of the input label sets, and evaluates correctness with two threshold-free metric families (Alfassy et al., 2019).

The first is classification-based evaluation. For a synthesized feature $\hat Y \subseteq [l]$3 with expected label set $\hat Y \subseteq [l]$4, a multi-label classifier $\hat Y \subseteq [l]$5 produces per-class scores $\hat Y \subseteq [l]$6, and mean Average Precision is computed across classes:

$\hat Y \subseteq [l]$7

No thresholding is needed because AP/mAP is ranking-based.

The second is retrieval-based evaluation. Each synthesized feature is used as a query against a labeled pool of real images, and set overlap is measured by IoU:

$\hat Y \subseteq [l]$8

For a query $\hat Y \subseteq [l]$9 with expected label set $$\mathrm{TP} = |Y \cap \hat Y|,\qquad \mathrm{FP} = |\hat Y \setminus Y|,\qquad \mathrm{FN} = |Y \setminus \hat Y|,\qquad \mathrm{TN} = l - \mathrm{TP} - \mathrm{FP} - \mathrm{FN}.$$0 and top-$$\mathrm{TP} = |Y \cap \hat Y|,\qquad \mathrm{FP} = |\hat Y \setminus Y|,\qquad \mathrm{FN} = |Y \setminus \hat Y|,\qquad \mathrm{TN} = l - \mathrm{TP} - \mathrm{FP} - \mathrm{FN}.$$1 retrieved label sets $$\mathrm{TP} = |Y \cap \hat Y|,\qquad \mathrm{FP} = |\hat Y \setminus Y|,\qquad \mathrm{FN} = |Y \setminus \hat Y|,\qquad \mathrm{TN} = l - \mathrm{TP} - \mathrm{FP} - \mathrm{FN}.$$2,

$$\mathrm{TP} = |Y \cap \hat Y|,\qquad \mathrm{FP} = |\hat Y \setminus Y|,\qquad \mathrm{FN} = |Y \setminus \hat Y|,\qquad \mathrm{TN} = l - \mathrm{TP} - \mathrm{FP} - \mathrm{FN}.$$3

Reported values use $$\mathrm{TP} = |Y \cap \hat Y|,\qquad \mathrm{FP} = |\hat Y \setminus Y|,\qquad \mathrm{FN} = |Y \setminus \hat Y|,\qquad \mathrm{TN} = l - \mathrm{TP} - \mathrm{FP} - \mathrm{FN}.$$4 and separate seen from unseen label sets (Alfassy et al., 2019).

The paper explicitly does not use subset accuracy, Hamming loss, or per-instance precision/recall/$$\mathrm{TP} = |Y \cap \hat Y|,\qquad \mathrm{FP} = |\hat Y \setminus Y|,\qquad \mathrm{FN} = |Y \setminus \hat Y|,\qquad \mathrm{TN} = l - \mathrm{TP} - \mathrm{FP} - \mathrm{FN}.$$5 for operation correctness. In its COCO validation results, classification mAP on seen classes is $$\mathrm{TP} = |Y \cap \hat Y|,\qquad \mathrm{FP} = |\hat Y \setminus Y|,\qquad \mathrm{FN} = |Y \setminus \hat Y|,\qquad \mathrm{TN} = l - \mathrm{TP} - \mathrm{FP} - \mathrm{FN}.$$6 for intersection, $$\mathrm{TP} = |Y \cap \hat Y|,\qquad \mathrm{FP} = |\hat Y \setminus Y|,\qquad \mathrm{FN} = |Y \setminus \hat Y|,\qquad \mathrm{TN} = l - \mathrm{TP} - \mathrm{FP} - \mathrm{FN}.$$7 for union, and $$\mathrm{TP} = |Y \cap \hat Y|,\qquad \mathrm{FP} = |\hat Y \setminus Y|,\qquad \mathrm{FN} = |Y \setminus \hat Y|,\qquad \mathrm{TN} = l - \mathrm{TP} - \mathrm{FP} - \mathrm{FN}.$$8 for subtraction, while retrieval mIoU@1/3/5 for seen intersection is $$\mathrm{TP} = |Y \cap \hat Y|,\qquad \mathrm{FP} = |\hat Y \setminus Y|,\qquad \mathrm{FN} = |Y \setminus \hat Y|,\qquad \mathrm{TN} = l - \mathrm{TP} - \mathrm{FP} - \mathrm{FN}.$$9 and for seen union $\mathrm{TP},\mathrm{FP},\mathrm{FN},\mathrm{TN}$0. In few-shot classification on unseen COCO classes, learned union augmentation yields mAP $\mathrm{TP},\mathrm{FP},\mathrm{FN},\mathrm{TN}$1 in 1-shot and $\mathrm{TP},\mathrm{FP},\mathrm{FN},\mathrm{TN}$2 in 5-shot, compared with $\mathrm{TP},\mathrm{FP},\mathrm{FN},\mathrm{TN}$3 and $\mathrm{TP},\mathrm{FP},\mathrm{FN},\mathrm{TN}$4 for mixUp (Alfassy et al., 2019).

These choices illustrate an important design principle. When the task is not merely to output a binary set but to synthesize a representation whose induced label set should be correct and geometrically meaningful, class-wise ranking metrics and retrieval overlap metrics can be more diagnostic than exact match or instance-level thresholded scores. This suggests a broader distinction between set-level correctness in output space and set-level locality in representation space.

5. Hierarchical, bag-level, and label-free generalizations

Hierarchical label spaces require metrics that preserve structural information beyond flat set overlap. CoPHE addresses large-scale multi-label text classification on the ICD-9 ontology by replacing inconsistent parent/grandparent relations with a depth-based representation using levels $\mathrm{TP},\mathrm{FP},\mathrm{FN},\mathrm{TN}$5, $\mathrm{TP},\mathrm{FP},\mathrm{FN},\mathrm{TN}$6, $\mathrm{TP},\mathrm{FP},\mathrm{FN},\mathrm{TN}$7, and an optional chapter level $\mathrm{TP},\mathrm{FP},\mathrm{FN},\mathrm{TN}$8. For document $\mathrm{TP},\mathrm{FP},\mathrm{FN},\mathrm{TN}$9 and node $\hat Y_i$0, predicted and gold descendant counts are

$\hat Y_i$1

Per-node count-preserving confusion counts are

$\hat Y_i$2

which are micro-aggregated across nodes, levels, and documents into hierarchical precision, recall, and $\hat Y_i$3. The count-preserving property retains multiplicities that set-based augmentation discards, so over- and under-prediction within ancestor families remain visible (Falis et al., 2021).

In learning from label proportions, LLP-Bench uses label-set-level hardness metrics not to score predictions directly, but to characterize how informative bag supervision is. The four reported metrics are LabelPropStdev, InterIntraRatio, MeanBagSize, and CumuBagSizeDist. LabelPropStdev is the standard deviation of bag label proportions,

$\hat Y_i$4

and is interpreted as “supervision spread” over bags. InterIntraRatio compares mean inter-bag to mean intra-bag squared Euclidean separation,

$\hat Y_i$5

and measures geometric separability. MeanBagSize measures constraint density per instance, while CumuBagSizeDist summarizes the empirical bag-size distribution by $\hat Y_i$6. Across LLP-Bench feature-bag datasets, larger LabelPropStdev and InterIntraRatio correlate with better AUC or lower MSE, whereas larger MeanBagSize and heavier tails are associated with harder LLP (Brahmbhatt et al., 2023).

A different generalization appears in log parsing, where PMSS is a label-free template-level metric. Given parser-produced event groups and templates, PMSS assigns each message a silhouette-like score using the parser’s template as medoid and token-level Levenshtein distance:

$\hat Y_i$7

with $\hat Y_i$8 the distance to the assigned template and $\hat Y_i$9 the minimum distance to neighboring templates after sorting. Group-level scores are averaged into

$r_j$0

PMSS evaluates cohesion and separation without ground-truth labels and is significantly positively correlated with the label-based template-level metrics FGA and FTA, with Spearman’s $r_j$1 of $r_j$2 and $r_j$3, respectively (Qin et al., 26 Dec 2025).

6. Interpretation, selection criteria, and limitations

Label set level metrics differ chiefly in what they reward. Instance-averaged set metrics are appropriate when performance is fundamentally about the predicted set per instance, when intra-instance label overlap and set size matter, and when false positives and false negatives interact at the set level. Within this family, $r_j$4 is the harmonic mean of precision and recall, whereas Jaccard normalizes by union size and penalizes over- and under-prediction more aggressively when $r_j$5 is small (Mohri et al., 27 May 2026).

Several works make the corresponding negative point. In LaSO, exact match, Hamming loss, and instance-level precision/recall/$r_j$6 are not used for operation correctness; instead, mAP and mIoU are chosen because they are threshold-free and directly reflect label-set manipulation and embedding locality (Alfassy et al., 2019). In generalized metric optimization, exact match is one example-based metric among many, but the broader linear-fractional formulation covers averaged precision, recall, $r_j$7-measure, Jaccard, and averaged accuracy, and supports direct optimization with non-asymptotic guarantees (Mohri et al., 27 May 2026). In XMLC, standard precision@k and propensity-scored variants may barely register severe tail neglect, whereas macro utilities and coverage@k are highly sensitive to it (Schultheis et al., 2023).

Hierarchical and structured settings add further caveats. CoPHE assumes a tree-structured ontology and leaves handling general DAGs, such as SNOMED CT, for future work (Falis et al., 2021). The generalized EUM theory currently targets linear-fractional metrics, and extending beyond that class remains open; noisy or partial labels, architectural constants in $r_j$8-consistency bounds, and learned metric-adaptive threshold policies are also identified as open questions (Mohri et al., 27 May 2026). PMSS avoids ground-truth dependence, but its conclusions depend on tokenization, variable handling, the Levenshtein distance, and a neighbor-template approximation for inter-group dissimilarity (Qin et al., 26 Dec 2025).

A final distinction concerns nomenclature. In evaluation, label set level metrics score predictions, bags, groups, or template sets. In metric labeling, by contrast, a metric on the label set is part of the optimization objective:

$r_j$9

and the triangle inequality of $\hat Y \subseteq [l]$00 underlies the center-star $\hat Y \subseteq [l]$01-approximation on complete graphs (Felzenszwalb, 2018). This suggests that “label set level metrics” names two related but different mathematical objects: evaluation functionals over predicted sets, and metrics defined on label spaces themselves.