Use-Case-Specific Evaluation Metrics

• Use-case-specific evaluation metrics are specialized tools that integrate domain knowledge, stakeholder priorities, and operational constraints to capture nuanced system performance.
• They incorporate methods like weighted error costing and tailored aggregation to emphasize critical, low-frequency, or high-cost classes while countering limitations of conventional metrics.
• These metrics improve model selection and deployment by offering transparent trade-offs and aligning evaluation with the practical demands of diverse real-world applications.

Use-case-specific evaluation metrics are specialized measures designed to reflect the true effectiveness of machine learning systems in their intended deployment contexts. Unlike general-purpose metrics such as accuracy or macro-F1, which may fail to capture the nuances and priorities of practical applications, use-case-specific metrics incorporate domain knowledge, stakeholder priorities, data distributions, and operational constraints. The motivation for developing these metrics stems from well-documented failures of conventional metrics, particularly in imbalanced, multi-class, or safety-critical settings, where performance on critical but rare classes or highly informative errors carries disproportionately greater significance.

1. Motivation: Shortcomings of Conventional Metrics

Classical metrics like accuracy and macro-F1 summarize overall correctness or balanced class-wise performance, but exhibit two fundamental pitfalls in many real-world scenarios:

• Class Imbalance Masking: When one class dominates, accuracy becomes non-informative about minority classes—e.g., in rumour stance classification, the “comment” class comprises 66–72% of instances. A classifier can achieve deceptively high accuracy or macro-F1 even when crucial classes (“support”, “deny”) are neglected (Scarton et al., 2020).
• Insensitivity to Application Priority: Not all errors have the same consequence. For example, in rumour verification and clinical diagnosis, missing a “deny” or a “disease present” case, respectively, is more damaging than other errors. Conventional metrics, by equally weighting all classes or samples, cannot encode this asymmetry.

Similar issues arise in language generation, where BLEU, ROUGE, or CIDEr-D often prefer repetitive system outputs to human-authored texts, ignore rare but critical linguistic phenomena, and can be “gamed” to produce high scores for trivial outputs (Caglayan et al., 2020). In time series anomaly detection, point-wise f-score and its relatives can reward partial or imprecise detections and are highly sensitive to class imbalance, missing critical use-case needs (Sørbø et al., 2023).

This systematic misalignment has motivated a proliferation of research on metrics that explicitly target the desiderata of real-world use cases.

2. Design Principles of Use-Case-Specific Metrics

A. Emphasizing Informative or Rare Classes

Metrics such as the weighted area under the curve (wAUC), geometric mean of recalls (GMR), and weighted macro-Fβ (wFB) penalize models that underperform on high-value, low-frequency classes:

• GMR (Geometric Mean of Recalls):

$$\mathrm{GMR} = \left(\prod_{c=1}^C R_c\right)^{1/C}$$

where $R_c$ is the recall for class $c$. As the geometric mean is sensitive to very low values, extremely poor recall in a minority class drastically lowers GMR (Scarton et al., 2020).

• Weighted macro-Fβ (wFB):

$\mathrm{wFB} = \sum_{c=1}^{C} w_c F_{\beta,c}$

Class weights $w_c$ encode domain priorities, such as $w_{support} = 0.4$, $w_{deny} = 0.4$, $w_{query} = 0.15$, $w_{comment} = 0.05$. $\beta$ can be increased (e.g., $\beta=2$) to prioritize recall, which is critical when missing a minority class is highly undesirable.

B. Domain-Tailored Aggregation and Error Costing

The expected cost (EC) metric generalizes error rate by integrating domain-specific misclassification costs and priors:

$$EC = \frac{1}{N} \sum_t C(h^{(t)}, d^{(t)}) = \sum_{i=1}^K \sum_{j=1}^D c_{ij} P_i R_{ij}$$

where $c_{ij}$ is the application-determined cost of predicting $D_j$ when the true class is $H_i$, and $P_i$ can reflect operational class proportions (Ferrer, 2022). EC allows, for example, false negatives in disease detection to be given orders of magnitude higher cost than false positives.

Proper scoring rules (PSRs), e.g., cross-entropy and Brier score, enable calibration-sensitive assessment tuned to the particular discrimination and confidence needs of the use case (Ferrer, 2022). Calibration loss computed via PSRs is preferable to expected calibration error (ECE), especially in multi-class and risk-sensitive settings.

3. Case Studies: Constructing and Applying Use-Case-Specific Metrics

Rumour Stance Classification

The RumourEval shared tasks highlighted that macro-F1 and accuracy can reward systems that simply predict the majority (“comment”) class, missing “support” and “deny” entirely (Scarton et al., 2020). The introduction of GMR, wAUC, and wFB, with high weights on “support” and “deny”, transformed system ranking. For instance, Turing achieved top accuracy by ignoring “deny” instances altogether (zero recall), but ranked lower when evaluated with GMR and wFB, which expose this failure.

Dialogue and Generation Tasks

Hierarchical, modular metrics such as USL-H (Understandability, Sensibleness, Likability in Hierarchy) allow practitioners to prioritize and reweight qualities critical to a use case:

$$s_{\text{USL–H}} = \alpha_1 s_U + \alpha_2 s_S + \alpha_3 s_S s_L$$

with sub-metrics for understandability, sensibleness, and likability (e.g., empathy, specificity), and task-dependent $\alpha$ (Phy et al., 2020). Experiments show that this hierarchical metric configuration yields stronger correlation with human judgment relative to flat metrics like BLEU or BERTScore.

In time series anomaly detection, metrics must reflect whether early detection, event coverage, and brevity of alarms matter most. The taxonomy in (Sørbø et al., 2023) formalizes metric properties such as early detection, partial detection, proximity, and gives selection criteria based on downstream needs.

Explanation and Recourse Selection

For explainability and recourse (e.g., counterfactual explanations in credit decisions), artificial metrics such as $L_1$ proximity are often misaligned with user priorities. Empirical studies show user preference matches proximity-based optima in only about 64% of cases, highlighting the need for metrics grounded in user-perceived effort. Personalized weighted proximity and acceptability thresholds (as in the AWP model) yield far greater user alignment, with up to 84% prediction accuracy for user-preferred explanations (Choudhury et al., 20 Jul 2025).

4. Evaluation Methodology: Statistical Procedures and Metric Construction

The development and selection of use-case-specific metrics is increasingly supported by rigorous meta-evaluation frameworks:

• Reliability and Validity Assessment: The MetricEval framework applies measurement theory concepts (reliability, validity, stability, consistency) with formal estimators such as test–retest correlations and Cronbach’s α to examine if a metric robustly captures its intended construct (Xiao et al., 2023).
• Pairwise Aggregation: In grammatical error correction (GEC), sentence-level scores are aggregated via pairwise comparisons and rating algorithms (TrueSkill), aligning system rankings more closely with human evaluation, especially for tasks with substantial rewriting where absolute scores may not transfer into ordinal preferences (Goto et al., 13 Feb 2025).
• Holistic Indices: Multi-objective contexts, such as federated learning, require composite indices (e.g., HEM) that aggregate accuracy, convergence, computational efficiency, fairness, and personalization, with weights determined by use-case needs (e.g., IoT vs. institutional deployment), leading to more representative algorithm selection (Li et al., 3 May 2024).

5. Impact on Deployment, Research, and Future Directions

The adoption of use-case-specific metrics impacts both technology development and practical deployment:

• Model Selection: Systems that excel under generic metrics may be suboptimal or even dangerous when deployed—e.g., high-accuracy vision models can result in large downstream errors in ecological population estimates if they misclassify rare but biologically meaningful behaviors (Chan et al., 5 May 2025).
• Transparent Trade-offs: Weighted metrics and multi-objective frameworks clarify trade-offs (e.g., speed vs. accuracy vs. fairness), aiding informed stakeholder decision making in high-stakes domains.
• Benchmark and Dataset Evolution: The field is moving toward embedding application-specific metrics into dataset benchmarks, requiring molecule generators, vision systems, and LLMs to be evaluated on criteria that reflect real operational requirements and societal impact, as in graph generative modeling (Thompson et al., 2022), DNN compression (Ghobrial et al., 2023), or ecological studies (Chan et al., 5 May 2025).
• Human-Centric Adaptation: User studies driving metrics, as in counterfactual recourse (Choudhury et al., 20 Jul 2025), point toward a paradigm where explainability, actionability, and acceptance are not abstract desiderata but quantifiable, user-driven criteria for system evaluation.

6. Open Challenges and Prospects

While use-case-specific evaluation metrics are essential for robust deployment, their construction poses several challenges:

• Determining Appropriate Weights and Cost Matrices: Identifying and justifying the assignment of weights or costs is inherently subjective and must be informed by empirical paper, operational data, and stakeholder input.
• Extending to Multi-Modal and Complex Pipelines: Many workflows involve multi-stage pipelines or jointly optimized objectives (e.g., vision-for-behavioral ecology); cross-component evaluation integrates metrics with distinct mathematical properties.
• Generalization Across Domains: While frameworks and the need for customization are clear, the transferability of weighting schemes, aggregation strategies, and statistical methods in new domains requires further research and transparent reporting.

7. Summary Table: Properties of Use-Case-Specific Metrics

Metric/Class	Customizable Weights	Reflects Minority/Cost Asymmetry	Task Alignment
Accuracy	No	No	Poor
Macro-F1	No	Weak	Limited
GMR, wAUC, wFB	Yes	Yes	Strong
Expected Cost (EC)	Yes	Yes (cost/prior matrix)	Strong
USL-H (Dialogue)	Yes	Yes (qualities, hierarchy)	Strong
Application-Specific	Yes	Yes (domain-anchored metrics)	Maximal

Conclusion

Use-case-specific evaluation metrics provide a principled framework for resolving the discord between conventional evaluation and operational performance. By explicitly encoding application priorities, error costs, user feasibility, and domain-specific artifacts, they support model development, deployment, and benchmarking that are truly fit for purpose. Research continues to advance methodologies for metric design, reliability analysis, and human alignment, marking a shift toward evaluation as a central, application-aware activity in machine learning and AI system lifecycle.