CARE Metric: Interpretable Evaluation Across Domains

• CARE metric is a family of rigorously defined, interpretable evaluation metrics that quantify performance in radiology, anomaly detection, and financial risk forecasting.
• It integrates domain-specific methods by employing LLM-driven MCQ analysis for radiology, critical event detection for machinery, and expectile modeling for tail risk.
• CARE metric enhances transparency by decomposing performance into interpretable sub-scores, enabling targeted improvements and robust benchmarking in diverse applications.

CARE (an acronym variously instantiated as Clinically-grounded Agent-based Report Evaluation, Coverage-Accuracy-Reliability-Earliness, or Conditional Autoregressive Expectile, depending on the research domain) refers to a family of rigorous, interpretable metrics designed for distinct high-value evaluation tasks in machine learning, anomaly detection, and financial time series modeling. The term "CARE metric" encompasses foundational methodologies for (1) clinical report evaluation via LLM-driven MCQA, (2) performance quantification of anomaly detectors on real-world machinery datasets, and (3) tail risk forecasting in finance. Each instantiation of CARE is methodologically distinct yet shares an emphasis on interpretable, multi-faceted, and domain-grounded evaluation.

1. CARE in Radiology: Clinically-grounded Agent-based Report Evaluation

CARE—Clinically-grounded Agent-based Report Evaluation—provides an interpretable, LLM-driven scoring framework to assess the clinical fidelity of generated radiology reports. It operationalizes report comparison as a dynamic multiple-choice question answering (MCQA) task using two LLM-based agents—one with access to the ground-truth report (Agent₍GT₎) and the other with the candidate report (Agent₍GEN₎) (Dua et al., 4 Aug 2025).

Metric Workflow:

• Each agent generates clinically meaningful MCQs from its report, forming two question sets.
• To ensure clinical specificity, each question is retained only if answered correctly “with report” and incorrectly “without report” by the LLM.
• Agents answer both sets, producing agreement patterns interpretable as proxies for precision and recall:
  • ICARE-GT (S_GT): Agreement on ground-truth–generated questions (clinical precision).
  • ICARE-GEN (S_GEN): Agreement on generated-report–generated questions (clinical recall).
• The overall CARE score is the unweighted mean: $$\mathrm{CARE} = (S_{\mathrm{GT}} + S_{\mathrm{GEN}}) / 2$$.

Algorithmic Summary:

Step	Action	Purpose
MCQ Generation	Each agent prompts LLM for report-based MCQs	Surface true report content
MCQ Filtering	Only questions correctly answered with the report and not otherwise are retained	Enforce report-specificity
Answering	Agents answer both sets using respective reports	Gather agreement data
Scoring	Compute S_GT, S_GEN, CARE	Quantify fidelity and error type

By linking scores to interpretable QA pairs, clinicians can localize omissions or hallucinations via specific disagreements, supporting model development, deployment monitoring, and regulatory documentation.

2. CARE for Predictive Maintenance: Wind Turbine Anomaly Detection

Within anomaly detection for wind turbines, CARE (Coverage, Accuracy, Reliability, Earliness) is a weighted composite metric designed to evaluate model performance across four operationally critical axes (Gück et al., 16 Apr 2024):

1. Coverage: Point-wise $F_\beta$ score on labeled anomaly-event datasets, reflecting detection of true anomalies with an explicit bias toward penalizing false positives ($\beta=0.5$).
2. Accuracy: Proportion of true negatives on purely normal datasets, strictly penalizing false alarms.
3. Reliability: Event-level $F_\beta$ via a "criticality" algorithm that collapses time series to binary alarms based on sustained detection (consecutive flagged anomalies), ensuring event-level robustness.
4. Earliness: Weighted score rewarding earlier detection within annotated anomaly windows, with detections in the latter half of the window linearly downweighted.

CARE Aggregation and Safeguards:

Let $\overline{F_\beta}$ (Coverage), $\overline{WS}$ (Earliness), $EF_\beta$ (Reliability), and $\overline{Acc}$ (Accuracy) be the averaged scores by category, and $WA$ the weighted average (normal datasets doubly weighted):

$$WA = \frac{1}{5} \left( \overline{F_\beta} + \overline{WS} + EF_\beta + 2\overline{Acc} \right)$$

$$CARE = \begin{cases} 0, & \text{if no anomalies detected} \ \overline{Acc}, & \text{if } \overline{Acc} < 0.5 \ WA, & \text{otherwise} \end{cases}$$

By ensuring performance on normal data ($\overline{Acc}$) is a gating term, the metric prevents models from "winning" through either over-flagging or neglecting detection until late in the anomaly period.

Benchmark Results:

• Tuned Autoencoder: CARE ≈ 0.66 (best-balanced).
• Isolation Forest: CARE ≈ 0.48 (excessive false alarms).
• Random, all-anomaly, and all-normal baselines: CARE ≤ 0.5.

3. CARE in Financial Risk: Conditional Autoregressive Expectile Modeling

In financial econometrics, CARE stands for Conditional Autoregressive Expectile, a semi-parametric time-series forecast model for tail risk, introduced as an alternative to GARCH and quantile regression for Value-at-Risk (VaR) and Expected Shortfall (ES) estimation (Gerlach et al., 2016).

Expectile Modeling:

• For observed return $r_t$, the $\tau$-level conditional expectile $\mu_t$ minimizes:

$$\min_{\mu}\;E\bigl[|\tau-\mathbf{1}\{Y<\mu\}|(Y-\mu)^2\bigr]$$

• Simple (SAV) CARE dynamics:

$$\mu_t = \beta_1 + \beta_2\,\mu_{t-1} + \beta_3\,|r_{t-1}|$$

• ES is computed directly via Newey–Powell scaling:

$$ES_\alpha = \left(1 + \frac{\tau}{(1-2\tau)\alpha}\right)\mu_t$$

Realized-CARE Extension:

• Adds a measurement equation relating expectile $\mu_t$ and realized measures $x_t$ (e.g., realized volatility, range):

$$\begin{cases} \mu_t = \beta_1 + \beta_2\,\mu_{t-1} + \beta_3\,x_{t-1} \ x_t = \xi + \phi |\mu_t| + \tau_1 \epsilon_t + \tau_2 (\epsilon_t^2 - \overline{\epsilon}^2) + u_t \end{cases}$$

• Expectile level ($\tau$) is optimized by minimizing quantile (tick) loss, rather than matching empirical violation rates:

$$QL(\tau) = \sum_{t=1}^m (\alpha - \mathbf{1}\{r_t<\mu_t(\tau)\})(r_t - \mu_t(\tau))$$

Empirical Performance:

• Realized-CARE models outperform GARCH, standard CARE, and even Realized-GARCH on violation rates, joint VaR-ES loss, and model confidence set inclusion, particularly with sub-sampled realized range.

4. Comparative Structure and Interpretability

Despite distinct application domains, all CARE metrics share:

• Composite, multi-objective design: integrating coverage/sensitivity, specificity, event-wise reliability, and timing/precision of detection or prediction.
• Explicit interpretability: All report decision elements (question–answer pairs, confusion-matrix statistics, event-alarm aggregation, etc.) are directly traceable.
• Robustness safeguards: Gating on normal-behavior performance or penalizing late/over-flagged predictions prevents pathological optimization.
• Alignment with expert or operational priorities: Radiology CARE aligns with board-certified specialist judgment; anomaly detection CARE aligns with maintenance cost trade-offs; financial CARE aligns with capital efficiency and regulatory back-testing.

5. Methodological Details and Pseudocode

All CARE incarnations specify algorithmic steps and equations. For radiology, the pipeline includes LLM instantiation, MCQ creation, filtering (dependent on LLM answer consistency), agreement extraction, and symmetric precision–recall scoring (Dua et al., 4 Aug 2025). For wind turbine anomaly detection, explicit pseudocode defines the criticality counter for event-level alarms, and well-defined formulas aggregate coverage, accuracy, reliability, and earliness (Gück et al., 16 Apr 2024). For time series, the model iterates over parameter estimation via ALS or Bayesian MCMC, grid search over expectile levels, and joint loss minimization (Gerlach et al., 2016).

6. Practical Impact, Benchmarks, and Application Domains

CARE has established new evaluation standards in each of its domains. In radiology, it robustly correlates with expert review, exposes omission/hallucination patterns at the clinical finding level, and supports deployment monitoring. In anomaly detection, it defines the currently most stringent benchmark for real-world wind-farm datasets, penalizing both undetected and false events as well as delayed alerts. In finance, CARE’s improved VaR/ES performance enables more efficient allocation under Basel accords, with strong empirical support across diverse asset classes.

7. Broader Implications and Future Directions

The CARE family illustrates a trend towards interpretable, multi-criteria evaluation grounded in real operational and clinical requirements. While each metric is highly tailored, the underlying pattern—decomposition of performance into domain-relevant, interpretable subscores, corresponding aggregation rules, and robust filtering against trivial solutions—suggests broad applicability for critical decision settings beyond those studied. A plausible implication is the increasing integration of agent-based evaluation, temporal weighting, and semi-parametric approaches for reliability and transparency in automated decision-support systems.