Additional benefit of joint NLL–CwECE ranking

Determine whether jointly ranking accuracy-eligible checkpoints by normalized negative log-likelihood and class-wise calibration error provides an additional benefit over accuracy-constrained single-objective reliability ranking in domain generalization checkpoint selection.

Background

The paper introduces accuracy-constrained reliability selection (AC), which first retains checkpoints whose source-validation accuracy is within a prescribed tolerance of the best checkpoint and then ranks the eligible checkpoints using source reliability metrics. The reference configuration, AC-NC, jointly aggregates normalized negative log-likelihood (NLL) and class-wise expected calibration error (CwECE) using the infinity norm, while AC-NLL and AC-CwECE rank checkpoints using only one reliability objective.

Experiments on PACS, OfficeHome, and TerraIncognita do not resolve whether the joint NLL–CwECE objective is superior to single-objective ranking. The reported paired comparisons have confidence intervals that include zero, so establishing an additional benefit of joint ranking requires further evidence beyond the evaluated trajectories and cohorts.