Community-aware evaluation and threshold calibration for open-set plankton image recognition

Abstract: Automated plankton image recognition is increasingly used in aquatic ecosystem monitoring, but deployed classifiers inevitably encounter unseen taxa and non-target particles. Open-set recognition methods are usually evaluated with sample-level metrics such as AUROC, AUPR, and FPR@95% unknown-recall operating points, whereas ecological monitoring depends on community-level estimates of taxon abundance and diversity. This study examines the mismatch between these objectives using controlled pseudo-communities and three datasets spanning marine zooplankton imaged by ZooScan, marine phytoplankton imaged by IFCB, and freshwater plankton imaged by an in-situ camera. We define Open-Set Community Distortion (OSCD), a Bray-Curtis-style error over known taxa plus an unknown bin, with directional components distinguishing known-taxon overestimation from underestimation. Closed-set classifiers achieved high known-class accuracy, but unknown samples were often absorbed with high confidence and in structured ways. Sample-level OOD metrics were not sufficient to select ecological operating points: for MSP, FPR@95% unknown-recall thresholds produced large test-community OSCD on all three datasets mainly because true known taxa were over-rejected into the unknown bin. Community-aware threshold calibration reduced MSP OSCD relative to fixed 95% known recall on SYKE-ZooScan 2024 and SYKE-IFCB 2022; on ZooLake the fixed-recall baseline was already close to the community-aware threshold, and the best community-level method was a prototype-distance variant rather than MSP. The benefit of community-aware calibration therefore depends on validation-community representativeness and the gap between fixed recall and the community optimum. These results show that open-set plankton recognition should be evaluated as an ecological measurement problem, not only as a sample-level detection task.

• The paper introduces Open-Set Community Distortion (OSCD), showing that community-aware calibration reduced test OSCD from 0.4977 to 0.1191 while increasing known-sample recall from 0.4275 to 0.9217 on the primary benchmark.
• Unknown plankton were absorbed into known taxa with high confidence—up to 50.4% exceeded 0.90 confidence—and these structured errors created taxon-specific abundance biases that sample-level metrics failed to reveal.
• Community-aware thresholds improved results when unknown ratios and validation communities reflected deployment conditions, but long-tail and endpoint-specific goals such as richness, diversity, and dominance require tailored calibration objectives.

Motivation: from image labels to ecological measurements

Automated plankton imaging systems such as ZooScan and IFCB generate classification outputs that are ultimately aggregated into community-level summaries—relative abundances, diversity indices, evenness, richness, and dominant-taxon structure. Yet open-set recognition methods for plankton imagery are conventionally evaluated with sample-level metrics (AUROC, AUPR, FPR@95% unknown recall), which measure whether individual known and unknown images can be separated but say nothing about whether the deployed threshold preserves abundance estimates. This paper argues that thresholded open-set recognition should be treated as an abundance-estimation problem, and it substantiates that argument with controlled pseudo-community experiments on three datasets spanning marine zooplankton (SYKE-ZooScan 2024), marine phytoplankton (SYKE-IFCB 2022), and freshwater plankton (ZooLake) (2605.15835).

Evaluation framework

The methodological core is Open-Set Community Distortion (OSCD), a Bray-Curtis distance between true and predicted relative abundance vectors over $K$ known taxa plus one unknown bin. Two directional diagnostics accompany it: OSCD$^{+}$ quantifies overestimation of known taxa (unknown absorption into known classes, potentially creating false apparent blooms), while OSCD$^{-}$ quantifies underestimation of known taxa (over-rejection of real organisms into the unknown bin). For a single normalized community, OSCD equals $\max(\mathrm{OSCD}^{+}, \mathrm{OSCD}^{-})$; the authors are careful to note this identity does not hold after averaging across pseudo-communities, so the directional components must be read as diagnostics rather than an additive decomposition.

Pseudo-communities of 500 images were resampled with replacement under six controlled structures (empirical, balanced, unknown-ratio-controlled, dominant-taxa, long-tail, non-target-enriched) at unknown ratios of 0–0.4, with five seeds used to verify robustness. The classifier is deliberately simple—a frozen DINOv2 ViT-S/14 encoder with a linear head—and OOD scores (MSP, energy, prototype distances, Mahalanobis) are applied post hoc, isolating the threshold-selection question from model-design questions.

Structured, high-confidence unknown absorption

Closed-set performance was strong on all three datasets (known-class accuracy 0.9655, 0.9643, 0.9760), so subsequent errors cannot be attributed to failed training. Unknown samples were nonetheless absorbed with high confidence: mean maximum softmax confidence reached 0.71 on SYKE-ZooScan 2024, 0.82 on SYKE-IFCB 2022, and 0.77 on ZooLake, with 28.0%, 50.4%, and 38.5% of unknowns exceeding 0.90 confidence respectively. Notably, IFCB phytoplankton imagery was at least as vulnerable as the ZooScan benchmark.

More consequential is that absorption was structured rather than uniform. Each unknown category concentrated its misclassifications in one or a few specific known taxa—for example, Bivalvia was predominantly absorbed into Gastropoda on SYKE-ZooScan 2024. The implication is direct: open-set errors do not average out at the community level but produce taxon-specific abundance biases, so the ecological damage depends on which taxa receive absorbed mass, not merely how many images are misassigned.

Sample-level thresholds distort communities through over-rejection

MSP achieved the best sample-level AUROC on all three datasets (0.9164, 0.8828, 0.8932), but method rankings by AUROC did not reliably predict community-level rankings; Spearman agreement ranged from 0.90 on SYKE-ZooScan down to 0.29 on SYKE-IFCB, and on ZooLake a prototype-distance variant outperformed MSP at the community level despite MSP's higher AUROC.

The sharpest result concerns threshold selection. On the primary benchmark, the sample-level detection-F1 optimum for MSP occurred at a dramatically lower known-sample recall than the community optimum ($\Delta$recall ≈ 0.48). Community-aware calibration simultaneously increased test known-sample recall from 0.4275 to 0.9217 and reduced mean test-community OSCD from 0.4977 to 0.1191—an unambiguous demonstration that detection-optimal thresholds destroy abundance estimates. Across all three datasets, FPR@95% unknown-recall thresholds produced large OSCD dominated almost entirely by OSCD$^{-}$ (e.g., 0.6953 vs. OSCD$^{+}$ = 0.0000 on SYKE-ZooScan): aggressive rejection erases real known taxa rather than protecting the community from contamination. This failure mode is invisible to AUROC or detection F1 but decisive for ecological inference.

Community-aware calibration: gains and conditions

Selecting the threshold to minimize validation-community OSCD improved MSP results on two datasets. Across five seeds, MSP community-aware calibration reduced test OSCD from $0.1353 \pm 0.0005$ to $0.1184 \pm 0.0007$ on SYKE-ZooScan (global oracle $0.1162 \pm 0.0003$), and from $^{+}$0 to $^{+}$1 on SYKE-IFCB. On ZooLake, however, fixed 95% known recall already sat near the community optimum ($^{+}$2 vs. $^{+}$3), leaving no room for threshold-level gain—the best non-oracle method there was instead a prototype-distance score ($^{+}$4).

An applicability-boundary analysis on the primary benchmark located the transition around an unknown ratio of 0.2: community-aware calibration became useful once unknowns reached 20%, with the clearest gains at ratio 0.4 (e.g., dominant-taxa communities improved from $^{+}$5 to $^{+}$6). Two boundary cases qualify these gains. Non-target-enriched communities showed non-monotonic behavior—calibration helped at ratio 0.1 but hurt at 0.2 and 0.4—because non-target categories were absent from positive-ratio validation communities. Long-tail communities saw negligible improvement (best sample $^{+}$7 vs. CA $^{+}$8, against a setting oracle of $^{+}$9), suggesting they require diversity-sensitive objectives rather than OSCD alone. Most pointedly, empirical validation communities had mean unknown ratio 0.1004 versus 0.7061 in empirical test communities, and calibrated thresholds did not transfer—validation communities must represent deployment structure for calibration to be reliable.

Ecological endpoint trade-offs

OSCD proved a strong proxy for relative-abundance error (Spearman $^{-}$0) but only moderate for diversity-related summaries (Shannon $^{-}$1, Simpson $^{-}$2, Pielou $^{-}$3) and weak for richness error ($^{-}$4) and top-3 dominant-taxon error ($^{-}$5). Thresholds calibrated to different endpoints selected different operating points: Shannon-/Simpson-aware calibration lowered diversity error but raised OSCD to 0.2124, while top-3-aware calibration improved dominance overlap (0.8160) at the cost of OSCD 0.2672. No single threshold optimizes all ecological summaries; OSCD is a defensible default for composition targets but should be paired with endpoint-specific calibration when diversity or dominance structure is the monitoring priority.

Limitations and open questions

The paper is explicit about its constraints. All experiments use a frozen DINOv2 encoder with post-hoc scores; fine-tuned encoders, open-set training losses, generative unknown models, or hierarchy-aware classifiers may change both sample-level separation and the relative ranking of scores, and would still require community-level threshold evaluation. The benefit of community-aware calibration is conditional on validation-community representativeness and on a meaningful gap between fixed-recall and community-optimal thresholds, and the paired significance tests serve only as robustness diagnostics over pseudo-community resampling rather than evidence from independent deployments. Open questions include whether the observed unknown-ratio transition near 0.2 generalizes beyond these three datasets, whether diversity-targeted calibration can close the long-tail gap to the setting oracle, and how the framework behaves in real cross-system deployments where validation-test representativeness drifts continuously.

Conclusion

This study reframes open-set plankton recognition as an ecological measurement problem. Its central empirical findings—that closed-set models absorb unknowns with high confidence in taxon-specific patterns, that sample-level optimal thresholds fail primarily by underestimating known taxa, and that community-aware calibration helps only when validation communities reflect deployment—are established consistently across marine zooplankton, marine phytoplankton, and freshwater imagery. The practical guidance is concrete: avoid detection-F1, Youden, and FPR@95% criteria as deployment thresholds without checking community consequences, design validation communities to match expected deployment structure, and match the calibration objective to the ecological endpoint being monitored.