- The paper presents a reproducible pipeline for training a Hierarchical Self-Attention Network (HiSAN) classifier on in-domain pathology data, achieving an FNR of 0.003 and FPR of 0.097, compared to the baseline model's 0.010 and 0.183, respectively.
- The pipeline involves stratified sampling, exclusion of unvalidated data, production-matched evaluation, and threshold selection based on false-negative and false-positive rates, making it adaptable for other cancer registries.
- This pipeline dramatically reduces the false-positive rate and false-negative rate on pathology data, enhancing the efficiency of cancer registry classification.
Motivation and problem setting
Cancer registries such as the SEER Kentucky Cancer Registry (KCR) rely on manual review of high volumes of pathology reports to identify reportable cancer cases, a task that is time-consuming and error-prone. The MOSSAIC OncoID classifier, a Hierarchical Self-Attention Network (HiSAN) trained on Seattle SEER registry data, was developed to automate this triage (2606.16026). Although the original model achieved 98.6% accuracy on Seattle data, its deployment at KCR produced a high false-positive rate that substantially increased reviewer workload — a concrete instance of the cross-registry distribution shift documented by Rios et al. The paper's central claim is that retraining the same architecture from scratch on in-domain KCR data, using a carefully curated pipeline and a production-matched holdout, recovers most of this lost performance while preserving near-perfect recall.
Data curation
The training corpus was assembled entirely from KCR's routine pathology feeds and comprises three subsets: reports linked to cancer abstracts in the registry's central database (SEERDMS), unlinked reports categorized by an Oncology Data Specialist (ODS), and reports labeled non-reportable by the incumbent Seattle model (SM−). Linked reports were stratified by topography and split 80:20 into model-building and holdout sets; unlinked reports were stratified by ODS category and split 90:10, with the differing ratios chosen to match the observed linked-to-unlinked proportion in production streams. A facility-stratified random sample of SM− reports roughly equal in size to the human-labeled holdout was appended so that evaluation mirrors the real deployment distribution.
Two curation decisions are notable. First, SM− reports were excluded from training because their labels lack human validation, avoiding bias in operating-point selection. Second, all training labels derive from rapid single-reviewer ODS categorization prioritizing recall over precision — a choice the authors later show introduces measurable label noise. The final model-building set contained 1,380,533 reports (1.9:1 reportable-to-non-reportable), and the holdout set 418,166 reports (0.586:1).
Model training and operating-point selection
Training used the MOSSAIC HiSAN stack (BARDI for preprocessing, FrESCO for training on dual H100 GPUs), with a 70/15/15 train/validation/test split, a 117,751-word vocabulary, and class-weighted loss functions to reflect the operational priority on recall. Operating-point selection followed an explicit criterion: minimize false-negative rate (FNR) subject to a false-positive rate (FPR) below 10% on the production-matched holdout. Unweighted loss yielded the best F1 (0.971) but FNR 0.024; 30:1 weighting gave FNR 0.002 but FPR 0.128; the selected 15:1 weighting achieved FNR 0.003 and FPR 0.097 (F1 0.922). This trade-off is deliberate: the registry accepts nearly five times more false positives than the unweighted optimum in exchange for missing almost no reportable cases.
Comparison against the Seattle baseline
On the complete holdout set, the Kentucky-trained model outperformed the production Seattle model across all metrics: FNR improved from 0.010 to 0.003, FPR from 0.183 to 0.097, and F1 from 0.860 to 0.922. Restricted to human-labeled reports only, the contrast is sharper: FPR fell from 0.819 to 0.290 — a roughly 65% reduction in unnecessary reviews — with F1 rising from 0.869 to 0.950. The authors candidly note a confound: because the Kentucky model was trained on data already filtered by SM, its performance is not fully independent of the baseline, which motivates their inclusion of both SM positives and negatives in the holdout.
Stratified results reveal heterogeneity. The linked holdout subset achieves F1 0.989 but an FPR of 0.789, driven partly by "No cancer" and "Project-Specific" reports linked to patients under active cancer care being predicted reportable — plausibly correct behavior given contextual signal. The unlinked subset shows F1 0.890 and FPR 0.265. Per-site analysis shows 45 primary sites with zero FNR, very low FNRs for high-volume sites (breast 0.0001, lung 0.0005), and the highest FNRs concentrated in rare sites such as tongue, mouth, and gum — the expected supervised-learning limitation where rare-class examples are scarce. Facility-level FPR on SM− reports does not correlate with facility volume, suggesting linguistic variation among Kentucky pathology laboratories or uneven performance across cancer types.
Label noise quantification
A blinded three-reviewer audit of 600 holdout reports (300 per class) against the 2024 SEER Coding and Reportability Guidelines estimated true positive prevalence at 0.398 versus the dataset label prevalence of 0.500 (p = 0.0004), implying approximately 20% of nominally reportable reports are mislabeled. Sixty-two originally positive reports were reclassified as non-reportable during review, yet the KY model correctly flagged only one of these as negative — meaning much of the apparent error attributable to the model is actually ground-truth noise. Only one report flipped from negative to positive ("New diagnosis CML"), which the KY model itself had correctly identified as reportable. A qualitative review of misclassified reports corroborates this: 26 of 30 sampled false negatives contained no textual evidence of reportable findings (e.g., explicit negation language), while 9 of 30 sampled false positives carried clearly reportable language despite negative labels. These findings indicate the reported FNR of 0.003 likely overstates true missed-case rate, though the authors characterize the qualitative component as non-rigorous.
Limitations and open questions
The paper concedes several constraints. Labels come from a single rapid ODS review optimized for recall, and the ~20% positive-label noise rate bounds how precisely any downstream metric can be interpreted. The KY model's dependence on SM-filtered training data prevents a fully independent comparison with the Seattle baseline. Privacy regulations preclude releasing the data or performing cross-registry analyses, limiting external validation of the recipe. Rare primary sites retain elevated FNRs, and the mechanism behind facility-level FPR variation remains unresolved between linguistic drift and site-mix explanations. Open questions include whether label smoothing on reviewer-disagreement cases improves robustness, whether subgroup stability across demographic and clinical characteristics holds, and whether LLM-based approaches can be deployed given privacy requirements, GPU constraints, and absent hallucination-validation frameworks.
Conclusion
This work demonstrates that in-domain supervised retraining of a HiSAN classifier on routinely collected registry data, evaluated against a production-matched holdout, substantially reduces reviewer burden at KCR relative to a cross-registry baseline (FPR 0.097 vs. 0.183; F1 0.922 vs. 0.860) while maintaining an FNR of 0.003. Its broader contribution is a reproducible curation-to-deployment recipe — stratified sampling, exclusion of unvalidated negatives, production-matched evaluation, threshold selection under an explicit FNR/FPR criterion, and blinded label auditing — that other registries can adapt. The systematic audit showing ~20% positive-label noise is a cautionary result: apparent model errors at low FNR may largely reflect ground-truth imperfection, underscoring the value of pairing deployment metrics with independent label-quality estimation.