Reliability of DepSeverity gold labels

Determine how reliable the gold severity labels in the DepSeverity corpus are, including whether they accurately reflect clinical depression severity rather than annotation artifacts or other dataset-specific signals.

Background

DepSeverity is a relabeled version of the Dreaddit stress dataset, and the original annotation documents are unavailable. The paper’s human probe found that a reader applying a rubric based on BDI-II bands and DSM-5 criteria agreed substantially less with DepSeverity labels than the evaluated models did, raising concerns about whether the benchmark labels measure clinically grounded depression severity.

The paper identifies several competing explanations for the observed discrepancy: noise in the gold labels, a mismatch between the researchers’ rubric and the original annotators’ guidelines, or labels that reflect patterns in the annotation process and corpus metadata. Because only one non-clinician annotator was used and the original guidelines could not be accessed, the reliability and clinical validity of the DepSeverity labels remain unresolved.