Uncovering Misattributed Suicide Causes through Annotation Inconsistency Detection in Death Investigation Notes (2403.19432v2)
Abstract: Data accuracy is essential for scientific research and policy development. The National Violent Death Reporting System (NVDRS) data is widely used for discovering the patterns and causes of death. Recent studies suggested the annotation inconsistencies within the NVDRS and the potential impact on erroneous suicide-cause attributions. We present an empirical NLP approach to detect annotation inconsistencies and adopt a cross-validation-like paradigm to identify problematic instances. We analyzed 267,804 suicide death incidents between 2003 and 2020 from the NVDRS. Our results showed that incorporating the target state's data into training the suicide-crisis classifier brought an increase of 5.4% to the F-1 score on the target state's test set and a decrease of 1.1% on other states' test set. To conclude, we demonstrated the annotation inconsistencies in NVDRS's death investigation notes, identified problematic instances, evaluated the effectiveness of correcting problematic instances, and eventually proposed an NLP improvement solution.
- Surveillance for Violent Deaths - National Violent Death Reporting System, 48 States, the District of Columbia, and Puerto Rico, 2020. MMWR Surveill Summ. 2023 May;72(5):1–38.
- Inconsistency Detection in Semantic Annotation. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16). Portorož, Slovenia: European Language Resources Association (ELRA); 2016. p. 3986–3990.
- An NLP approach to identify SDoH-related circumstance and suicide crisis from death investigation narratives. J Am Med Inform Assoc. 2023 Apr;30(8):1408–1417.
- Kvĕtoň P, Oliva K. (Semi-)Automatic Detection of Errors in PoS-Tagged Corpora. In: COLING 2002: The 19th International Conference on Computational Linguistics; 2002. .
- On-Line Error Detection of Annotated Corpus Using Modular Neural Networks. In: Proceedings of the International Conference on Artificial Neural Networks. ICANN ’01. Berlin, Heidelberg: Springer-Verlag; 2001. p. 1185–1192.
- Ule T, Simov K. Unexpected Productions May Well be Errors. In: Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04). Lisbon, Portugal: European Language Resources Association (ELRA); 2004. p. 1795–1798.
- Loftsson H. Correcting a PoS-tagged corpus using three complementary methods. In: Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics. EACL ’09. USA: Association for Computational Linguistics; 2009. p. 523–531.
- Kato Y, Matsubara S. Correcting errors in a treebank based on synchronous tree substitution grammar. In: Proceedings of the ACL 2010 Conference Short Papers. ACLShort ’10. USA: Association for Computational Linguistics; 2010. p. 74–79.
- Manning CD. Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics? In: Computational Linguistics and Intelligent Text Processing. Springer Berlin Heidelberg; 2011. p. 171–189.
- Vietnamese treebank construction and entropy-based error detection. Language Resources and Evaluation. 2015 Sep;49(3):487–519.
- Validating Label Consistency in NER Data Annotation. In: Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems. Punta Cana, Dominican Republic: Association for Computational Linguistics; 2021. p. 11–15.
- Detecting Label Errors by using Pre-Trained Language Models. arXiv. 2022 May.
- Eskin E. Detecting Errors within a Corpus using Anomaly Detection. In: 1st Meeting of the North American Chapter of the Association for Computational Linguistics; 2000. .
- Nakagawa T, Matsumoto Y. Detecting Errors in Corpora Using Support Vector Machines. In: COLING 2002: The 19th International Conference on Computational Linguistics; 2002. .
- Dligach D, Palmer M. Reducing the Need for Double Annotation. In: Proceedings of the 5th Linguistic Annotation Workshop. Portland, Oregon, USA: Association for Computational Linguistics; 2011. p. 65–73.
- Spotting Spurious Data with Neural Networks. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). New Orleans, Louisiana: Association for Computational Linguistics; 2018. p. 2006–2016.
- Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics. arXiv. 2020 Sep.
- A Study of Incorrect Paraphrases in Crowdsourced User Utterances. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis, Minnesota: Association for Computational Linguistics; 2019. p. 295–306.
- CrossWeigh: Training named entity tagger from imperfect annotations. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Stroudsburg, PA, USA: Association for Computational Linguistics; 2019. .
- Confident Learning: Estimating Uncertainty in Dataset Labels. J Artif Intell Res. 2021 May;70:1373–1411.
- Rehbein I, Ruppenhofer J. Detecting annotation noise in automatically labelled data. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Vancouver, Canada: Association for Computational Linguistics; 2017. p. 1160–1170.
- BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. 2020 Feb;36(4):1234–1240.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.