Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Re-Examine Distantly Supervised NER: A New Benchmark and a Simple Approach (2402.14948v3)

Published 22 Feb 2024 in cs.CL and cs.LG

Abstract: Distantly-Supervised Named Entity Recognition (DS-NER) uses knowledge bases or dictionaries for annotations, reducing manual efforts but rely on large human labeled validation set. In this paper, we introduce a real-life DS-NER dataset, QTL, where the training data is annotated using domain dictionaries and the test data is annotated by domain experts. This dataset has a small validation set, reflecting real-life scenarios. Existing DS-NER approaches fail when applied to QTL, which motivate us to re-examine existing DS-NER approaches. We found that many of them rely on large validation sets and some used test set for tuning inappropriately. To solve this issue, we proposed a new approach, token-level Curriculum-based Positive-Unlabeled Learning (CuPUL), which uses curriculum learning to order training samples from easy to hard. This method stabilizes training, making it robust and effective on small validation sets. CuPUL also addresses false negative issues using the Positive-Unlabeled learning paradigm, demonstrating improved performance in real-life applications.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (35)
  1. Named entity recognition in wikipedia. In Proceedings of the 2009 workshop on the people’s web meets NLP: Collaboratively constructed semantic resources (People’s Web), pages 10–18.
  2. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 41–48.
  3. Visualizing and understanding curriculum learning for long short-term memory networks. In Proceedings of the AAAI Conference on Artificial Intelligence.
  4. K. Gnana Sheela and S.N. Deepa. 2013. Neural network based hybrid computing model for wind speed prediction. Neurocomputing, 122:425–429.
  5. Multimedia lab@ acl wnut ner shared task: Named entity recognition for twitter microposts using distributed word representations. In Proceedings of the workshop on noisy user-generated text, pages 146–153.
  6. Michael A Hedderich and Dietrich Klakow. 2018. Training a neural network in a low-resource setting on automatically annotated noisy data. In Proceedings of the Workshop on Deep Learning Approaches for Low-Resource NLP, pages 12–18.
  7. Yuyun Huang and Jinhua Du. 2019. Self-attention enhanced CNNs and collaborative curriculum learning for distantly supervised relation extraction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 389–398, Hong Kong, China. Association for Computational Linguistics.
  8. Active curriculum learning. In Proceedings of the First Workshop on Interactive Learning for Natural Language Processing, pages 40–45, Online. Association for Computational Linguistics.
  9. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR, abs/1412.6980.
  10. Tom Kocmi and Ondřej Bojar. 2017. Curriculum learning and minibatch bucketing in neural machine translation. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pages 379–386.
  11. A class-rebalancing self-training framework for distantly-supervised named entity recognition. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11054–11068, Toronto, Canada. Association for Computational Linguistics.
  12. Bond: Bert-assisted open-domain named entity recognition with distant supervision. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1054–1064.
  13. Noisy-labeled ner with confidence estimation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3437–3445.
  14. Norm-based curriculum learning for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 427–436.
  15. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  16. Applying natural annotation and curriculum learning to named entity recognition for under-resourced languages. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4468–4480, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  17. Distantly-supervised named entity recognition with noise-robust learning and language model augmented self-training. In Proceedings of the 2021 Conference on EMNLP, pages 10367–10378.
  18. Distantly supervised named entity recognition using positive-unlabeled learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2409–2419.
  19. Competence-based curriculum learning for neural machine translation. arXiv preprint arXiv:1903.09848.
  20. Distantly-supervised named entity recognition with adaptive teacher learning and fine-grained student ensemble. AAAI’23/IAAI’23/EAAI’23. AAAI Press.
  21. Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the 13th Conference on Computational Natural Language Learning, pages 147–155.
  22. Learning named entity tagger using domain-specific dictionary. In Proceedings of the 2018 Conference on EMNLP.
  23. SANTA: Separate strategies for inaccurate and incomplete annotation noise in distantly-supervised named entity recognition. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3883–3896, Toronto, Canada. Association for Computational Linguistics.
  24. From baby steps to leapfrog: How “less is more” in unsupervised dependency parsing. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 751–759.
  25. Simple and effective curriculum pointer-generator networks for reading comprehension over long narratives. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4922–4931, Florence, Italy. Association for Computational Linguistics.
  26. Erik F Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: language-independent named entity recognition. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4, pages 142–147.
  27. Denoising neural machine translation training with trusted data and online data selection. In Proceedings of the 3rd Conference on Machine Translation: Research Papers, pages 133–143.
  28. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA, 23.
  29. Improving low-resource named entity recognition via label-aware data augmentation and curriculum denoising. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 1131–1142, Huhhot, China. Chinese Information Processing Society of China.
  30. Curriculum learning for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6095–6104, Online.
  31. Sampling better negatives for distantly supervised named entity recognition. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4874–4882, Toronto, Canada. Association for Computational Linguistics.
  32. Denoising distantly supervised named entity recognition via a hypergeometric probabilistic model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14481–14488.
  33. Improving distantly-supervised named entity recognition with self-collaborative denoising learning. In Proceedings of the 2021 Conference on EMNLP, pages 10746–10757.
  34. Distantly supervised named entity recognition via confidence-based multi-class positive and unlabeled learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7198–7211.
  35. Uncertainty-aware curriculum learning for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6934–6944.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now