Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Truth Discovery in Sequence Labels from Crowds (2109.04470v2)

Published 9 Sep 2021 in cs.HC and cs.LG

Abstract: Annotation quality and quantity positively affect the learning performance of sequence labeling, a vital task in Natural Language Processing. Hiring domain experts to annotate a corpus is very costly in terms of money and time. Crowdsourcing platforms, such as Amazon Mechanical Turk (AMT), have been deployed to assist in this purpose. However, the annotations collected this way are prone to human errors due to the lack of expertise of the crowd workers. Existing literature in annotation aggregation assumes that annotations are independent and thus faces challenges when handling the sequential label aggregation tasks with complex dependencies. To conquer the challenges, we propose an optimization-based method that infers the ground truth labels using annotations provided by workers for sequential labeling tasks. The proposed Aggregation method for Sequential Labels from Crowds ($AggSLC$) jointly considers the characteristics of sequential labeling tasks, workers' reliabilities, and advanced machine learning techniques. Theoretical analysis on the algorithm's convergence further demonstrates that the proposed $AggSLC$ halts after a finite number of iterations. We evaluate $AggSLC$ on different crowdsourced datasets for Named Entity Recognition (NER) tasks and Information Extraction tasks in biomedical (PICO), as well as a simulated dataset. Our results show that the proposed method outperforms the state-of-the-art aggregation methods. To achieve insights into the framework, we study the effectiveness of $AggSLC$'s components through ablation studies.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. S. Albarqouni, C. Baur, F. Achilles, V. Belagiannis, S. Demirci, and N. Navab, “Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images,” IEEE transactions on medical imaging, vol. 35, no. 5, pp. 1313–1321, 2016.
  2. D. Ba, B. Babadi, P. L. Purdon, and E. N. Brown, “Convergence and stability of iteratively re-weighted least squares algorithms,” IEEE Transactions on Signal Processing, vol. 62, no. 1, pp. 183–195, 2013.
  3. X. Chen and A. Gupta, “Webly supervised learning of convolutional networks,” in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 1431–1439.
  4. H. Cunningham, “Gate, a general architecture for text engineering,” Computers and the Humanities, vol. 36, no. 2, pp. 223–254, 2002.
  5. K. Dawar, A. J. Samuel, and R. Alvarado, “Comparing topic modeling and named entity recognition techniques for the semantic indexing of a landscape architecture textbook,” in 2019 Systems and Information Engineering Design Symposium (SIEDS).   IEEE, 2019, pp. 1–6.
  6. A. P. Dawid and A. Skene, “Maximum likelihood estimation of observer error-rates using the em algorithm.”
  7. G. de Haan, M. Bekker, and H. de Greef, “Spotlight,” EACE Quarterly, vol. 3, no. 2, pp. 22–23, 1999.
  8. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” Proceedings of NAACL-HLT 2019, Association for Computational Linguistics, p. 4171–4186, 2019.
  9. B. Frénay and M. Verleysen, “Classification in the presence of label noise: a survey,” IEEE transactions on neural networks and learning systems, vol. 25, no. 5, pp. 845–869, 2013.
  10. A. Ghosh, H. Kumar, and P. Sastry, “Robust loss functions under label noise for deep neural networks,” in Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 2017, pp. 1919–1925.
  11. I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” In International Conference on Learning Representations, 2015.
  12. P. Groot, A. Birlutiu, and T. Heskes, “Learning from multiple annotators with gaussian processes,” in International Conference on Artificial Neural Networks.   Springer, 2011, pp. 159–164.
  13. M. Y. Guan, V. Gulshan, A. M. Dai, and G. E. Hinton, “Who said what: Modeling individual labelers improves classification,” arXiv preprint arXiv:1703.08774, 2017.
  14. B. Han, J. Yao, G. Niu, M. Zhou, I. Tsang, Y. Zhang, and M. Sugiyama, “Masking: A new perspective of noisy supervision,” in Advances in Neural Information Processing Systems, 2018, pp. 5836–5846.
  15. J. M. Johnson and T. M. Khoshgoftaar, “Survey on deep learning with class imbalance,” Journal of Big Data, vol. 6, no. 1, p. 27, 2019.
  16. G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, and C. Dyer, “Neural architectures for named entity recognition,” arXiv preprint arXiv:1603.01360, 2016.
  17. O. Lan, X. Huang, B. Y. Lin, H. Jiang, L. Liu, and X. Ren, “Learning to contextually aggregate multi-source supervision for sequence labeling,” arXiv preprint arXiv:1910.04289, 2019.
  18. K. Lange and J. S. Sinsheimer, “Normal/independent distributions and their applications in robust regression,” Journal of Computational and Graphical Statistics, vol. 2, no. 2, pp. 175–198, 1993.
  19. Q. Li, Y. Li, J. Gao, L. Su, B. Zhao, M. Demirbas, W. Fan, and J. Han, “A confidence-aware approach for truth discovery on long-tail data,” Proceedings of the VLDB Endowment, vol. 8, no. 4, pp. 425–436, 2014.
  20. Q. Li, Y. Li, J. Gao, B. Zhao, W. Fan, and J. Han, “Resolving conflicts in heterogeneous data by truth discovery and source reliability estimation,” in Proc. of the ACM SIGMOD International Conference on Management of Data (SIGMOD’14), 2014, pp. 1187–1198.
  21. Y. Li, J. Gao, C. Meng, Q. Li, L. Su, B. Zhao, W. Fan, and J. Han, “A survey on truth discovery,” ACM Sigkdd Explorations Newsletter, vol. 17, no. 2, pp. 1–16, 2016.
  22. V. Lomonaco and D. Maltoni, “Comparing incremental learning strategies for convolutional neural networks,” in IAPR Workshop on Artificial Neural Networks in Pattern Recognition.   Springer, 2016, pp. 175–184.
  23. Y. Lyu and I. W. Tsang, “Curriculum loss: Robust learning and generalization against label corruption,” In International Conference on Learning Representations, 2020.
  24. C. Meng, H. Xiao, L. Su, and Y. Cheng, “Tackling the redundancy and sparsity in crowd sensing applications,” in Proceedings of the 14th ACM Conference on Embedded Network Sensor Systems CD-ROM, 2016, pp. 150–163.
  25. T. K. Moon, “The expectation-maximization algorithm,” IEEE Signal processing magazine, vol. 13, no. 6, pp. 47–60, 1996.
  26. A. T. Nguyen, B. C. Wallace, J. J. Li, A. Nenkova, and M. Lease, “Aggregating and predicting sequence labels from crowd annotations,” in Proceedings of the conference. Association for Computational Linguistics. Meeting, vol. 2017.   NIH Public Access, 2017, p. 299.
  27. N. Nguyen and Y. Guo, “Comparisons of sequence labeling algorithms and extensions,” in Proceedings of the 24th international conference on Machine learning, 2007, pp. 681–688.
  28. S. Novotney and C. Callison-Burch, “Cheap, fast and good enough: Automatic speech recognition with non-expert transcription,” in Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2010, pp. 207–215.
  29. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.
  30. G. Pereyra, G. Tucker, J. Chorowski, Ł. Kaiser, and G. Hinton, “Regularizing neural networks by penalizing confident output distributions,” In International Conference on Learning Representations, 2017.
  31. L. Rabiner and B. Juang, “An introduction to hidden markov models,” IEEE ASSP Magazine, vol. 3, no. 1, pp. 4–16, 1986.
  32. V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy, “Learning from crowds,” Journal of Machine Learning Research, vol. 11, no. 43, pp. 1297–1322, 2010. [Online]. Available: http://jmlr.org/papers/v11/raykar10a.html
  33. F. Rodrigues, F. Pereira, and B. Ribeiro, “Gaussian process classification and active learning with multiple annotators,” in International conference on machine learning, 2014, pp. 433–441.
  34. ——, “Sequence labeling with multiple annotators,” Machine learning, vol. 95, no. 2, pp. 165–181, 2014.
  35. F. Rodrigues and F. C. Pereira, “Deep learning from crowds,” in Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
  36. N. Sabetpour, A. Kulkarni, and Q. Li, “Optsla: an optimization-based approach for sequential label aggregation,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, 2020, pp. 1335–1340.
  37. E. F. Sang and F. De Meulder, “Introduction to the conll-2003 shared task: Language-independent named entity recognition,” Proceedings of CoNLL-2003, Edmonton, Canada, 142 - 145, 2003.
  38. H. Shelar, G. Kaur, N. Heda, and P. Agrawal, “Named entity recognition approaches and their comparison for custom ner model,” Science & Technology Libraries, vol. 39, no. 3, pp. 324–337, 2020.
  39. E. D. Simpson and I. Gurevych, “A Bayesian approach for sequence tagging with crowds,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).   Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 1093–1104. [Online]. Available: https://www.aclweb.org/anthology/D19-1101
  40. R. Snow, B. O’Connor, D. Jurafsky, and A. Y. Ng, “Cheap and fast — but is it good? Evaluating non-expert annotations for natural language tasks,” in Proc. of the Conference on Empirical Methods in Natural Language Processing (EMNLP’08), 2008, pp. 254–263.
  41. H. Song, M. Kim, D. Park, and J.-G. Lee, “Learning from noisy labels with deep neural networks: A survey,” arXiv preprint arXiv:2007.08199, 2020.
  42. S. Sukhbaatar, J. Bruna, M. Paluri, L. Bourdev, and R. Fergus, “Training convolutional networks with noisy labels,” In International Conference on Learning Representations, 2015.
  43. R. Tanno, A. Saeedi, S. Sankaranarayanan, D. C. Alexander, and N. Silberman, “Learning from noisy labels by regularized estimation of annotator confusion,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 11 244–11 253.
  44. P. Tseng, “Convergence of a block coordinate descent method for nondifferentiable minimization,” Journal of optimization theory and applications, vol. 109, no. 3, pp. 475–494, 2001.
  45. Y. Wang, X. Ma, Z. Chen, Y. Luo, J. Yi, and J. Bailey, “Symmetric cross entropy for robust learning with noisy labels,” in Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 322–330.
  46. J. Whitehill, T.-f. Wu, J. Bergsma, J. R. Movellan, and P. L. Ruvolo, “Whose vote should count more: Optimal integration of labels from labelers of unknown expertise,” in Advances in neural information processing systems, 2009, pp. 2035–2043.
  47. J. Yao, J. Wang, I. W. Tsang, Y. Zhang, J. Sun, C. Zhang, and R. Zhang, “Deep learning from noisy image labels with quality embedding,” IEEE Transactions on Image Processing, vol. 28, no. 4, pp. 1909–1922, 2018.
  48. L. Yao, L. Su, Q. Li, Y. Li, F. Ma, J. Gao, and A. Zhang, “Online truth discovery on time series data,” in Proceedings of the 2018 SIAM International Conference on Data Mining.   SIAM, 2018, pp. 162–170.
  49. X. Yin, J. Han, and P. S. Yu, “Truth discovery with multiple conflicting information providers on the web,” IEEE Transactions on Knowledge and Data Engineering, vol. 20, no. 6, pp. 796–808, 2008.
  50. J. Zhang, X. Wu, and V. S. Sheng, “Learning from crowdsourced labeled data: a survey,” Artificial Intelligence Review, vol. 46, no. 4, pp. 543–576, 2016.
  51. Z. Zhang and M. Sabuncu, “Generalized cross entropy loss for training deep neural networks with noisy labels,” in Advances in neural information processing systems, 2018, pp. 8778–8788.
  52. Y. Zheng, G. Li, Y. Li, C. Shan, and R. Cheng, “Truth inference in crowdsourcing: Is the problem solved?” Proceedings of the VLDB Endowment, vol. 10, no. 5, pp. 541–552, 2017.
  53. S. Zhi, F. Yang, Z. Zhu, Q. Li, Z. Wang, and J. Han, “Dynamic truth discovery on numerical data,” in 2018 IEEE International Conference on Data Mining (ICDM).   IEEE, 2018, pp. 817–826.
  54. D. Zhou, S. Basu, Y. Mao, and J. C. Platt, “Learning from the wisdom of crowds by minimax entropy,” in Advances in Neural Information Processing Systems (NIPS’12), 2012, pp. 2195–2203.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Nasim Sabetpour (3 papers)
  2. Adithya Kulkarni (9 papers)
  3. Sihong Xie (37 papers)
  4. Qi Li (354 papers)
Citations (15)
View on Semantic Scholar