2000 character limit reached
Efficient Online Crowdsourcing with Complex Annotations
Published 25 Jan 2024 in cs.HC and cs.LG | (2401.15116v1)
Abstract: Crowdsourcing platforms use various truth discovery algorithms to aggregate annotations from multiple labelers. In an online setting, however, the main challenge is to decide whether to ask for more annotations for each item to efficiently trade off cost (i.e., the number of annotations) for quality of the aggregated annotations. In this paper, we propose a novel approach for general complex annotation (such as bounding boxes and taxonomy paths), that works in an online crowdsourcing setting. We prove that the expected average similarity of a labeler is linear in their accuracy \emph{conditional on the reported label}. This enables us to infer reported label accuracy in a broad range of scenarios. We conduct extensive evaluations on real-world crowdsourcing data from Meta and show the effectiveness of our proposed online algorithms in improving the cost-quality trade-off.
- How Many Workers to Ask? Adaptive Exploration for Collecting High Quality Labels. In Proceedings of the ACM International Conference on Research and Development in Information Retrieval (SIGIR), 473–482.
- Item response theory: Parameter estimation techniques. CRC press.
- Lean crowdsourcing: Combining humans and machines in an online system. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 7474–7483.
- Modeling and Aggregation of Complex Annotations via Annotation Distances. In Proceedings of The Web Conference (WWW), 1807–1818.
- Aggregating Complex Annotations via Merging and Matching. In Proceedings of the ACM Conference on Knowledge Discovery & Data Mining (SIGKDD), 86–94.
- A General Model for Aggregating Annotations Across Simple, Complex, and Multi-Object Annotation Tasks. Journal of Artificial Intelligence Research (JAIR), 78: 901–973.
- POMDP-based control of workflows for crowdsourcing. Artificial Intelligence, 202: 52–85.
- Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics, 28: 20–28.
- Crowdsourcing systems on the world-wide web. Communications of the ACM, 54(4): 86–96.
- Crowdsourcing Practice for Efficient Data Labeling: Aggregation, Incremental Relabeling, and Pricing. In Proceedings of the International Conference on Management of Data (SIGMOD), 2623–2627.
- Practice of Efficient Data Collection via Crowdsourcing: Aggregation, Incremental Relabelling, and Pricing. In Proceedings of the International Conference on Web Search and Data Mining (WSDM), 873–876.
- Fundamentals of item response theory, volume 2. Sage.
- A survey on task assignment in crowdsourcing. ACM Computing Surveys (CSUR), 55(3): 1–35.
- Repeated labeling using multiple noisy labelers. Data Mining and Knowledge Discovery, 28(2): 402–441.
- Iterative learning for reliable crowdsourcing systems. In Advances in Neural Information Processing Systems (NeurIPS), 1953–1961.
- Graph mining meets crowdsourcing: Extracting experts for answer aggregation. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 1272–1279.
- Bayesian classifier combination. In Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics (AISTATS), 619–627.
- Kobayashi, H. 2018. Frustratingly Easy Model Ensemble for Abstractive Summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 4165–4176.
- How to detect high-performing individuals and groups: Decision similarity predicts accuracy. Science Advances, 5(11).
- Crowdsourced data management: A survey. IEEE Transactions on Knowledge and Data Engineering, 28(9): 2296–2319.
- A neural model for aggregating coreference annotation in crowdsourcing. In Proceedings of the International Conference on Computational Linguistics (COLING), 5760–5773.
- To re (label), or not to re (label). In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 2, 151–158.
- Crowdsourcing control: Moving beyond multiple choice. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), 491–500.
- FaitCrowd: Fine grained truth discovery for crowdsourced data aggregation. In Proceedings of the Conference on Knowledge Discovery and Data Mining (SIGKDD), 745–754.
- Frustratingly Easy Truth Discovery. In Proceedings of the International Conference on Artificial Intelligence (AAAI).
- Aggregating and Predicting Sequence Labels from Crowd Annotations. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 299–309.
- CLARA: Confidence of Labels and Raters. In Proceedings of the Conference on Knowledge Discovery & Data Mining (SIGKDD), 2542–2552.
- A Probabilistic Annotation Model for Crowdsourcing Coreference. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 1926–1937.
- Learning From Crowds. Journal of Machine Learning Research (JMLR), 11: 1297–1322.
- Sequence labeling with multiple annotators. Machine Learning, 95(2): 165–181.
- Designing games with a purpose. Communications of the ACM, 51(8): 58–67.
- The multidimensional wisdom of crowds. In Advances in Neural Information Processing Systems (NeurIPS), 2424–2432.
- Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems (NeurIPS), 2035–2043.
- Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT), 1220–1229.
- Truth Inference in Crowdsourcing: Is the Problem Solved? Proceedings of the VLDB Endowment, 541–552.
- Learning from the Wisdom of Crowds by Minimax Entropy. In Advances in Neural Information Processing Systems (NeurIPS), 2195–2203.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
