2000 character limit reached
MEGAnno+: A Human-LLM Collaborative Annotation System (2402.18050v1)
Published 28 Feb 2024 in cs.CL and cs.HC
Abstract: LLMs can label data faster and cheaper than humans for various NLP tasks. Despite their prowess, LLMs may fall short in understanding of complex, sociocultural, or domain-specific context, potentially leading to incorrect annotations. Therefore, we advocate a collaborative approach where humans and LLMs work together to produce reliable and high-quality labels. We present MEGAnno+, a human-LLM collaborative annotation system that offers effective LLM agent and annotation management, convenient and robust LLM annotation, and exploratory verification of LLM labels by humans.
- Autolabel. Github.com/refuel-ai/autolabel.
- Humanloop.com. Humanloop.com.
- Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’21, page 298–306, New York, NY, USA. Association for Computing Machinery.
- How is chatgpt’s behavior changing over time?
- Black-box prompt optimization: Aligning large language models without model training.
- Is GPT-3 a good data annotator? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11173–11195, Toronto, Canada. Association for Computational Linguistics.
- Data quality in online human-subjects research: Comparisons between mturk, prolific, cloudresearch, qualtrics, and sona. Plos one, 18(3):e0279720.
- Challenges in data crowdsourcing. IEEE Transactions on Knowledge and Data Engineering, 28(4):901–911.
- Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30).
- AnnoLLM: Making large language models to be better crowdsourced annotators.
- The INCEpTION platform: Machine-assisted and knowledge-oriented interactive annotation. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, pages 5–9, Santa Fe, New Mexico. Association for Computational Linguistics.
- Jupyter notebooks – a publishing format for reproducible computational workflows. In Positioning and Power in Academic Publishing: Players, Agents and Agendas, pages 87 – 90. IOS Press.
- Chatgpt: Beginning of an end of manual linguistic data annotation? use case of automatic genre identification.
- Teaching models to express their uncertainty in words. Transactions on Machine Learning Research.
- Reply to mturk, prolific or panels? choosing the right audience for online research. SSRN Electronic Journal.
- Self-refine: Iterative refinement with self-feedback.
- Ines Montani and Matthew Honnibal. 2018. Prodigy: A new annotation tool for radically efficient machine teaching. Artificial Intelligence to appear.
- Pouya Pezeshkpour and Estevam Hruschka. 2023. Large language models sensitivity to the order of options in multiple-choice questions.
- Kim Bartel Sheehan. 2018. Crowdsourcing research: data collection with amazon’s mechanical turk. Communication Monographs, 85(1):140–156.
- Societal biases in language generation: Progress and challenges. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4275–4293, Online. Association for Computational Linguistics.
- Label sleuth: From unlabeled text to a classifier in a few hours. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 159–168, Abu Dhabi, UAE. Association for Computational Linguistics.
- Label Studio: Data labeling software. Open source software available from https://github.com/heartexlabs/label-studio.
- Petter Törnberg. 2023. Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning.
- Want to reduce labeling cost? GPT-3 can help. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4195–4205, Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Human-LLM collaborative annotation through effective verification of LLM labels. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, New York, NY, USA. Association for Computing Machinery.
- Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
- Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms.
- MEGAnno: Exploratory labeling for NLP in computational notebooks. In Proceedings of the Fourth Workshop on Data Science with Human-in-the-Loop (Language Advances), pages 1–7, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
- Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12697–12706. PMLR.
- Can chatgpt reproduce human-generated labels? a study of social computing tasks. arXiv preprint arXiv:2304.10145.
- Can Large Language Models Transform Computational Social Science? Computational Linguistics, pages 1–53.
- Hannah Kim (19 papers)
- Kushan Mitra (4 papers)
- Rafael Li Chen (4 papers)
- Sajjadur Rahman (16 papers)
- Dan Zhang (171 papers)