Bayesian Online Learning for Consensus Prediction (2312.07679v1)
Abstract: Given a pre-trained classifier and multiple human experts, we investigate the task of online classification where model predictions are provided for free but querying humans incurs a cost. In this practical but under-explored setting, oracle ground truth is not available. Instead, the prediction target is defined as the consensus vote of all experts. Given that querying full consensus can be costly, we propose a general framework for online Bayesian consensus estimation, leveraging properties of the multivariate hypergeometric distribution. Based on this framework, we propose a family of methods that dynamically estimate expert consensus from partial feedback by producing a posterior over expert and model beliefs. Analyzing this posterior induces an interpretable trade-off between querying cost and classification performance. We demonstrate the efficacy of our framework against a variety of baselines on CIFAR-10H and ImageNet-16H, two large-scale crowdsourced datasets.
- Integrating human and machine intelligence in galaxy morphology classification tasks. Monthly Notices of the Royal Astronomical Society, 476(4):5516–5534.
- Deep-learning-assisted diagnosis for knee magnetic resonance imaging: Development and retrospective validation of mrnet. PLOS Medicine, 15(11):1–19.
- Lean crowdsourcing: Combining humans and machines in an online system. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6109–6118.
- Human-AI ensembles: When can they work? Journal of Management, page 01492063231194968.
- Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46.
- Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.
- Online decision mediation. Advances in Neural Information Processing Systems, 35:1790–1805.
- Discrete Multivariate Distributions, volume 165. Wiley New York.
- Online active model selection for pre-trained classifiers. In International Conference on Artificial Intelligence and Statistics, pages 307–315. PMLR.
- Combining human predictions with model probabilities via confusion matrices and calibration. Advances in Neural Information Processing Systems, 34:4421–4434.
- Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto.
- The weighted majority algorithm. Information and computation, 108(2):212–261.
- Ask not what AI can do, but what AI should do: Towards a framework of task delegability. Advances in Neural Information Processing Systems, 32.
- Predict responsibly: Improving fairness and accuracy by learning to defer. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
- Learning to switch among agents in a team. Transactions on Machine Learning Research.
- Consistent estimators for learning to defer to an expert. In III, H. D. and Singh, A., editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7076–7087. PMLR.
- Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9617–9626.
- AI knowledge: Improving AI delegation through human enablement. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–17.
- Plank, B. (2022). The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Chexaid: deep learning assistance for physician diagnosis of tuberculosis using chest x-rays in patients with HIV. NPJ Digital Medicine, 3(1):1–8.
- A review and experimental analysis of active learning over crowdsourced data. Artificial Intelligence Review, 54:5283–5305.
- Machine learning with crowdsourcing: A brief summary of the past research and future directions. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 9837–9843.
- Bayesian modeling of human–AI complementarity. Proceedings of the National Academy of Sciences, 119(11):e2111547119.
- Evaluating ai systems under uncertain ground truth: a case study in dermatology. arXiv preprint arXiv:2307.02191.
- Learning from noisy labels by regularized estimation of annotator confusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11244–11253.
- Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385–1470.
- Lean multiclass crowdsourcing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2714–2723.
- Calibrated learning to defer with one-vs-all classifiers. In International Conference on Machine Learning, pages 22184–22202. PMLR.
- Help me to help you: machine augmented citizen science. ACM Transactions on Social Computing, 2(3):1–20.
- A transient search using combined human and machine classifications. Monthly Notices of the Royal Astronomical Society, 472(2):1315–1323.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
