Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
134 tokens/sec
GPT-4o
10 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Who Reviews The Reviewers? A Multi-Level Jury Problem (2211.08494v2)

Published 15 Nov 2022 in cs.LG, cs.AI, cs.MA, cs.SI, and econ.TH

Abstract: We consider the problem of determining a binary ground truth using advice from a group of independent reviewers (experts) who express their guess about a ground truth correctly with some independent probability (competence). In this setting, when all reviewers are competent (competence greater than one-half), the Condorcet Jury Theorem tells us that adding more reviewers increases the overall accuracy, and if all competences are known, then there exists an optimal weighting of the reviewers. However, in practical settings, reviewers may be noisy or incompetent, i.e., competence below half, and the number of experts may be small, so the asymptotic Condorcet Jury Theorem is not practically relevant. In such cases we explore appointing one or more chairs (judges) who determine the weight of each reviewer for aggregation, creating multiple levels. However, these chairs may be unable to correctly identify the competence of the reviewers they oversee, and therefore unable to compute the optimal weighting. We give conditions when a set of chairs is able to weight the reviewers optimally, and depending on the competence distribution of the agents, give results about when it is better to have more chairs or more reviewers. Through numerical simulations we show that in some cases it is better to have more chairs, but in many cases it is better to have more reviewers.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. Flexible representative democracy: an introduction with binary issues. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, 3–10.
  2. Strategyproof Peer Selection: Mechanisms, Analyses, and Experiments. In Proceedings of the 30th Conference on Artificial Intelligence (AAAI), 397–403. Phoenix, Arizona.
  3. Strategyproof peer selection using randomization, partitioning, and apportionment. Artificial Intelligence (AIJ), 275: 295–309.
  4. Beyond Condorcet: Optimal aggregation rules using voting records. Theory and decision, 72(1): 113–130.
  5. One person, one weight: when is weighted voting democratic? Social Choice and Welfare, 1–27.
  6. The effect of Sensor Failure on the Attitude and Rate Estimation of the MAP Spacecraft. In AIAA Guidance, Navigation, and Control Conference and Exhibit, 5485.
  7. Unveiling the Truth in Liquid Democracy with Misinformed Voters. In International Conference on Algorithmic Decision Theory, 132–146. Springer.
  8. A nonasymptotic Condorcet jury theorem. Social Choice and Welfare, 17(2): 189–199.
  9. A finite sample analysis of the Naive Bayes classifier. J. Mach. Learn. Res., 16(1): 1519–1545.
  10. When is Condorcet’s jury theorem valid? Social Choice and Welfare, 15(4): 481–488.
  11. Brabham, D. C. 2013. Using crowdsourcing in government. IBM Center for the Business of Government Washington, DC.
  12. Brabham, D. C. 2015. Crowdsourcing in the public sector. Georgetown University Press.
  13. How to use expert advice. Journal of the ACM (JACM), 44(3): 427–485.
  14. Condorcet, M. 1785. Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix. De l’Imprimerie Royale (Paris).
  15. Jury Theorems. In Zalta, E. N.; and Nodelman, U., eds., The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Spring 2023 edition.
  16. The accuracy of group majority decisions in groups with added members. Public Choice, 42(3): 273–285.
  17. Fair algorithms for selecting citizens’ assemblies. Nature, 596: 548–552.
  18. No-regret and incentive-compatible online learning. In International Conference on Machine Learning, 3270–3279. PMLR.
  19. Grofman, B. 1978. Judgmental competence of individuals and groups in a dichotomous choice situation: Is a majority of heads better than one? Journal of Mathematical Sociology, 6(1): 47–60.
  20. Determining optimal weights for expert judgment. In Information Pooling and Group Decision Making: Proceedings of the Second University of California, Irvine, Conference on Political Economy, 167–72. JAI Press Greenwich, CT.
  21. Group size and the performance of a composite group majority: Statistical truths and empirical results. Organizational Behavior and Human Performance, 33(3): 350–359.
  22. Thirteen theorems in search of the truth. Theory and decision, 15(3): 261–278.
  23. The essential ranking of decision rules in small panels of experts. Theory and Decision, 24: 253–268.
  24. Variability of decisional ability and the essential order of decision rules. Journal of Economic Behavior & Organization, 23(3): 343–354.
  25. PeerNomination: A novel peer selection algorithm to handle strategic and noisy assessments. Artificial Intelligence (AIJ), 316: 103843.
  26. A mathematical model for optimal decisions in a representative democracy. Advances in Neural Information Processing Systems, 31.
  27. Optimal decision rules in uncertain dichotomous choice situations. International Economic Review, 289–297.
  28. The PageRank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab.
  29. Weighted representative democracy. Journal of Mathematical Economics, 88: 52–63.
  30. The Optimal Size of an Epistemic Congress. arXiv preprint arXiv:2107.01042.
  31. Learning juror competence: A generalized Condorcet jury theorem. Politics, Philosophy & Economics, 10(3): 237–262.
  32. Boosting: Foundations and algorithms. Kybernetes, 42(1): 164–166.
  33. Shah, N. B. 2022. Challenges, experiments, and computational solutions in peer review. Communications of the ACM, 65(6): 76–87.
  34. Optimizing group judgmental accuracy in the presence of interdependencies. Public Choice, 43(3): 329–343.
  35. Surowiecki, J. 2005. The wisdom of crowds. Anchor.
  36. Taleb, N. N. 2007. The Black Swan: The Impact of the Highly Improbable. Random House.
  37. A characterization of weighted voting. Proceedings of the American mathematical society, 115(4): 1089–1094.
  38. Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, 847–855.
  39. Vovk, V. G. 1990. Aggregating strategies. Proc. of Computational Learning Theory, 1990.
  40. Walsh, T. 2014. The PeerRank Method for Peer Assessment. In Proceedings of the 21st European Conference on Artificial Intelligence (ECAI), 909–914. Prague, Czech Republic.
  41. Peer prediction without a common prior. In Proceedings of the 13th ACM Conference on Electronic Commerce, 964–981.
  42. Tracking Truth by Weighting Proxies in Liquid Democracy. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, 1482–1490.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now