Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Legitimate Power, Illegitimate Automation: The problem of ignoring legitimacy in automated decision systems (2404.15680v1)

Published 24 Apr 2024 in cs.CY

Abstract: Progress in machine learning and artificial intelligence has spurred the widespread adoption of automated decision systems (ADS). An extensive literature explores what conditions must be met for these systems' decisions to be fair. However, questions of legitimacy -- why those in control of ADS are entitled to make such decisions -- have received comparatively little attention. This paper shows that when such questions are raised theorists often incorrectly conflate legitimacy with either public acceptance or other substantive values such as fairness, accuracy, expertise or efficiency. In search of better theories, we conduct a critical analysis of the philosophical literature on the legitimacy of the state, focusing on consent, public reason, and democratic authorisation. This analysis reveals that the prevailing understanding of legitimacy in analytical political philosophy is also ill-suited to the task of establishing whether and when ADS are legitimate. The paper thus clarifies expectations for theories of ADS legitimacy and charts a path for a future research programme on the topic.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (107)
  1. 1949. The Nuremberg Code. Trials of War Criminals before the Nuremberg Military Tribunals under Control Council Law No. 10 (1949).
  2. Nathan P Adams. 2018. Institutional legitimacy. Journal of Political Philosophy 26, 1 (2018), 84–102.
  3. Machine Bias. http://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
  4. Arthur Applbaum. 2004. Legitimacy in a bastard kingdom. Center for Public Leadership Working Paper Series (2004).
  5. Richard J Arneson. 2003. Defending the purely instrumental account of democratic legitimacy. Journal of Political Philosophy 11, 1 (2003), 122–132.
  6. Fairness and Machine Learning: Limitations and Opportunities. MIT Press.
  7. Solon Barocas and Helen Nissenbaum. 2014. Big data’s end run around anonymity and consent. In Privacy, Big Data, and the Public Good: Frameworks for Engagement, Julia Lane, Victoria Stodden, Stefan Bender, and Helen Nissenbaum (Eds.). Cambridge University Press, New York, NY, USA, 44–75.
  8. Tim Bartley and Curtis Child. 2011. Movements, markets and fields: The effects of anti-sweatshop campaigns on US firms, 1993-2000. Social Forces 90, 2 (2011), 425–451.
  9. Ruha Benjamin. 2019. Race After Technology: Abolitionist Tools for the New Jim Code. Polity, Medford, MA, USA.
  10. Reuben Binns. 2018. Algorithmic accountability and public reason. Philosophy & Technology 31, 4 (2018), 543–556.
  11. Abeba Birhane. 2021. Algorithmic injustice: a relational ethics approach. Patterns 2, 2 (2021).
  12. The forgotten margins of AI ethics. In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency. 948–958.
  13. danah boyd. 2020. Questioning the legitimacy of data. Information Services & Use 40, 3 (2020), 259–272.
  14. Allen Buchanan. 2002. Political legitimacy and democracy. Ethics 112, 4 (2002), 689–719.
  15. Allen Buchanan and Robert O Keohane. 2006. The legitimacy of global governance institutions. Ethics & International Affairs 20, 4 (2006), 405–437.
  16. Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Proceedings of Machine Learning Research Conference on Fairness, Accountability and Transparency, Vol. 81. PMLR, 77–91.
  17. Jenna Burrell. 2016. How the machine ‘thinks’: Understanding opacity in machine learning algorithms. Big data & Society 3, 1 (2016), 1–12.
  18. Ryan Calo and Danielle Keats Citron. 2020. The automated administrative state: a crisis of legitimacy. Emory Law Journal 70 (2020), 797.
  19. Danielle Keats Citron. 2007. Technological due process. Washington University Law Review 85 (2007), 1249.
  20. Danielle Keats Citron and Frank Pasquale. 2014. The scored society: Due process for automated predictions. Washington Law Review 89 (2014), 1.
  21. Mariano-Florentino Cuéllar and Aziz Z Huq. 2020. Toward the democratic regulation of AI systems: A prolegomenon. University of Chicago, Public Law Working Paper 753 (2020).
  22. Pepper D Culpepper and Kathleen Thelen. 2020. Are we all Amazon primed? Consumers and the politics of platform power. Comparative Political Studies 53, 2 (2020), 288–318.
  23. M Tina Dacin. 1997. Isomorphism in context: The power and prescription of institutional norms. Academy of Management Journal 40, 1 (1997), 46–81.
  24. Algorithmic governance: Developing a research agenda through the power of collective intelligence. Big Data & Society 4, 2 (2017), 1–21.
  25. Norman Daniels and James Sabin. 1997. Limits to health care: fair procedures, democratic deliberation, and the legitimacy problem for insurers. Philosophy & Public Affairs 26, 4 (1997), 303–350.
  26. Karl de Fine Licht and Jenny de Fine Licht. 2020. Artificial intelligence, transparency, and public decision-making: Why explanations are key when trying to produce perceived legitimacy. AI & Society 35 (2020), 917–926.
  27. Catherine D’ignazio and Lauren F Klein. 2023. Data Feminism. MIT press.
  28. Paul J DiMaggio and Walter W Powell. 1983. The iron cage revisited: Institutional isomorphism and collective rationality in organizational fields. American Sociological Review 48, 2 (1983), 147–160.
  29. The crisis of democracy and the science of deliberation. Science 363, 6432 (2019), 1144–1146.
  30. Jessica M. Eaglin. 2016. Constructing Recidivism Risk. Corrections & Sentencing Law & Policy eJournal (2016). https://api.semanticscholar.org/CorpusID:151393264
  31. Virgnia Eubanks. 2018. Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. St. Martin’s Press, New York, NY, USA.
  32. Sina Fazelpour and Zachary C Lipton. 2020. Algorithmic fairness from a non-ideal perspective. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 57–63.
  33. Algorithmic fairness and the situated dynamics of justice. Canadian Journal of Philosophy 52, 1 (2022), 44–60.
  34. Disruption and legitimacy: big data in society. Information Systems Frontiers 25, 3 (2021), 1081–1100.
  35. Loren Fox. 2003. Enron: The Rise and Fall. John Wiley and Sons.
  36. Iason Gabriel. 2022. Toward a theory of justice for artificial intelligence. Daedalus 151, 2 (2022), 218–231.
  37. Datasheets for datasets. Commun. ACM 64, 12 (2021), 86–92.
  38. Tarleton Gillespie. 2013. The relevance of algorithms. Media Technologies: Essays on Communication, Materiality, and Society 2014 (2013), 167–193.
  39. Michele Estrin Gilman. 2022. Beyond window dressing: public participation for marginalized communities in the datafied society. Fordham Law Review 91 (2022), 503.
  40. Amanda Greene. 2016. Consent and political legitimacy. In Oxford Studies in Political Philosophy, Volume 2, David Sobel, Peter Vallentyne, and Steven Wall (Eds.). Oxford University Press. https://doi.org/10.1093/acprof:oso/9780198759621.003.0004
  41. Lelia Marie Hampton. 2021. Black Feminist Musings on Algorithmic Oppression. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. 1–1.
  42. Johannes Himmelreich. 2021. Should we automate democracy? In Oxford Handbook of Digital Ethics, Carissa Vèliz (Ed.). Oxford University Press.
  43. Johannes Himmelreich. 2023. Against “democratizing AI”. AI & Society 38, 4 (2023), 1333–1346.
  44. John Horton. 2003. Rawls, public reason and the limits of liberal justification. Contemporary Political Theory 2 (2003), 5–23.
  45. Russell Hotten. 2015. Volkswagen: The scandal explained. https://www.bbc.com/news/business-34324772
  46. A mulching proposal: Analysing and improving an algorithmic system for turning the elderly into high-nutrient slurry. In Extended abstracts of the 2019 CHI conference on human factors in computing systems. 1–11.
  47. Niko Kolodny. 2014a. Rule over none I: What justifies democracy? Philosophy & Public Affairs 42, 3 (2014), 195–229.
  48. Niko Kolodny. 2014b. Rule over none II: Social equality and the justification of democracy. Philosophy & Public Affairs 42, 4 (2014), 287–336.
  49. Niko Kolodny. 2016. Political rule and its discontents. Oxford Studies in Political Philosophy 2 (2016), 35–70.
  50. Niko Kolodny. 2023. The Pecking Order: Social Hierarchy as a Philosophical Problem. Harvard University Press.
  51. Hélène Landemore and Isabelle Ferreras. 2016. In defense of workplace democracy: Towards a justification of the firm–state analogy. Political Theory 44, 1 (2016), 53–81.
  52. Jaron Lanier. 2014. Who owns the future? Simon and Schuster.
  53. Benjamin Cedric Larsen. 2021. A framework for understanding AI-induced field change: How AI technologies are legitimized and institutionalized. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 683–694.
  54. Three Pathways for Standardisation and Ethical Disclosure by Default under the European Union Artificial Intelligence Act. (2023). https://doi.org/10.2139/ssrn.4365079
  55. Seth Lazar. Forthcominga. Connected by Code: Algorithmic Intermediaries and Political Philosophy. Oxford University Press.
  56. Seth Lazar. Forthcomingb. Legitimacy, authority, and democratic duties of explanation. In Oxford Studies in Political Philosophy, Volume 10, Steven Wall (Ed.). Oxford University Press.
  57. Kristian Lum and William Isaac. 2016. To predict and serve? Significance 13, 5 (2016), 14–19.
  58. Bethany McLean. 2017. How Wells Fargo’s Cutthroat Corporate Culture Allegedly Drove Bankers to Fraud. https://www.vanityfair.com/news/2017/05/wells-fargo-corporate-culture-fraud
  59. John W Meyer and Brian Rowan. 1977. Institutionalized organizations: Formal structure as myth and ceremony. Amer. J. Sociology 83, 2 (1977), 340–363.
  60. Model cards for model reporting. In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency. 220–229.
  61. Brent Mittelstadt. 2019. Principles alone cannot guarantee ethical AI. Nature Machine Intelligence 1, 11 (2019), 501–507.
  62. Brent Mittelstadt. 2021. Interpretability and transparency in artificial intelligence. The Oxford Handbook of Digital Ethics (online edn, Oxford Academic 20 (2021). https://doi.org/10.1093/oxfordhb/9780198857815.013
  63. Explaining explanations in AI. In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency. 279–288.
  64. Safiya Umoja Noble. 2018. Algorithms of Oppression. New York University Press.
  65. Ngozi Okidegbe. 2021. The democratizing potential of algorithms? Connecticut Law Review 53 (2021), 739–786.
  66. Cathy O’Neil. 2016. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Crown, New York, NY, USA.
  67. Samir Passi and Solon Barocas. 2019. Problem Formulation and Fairness. In Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3287560.3287567
  68. Alanna Petroff. 2015. What Volkswagen means to the German economy. https://money.cnn.com/2015/09/22/news/economy/volkswagen-germany-cars-economy/
  69. Philip Pettit. 2012. Legitimacy and justice in republican perspective. Current Legal Problems 65 (2012), 59–82.
  70. Eduardo Porter. 2012. Dividends Emerge in Pressing Apple Over Working Conditions in China. https://www.nytimes.com/2012/03/07/business/dividends-emerge-in-pressing-apple-over-working-conditions-in-china.html
  71. Duncan Purves and Jeremy Davis. 2022. Public trust, institutional legitimacy, and the Use of algorithms in criminal justice. Public Affairs Quarterly 36, 2 (2022), 136–162.
  72. John Rawls. 1997. The idea of public reason revisited. The University of Chicago Law Review 64, 3 (1997), 765–807.
  73. John Rawls. 2005. Political Liberalism: Expanded Edition. Columbia University Press.
  74. Joseph Raz. 1986. The Morality of Freedom. Clarendon Press.
  75. Joseph Raz. 1987. Government by consent. NOMOS: Authority Revisited 29 (1987), 76–95.
  76. Joseph Raz. 2005. The problem of authority: Revisiting the service conception. Minnesota Law Review 90 (2005), 1003.
  77. David A Reidy. 2000. Rawls’s wide view of public reason: Not wide enough. Res Publica 6 (2000), 49–72.
  78. Cynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence 1, 5 (2019), 206–215.
  79. Jathan Sadowski. 2019. When data is capital: Datafication, accumulation, and extraction. Big data & society 6, 1 (2019).
  80. Does big data utilization improve firm legitimacy? Technological Forecasting and Social Change 182 (2022), 121847.
  81. Democratising AI: Multiple meanings, goals, and methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 715–722.
  82. Fairness and abstraction in sociotechnical systems. In Proceedings of the AMC Conference on Fairness, Accountability, and Transparency. 59–68.
  83. Government Digital Service. 2023. GOV.UK Chat: privacy notice. Retrieved January 16, 2023 from https://www.gov.uk/government/publications/govuk-chat-privacy-notice/govuk-chat-privacy-notice
  84. A John Simmons. 1976. Tacit consent and political obligation. Philosophy & Public Affairs (1976), 274–291.
  85. A John Simmons. 1979. Moral Principles and Political Obligations. Princeton University Press. https://doi.org/doi.org/10.2307/j.ctvzxx97q
  86. A John Simmons. 1999. Justification and legitimacy. Ethics 109, 4 (1999), 739–771.
  87. Jocelyn Simonson. 2020. Police reform through a power lens. Yale Law Journal 130 (2020), 778.
  88. Mark C Suchman. 1995. Managing legitimacy: Strategic and institutional approaches. Academy of Management Review 20, 3 (1995), 571–610.
  89. Linnet Taylor. 2021. Public Actors Without Public Values: Legitimacy, Domination and the Regulation of the Technology Sector. Philosophy & Technology (Jan. 2021).
  90. Christopher A Thomas. 2014. The uses and abuses of legitimacy in international law. Oxford Journal of Legal Studies 34, 4 (2014), 729–758.
  91. C William Thomas. 2002. The rise and fall of Enron. Journal of Accountancy 193, 4 (2002), 41–52.
  92. Pamela S Tolbert and Lynne G Zucker. 1983. Institutional sources of change in the formal structure of organizations: The diffusion of civil service reform, 1880-1935. Administrative Science Quarterly 28, 1 (1983), 22–39.
  93. Leigh Plunkett Tost. 2011. An integrative model of legitimacy judgments. Academy of Management Review 36, 4 (2011), 686–710.
  94. Daniel Viehoff. 2014. Democratic equality and political authority. Philosophy & Public Affairs 42, 4 (2014), 337–375.
  95. Daniel Viehoff. 2019. Legitimacy as a right to err. In NOMOS: Political Legitimacy, Jack Knight and Melissa Schwartzberg (Eds.). Vol. 61. 173–199.
  96. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harvard Journal of Law & Technology 31 (2017), 841.
  97. Bias preservation in machine learning: the legality of fairness metrics under EU non-discrimination law. West Virginia Law Review 123 (2020), 735.
  98. Ari Waldman and Kirsten Martin. 2022. Governing algorithmic decisions: The role of decision importance and governance on perceived legitimacy of algorithmic decisions. Big Data & Society 9, 1 (2022).
  99. Ari Ezra Waldman. 2019. Power, process, and automated decision-making. Fordham Law Review 88 (2019), 613.
  100. Against predictive optimization: On the legitimacy of decision-making algorithms that optimize predictive accuracy. (2022). arXiv:https://ssrn.com/abstract=4238015
  101. Max Weber. 1946. Politics as a vocation. Oxford University Press, New York, NY, USA, 77–128.
  102. Max Weber. 1947. The Theory of Social and Economic Organization. The Free Press, New York, NY, USA.
  103. Max Weber. 2019. Economy and Society: A New Translation. Harvard University Press, Cambridge, MA, USA.
  104. Christopher H Wellman. 1996. Liberalism, samaritanism, and political legitimacy. Philosophy & Public Affairs 25, 3 (1996), 211–237.
  105. Christopher Heath Wellman. 2023. The Space between justice and legitimacy. Journal of Political Philosophy 31, 1 (2023), 3–23.
  106. Alan Wertheimer. 1990. Coercion. Princeton university press.
  107. Confronting Power and Corporate Capture at the FAccT Conference. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea). Association for Computing Machinery, New York, NY, USA, 1375–1386. https://doi.org/10.1145/3531146.3533194
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Jake Stone (1 paper)
  2. Brent Mittelstadt (14 papers)

Summary

Addressing Legitimacy in Automated Decision Systems: A Philosophical Inquiry

Introduction and Scope

This paper addresses a critical gap in the paper of automated decision systems (ADS). While the focus on fairness, accuracy, and efficiency is pervasive, less attention has been paid to their foundational legitimacy—why those controlling these systems should have the power to make such profound decisions impacting individuals and communities. The discussion revolves around distinguishing the descriptive legitimacy, often derived from public acceptance, from normative legitimacy, which involves deeper ethical justifications for the deployment and operation of ADS.

Theoretical Framework

The legitimacy of ADS is analysed through philosophical lenses traditionally applied to state power, adapting these to the private power exerted by tech companies through ADS. Three main philosophical theories are considered:

  1. Consent: The paper critiques the applicability of consent in legitimizing ADS, highlighting issues with both opacity (lack of transparency and understanding) and involuntariness (lack of genuine choice).
  2. Public Reason: This framework discusses whether broad, societal agreement on ADS policies could form a legitimate basis. However, it seems impractical for many specific policy decisions involved in ADS due to their technical complexity and the diverse impacts on different population segments.
  3. Democratic Authorisation: Supports the idea of legitimacy through democratic processes that ensure equal influence on decision-making about ADS among all stakeholders, recommending leveraging existing democratic institutions to regulate ADS.

Critical Analysis

One of the paper’s strengths is its deep dive into the separation between normative and descriptive legitimacy. It clearly delineates why popular perceptions of legitimacy (descriptive) do not necessarily confer ethical justification (normative). The paper also effectively uses examples to illustrate how ADS can deeply embed within societal structures and simulate undeserved legitimacy due to familiarity and perceived indispensability.

Practical Implications

The authors suggest a multifaceted approach to addressing the legitimacy of ADS. This includes greater transparency, informed consent, and democratic involvement in regulatory processes. They propose a longer-term vision where ADS are governed under principles that universally respect individual rights and societal norms, potentially through revised democratic mechanisms specifically designed for the digital age.

Future Directions

The paper positions itself as a starting point for more rigorous research into the legitimacy of ADS, pressing for interdisciplinary efforts that combine political philosophy, technology studies, and legislative frameworks. This includes a call for future studies to develop more concrete methods for assessing and ensuring legitimacy in practical applications, suggesting that new democratic frameworks may be required.

Conclusion

By differentiating between the normative and descriptive aspects of legitimacy and discussing the inadequacies of current approaches, this research sets the stage for a more nuanced understanding and application of philosophical theories to the governance of ADS. The potential for using democratic authorisation as a means to ensure that ADS operate under legitimate power is particularly compelling, highlighting a path forward that respects both individual autonomy and societal norms.

This thorough exploration serves not only as an academic inquiry but also as a crucial societal prompt to re-evaluate how emerging technologies are integrated into the fabric of daily life and governance, ensuring that they enhance rather than undermine democratic values and human rights.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown