Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives (2404.04059v2)

Published 5 Apr 2024 in cs.CY

Abstract: Human oversight is currently discussed as a potential safeguard to counter some of the negative aspects of high-risk AI applications. This prompts a critical examination of the role and conditions necessary for what is prominently termed effective or meaningful human oversight of these systems. This paper investigates effective human oversight by synthesizing insights from psychological, legal, philosophical, and technical domains. Based on the claim that the main objective of human oversight is risk mitigation, we propose a viable understanding of effectiveness in human oversight: for human oversight to be effective, the oversight person has to have (a) sufficient causal power with regard to the system and its effects, (b) suitable epistemic access to relevant aspects of the situation, (c) self-control, and (d) fitting intentions for their role. Furthermore, we argue that this is equivalent to saying that an oversight person is effective if and only if they are morally responsible and have fitting intentions. Against this backdrop, we suggest facilitators and inhibitors of effectiveness in human oversight when striving for practical applicability. We discuss factors in three domains, namely, the technical design of the system, individual factors of oversight persons, and the environmental circumstances in which they operate. Finally, this paper scrutinizes the upcoming AI Act of the European Union -- in particular Article 14 on Human Oversight -- as an exemplary regulatory framework in which we study the practicality of our understanding of effective human oversight. By analyzing the provisions and implications of the European AI Act proposal, we pinpoint how far that proposal aligns with our analyses regarding effective human oversight as well as how it might get enriched by our conceptual understanding of effectiveness in human oversight.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (81)
  1. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion 58 (2020), 82–115.
  2. Misuse of Diagnostic Aids in Process Control: The Effects of Automation Misses on Complacency and Automation Bias: (578262012-006). https://doi.org/10.1037/e578262012-006
  3. Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 7 (Oct. 2019), 2–11. https://doi.org/10.1609/hcomp.v7i1.5285
  4. Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21). ACM. https://doi.org/10.1145/3411764.3445717
  5. Megan L. Bartlett and Jason S. McCarley. 2020. Ironic efficiency in automation-aided signal detection. Ergonomics 64, 1 (Aug. 2020), 103–112. https://doi.org/10.1080/00140139.2020.1809716
  6. RTLola Cleared for Take-Off: Monitoring Autonomous Aircraft. In CAV 2020 (LNCS, Vol. 12225). Springer, 28–39. https://doi.org/10.1007/978-3-030-53291-8_3
  7. Software doping analysis for human oversight. Formal Methods in System Design (April 2024). https://doi.org/10.1007/s10703-024-00445-2
  8. Mark Bovens. 2007. Analysing and Assessing Accountability: A Conceptual Framework1. European Law Journal 13, 4 (June 2007), 447–468. https://doi.org/10.1111/j.1468-0386.2007.00378.x
  9. To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (April 2021), 1–21. https://doi.org/10.1145/3449287
  10. Facet Scales for Agreeableness and Conscientiousness: A Revision of the NEO Personality Inventory. Personality and Individual Differences 12, 9 (1991), 887–898. https://doi.org/10.1016/0191-8869(91)90177-d
  11. Council of the European Union. 2022. Proposal for a Regulation of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) and amending certain Union legislative acts – General Approach, 14954/22. https://data.consilium.europa.eu/doc/document/ST-14954-2022-INIT/en/pdf.
  12. A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20). ACM. https://doi.org/10.1145/3313831.3376638
  13. The job demands-resources model of burnout. Journal of Applied Psychology 86, 3 (2001), 499–512. https://doi.org/10.1037//0021-9010.86.3.499
  14. Human-Centered Explainable AI (HCXAI): Beyond Opening the Black-Box of AI. In CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI ’22). ACM. https://doi.org/10.1145/3491101.3503727
  15. Playing Dice With Criminal Sentences: The Influence of Irrelevant Anchors on Experts’ Judicial Decision Making. Personality and Social Psychology Bulletin 32, 2 (Feb. 2006), 188–200. https://doi.org/10.1177/0146167205282152
  16. Lena Enqvist. 2023. ‘Human oversight’in the EU artificial intelligence act: what, when and by whom? Law, Innovation and Technology 15, 2 (2023), 508–535.
  17. European Commission. 2021. Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts (proposal for a regulation) no 0106/2021. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52021PC0206.
  18. European Parliament. 2023. Amendments adopted by the European Parliament on 14 June 2023 on the proposal for a regulation of the European Parliament and of the Council on laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) and amending certain Union legislative acts (COM(2021)0206 – C9-0146/2021 – 2021/0106(COD)), P9_TA(2023)0236. https://www.europarl.europa.eu/doceo/document/TA-9-2023-0236_EN.pdf.
  19. European Parliament. 2024. European Parliament legislative resolution of 13 March 2024 on the proposal for a regulation of the European Parliament and of the Council on laying down harmonised rules on Artificial Intelligence (Artificial Intelligence Act) and amending certain Union Legislative Acts (COM(2021)0206 – C9-0146/2021 – 2021/0106(COD)), P9_TA(2024)0138. https://www.europarl.europa.eu/doceo/document/TA-9-2024-0138_EN.pdf.
  20. European Union. 2016. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32016R0679.
  21. AUIT – the Adaptive User Interfaces Toolkit for Designing XR Applications. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (UIST ’22). ACM. https://doi.org/10.1145/3526113.3545651
  22. John Martin Fischer and Mark Ravizza. 1998. Responsibility and Control: A Theory of Moral Responsibility. Cambridge University Press, New York.
  23. Adrian Furnham and Hua Chu Boo. 2011. A literature review of the anchoring effect. The Journal of Socio-Economics 40, 1 (Feb. 2011), 35–42. https://doi.org/10.1016/j.socec.2010.10.008
  24. Modeling Teamwork in Supervisory Control of Multiple Robots. IEEE Transactions on Human-Machine Systems 44, 4 (Aug. 2014), 441–453. https://doi.org/10.1109/thms.2014.2312391
  25. Non-task expert physicians benefit from correct explainable AI advice when reviewing X-rays. Scientific Reports 13, 1 (Jan. 2023). https://doi.org/10.1038/s41598-023-28633-w
  26. Government of Canada. 2023. Directive on Automated Decision-Making. https://www.tbs-sct.canada.ca/pol/doc-eng.aspx?id=32746&section=html
  27. Ben Green. 2022. The flaws of policies requiring human oversight of government algorithms. Computer Law & Security Review 45 (2022), 105681. https://doi.org/10.1016/j.clsr.2022.105681
  28. Ben Green and Yiling Chen. 2019. Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* ’19). ACM. https://doi.org/10.1145/3287560.3287563
  29. J.Richard Hackman and Greg R. Oldham. 1976. Motivation through the design of work: test of a theory. Organizational Behavior and Human Performance 16, 2 (Aug. 1976), 250–279. https://doi.org/10.1016/0030-5073(76)90016-7
  30. Skin cancer classification via convolutional neural networks: systematic review of studies involving human experts. European Journal of Cancer 156 (2021), 202–216. https://doi.org/10.1016/j.ejca.2021.06.049
  31. An accountability account: A review and synthesis of the theoretical and empirical research on felt accountability. Journal of Organizational Behavior 38, 2 (Sept. 2015), 204–224. https://doi.org/10.1002/job.2052
  32. Martin Hilbert. 2012. Toward a synthesis of cognitive biases: How noisy information processing can bias human decision making. Psychological Bulletin 138, 2 (March 2012), 211–237. https://doi.org/10.1037/a0025940
  33. Kevin Anthony Hoff and Masooda Bashir. 2015. Trust in automation: Integrating empirical evidence on factors that influence trust. Human factors 57, 3 (2015), 407–434.
  34. Metrics for explainable AI: Challenges and prospects. arXiv preprint arXiv:1812.04608 (2018).
  35. Integrating motivational, social, and contextual work design features: A meta-analytic summary and theoretical extension of the work design literature. Journal of Applied Psychology 92, 5 (2007), 1332–1356. https://doi.org/10.1037/0021-9010.92.5.1332
  36. The global landscape of AI ethics guidelines. Nature Machine Intelligence 1, 9 (Sept. 2019), 389–399. https://doi.org/10.1038/s42256-019-0088-2
  37. Meg Leta Jones. 2017. The right to a human in the loop: Political constructions of computer automation and personhood. Social Studies of Science 47, 2 (2017), 216–239. https://doi.org/10.1177/0306312717699716 PMID: 28406392.
  38. Lynda A. King and Daniel W. King. 1990. Role conflict and role ambiguity: A critical assessment of construct validity. Psychological Bulletin 107, 1 (1990), 48–64. https://doi.org/10.1037//0033-2909.107.1.48
  39. Israel Koren and C Mani Krishna. 2020. Fault-tolerant systems. Morgan Kaufmann.
  40. Riikka Koulu. 2020. Proceduralizing control and discretion: Human oversight in artificial intelligence policy. Maastricht Journal of European and Comparative Law 27 (2020), 720 – 735. https://api.semanticscholar.org/CorpusID:229412494
  41. Towards a Science of Human-AI Decision Making: An Overview of Design Space in Empirical Human-Subject Studies. In 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’23). ACM. https://doi.org/10.1145/3593013.3594087
  42. Markus Langer and Richard N. Landers. 2021. The future of artificial intelligence at work: A review on effects of decision automation and augmentation on workers targeted by algorithms and third-party observers. Computers in Human Behavior 123 (Oct. 2021), 106878. https://doi.org/10.1016/j.chb.2021.106878
  43. What do we want from Explainable Artificial Intelligence (XAI)? – A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research. Artificial Intelligence 296 (July 2021), 103473. https://doi.org/10.1016/j.artint.2021.103473
  44. Johann Laux. 2023. Institutionalised Distrust and Human Oversight of Artificial Intelligence: Toward a Democratic Design of AI Governance under the European Union AI Act. Available at SSRN 4377481 (2023).
  45. John D Lee and Katrina A See. 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 (2004), 50–80.
  46. Resistance to Medical Artificial Intelligence. Journal of Consumer Research 46, 4 (May 2019), 629–650. https://doi.org/10.1093/jcr/ucz013
  47. Joseph B. Lyons and Svyatoslav Y. Guznov. 2018. Individual differences in human–machine trust: A multi-study look at the perfect automation schema. Theoretical Issues in Ergonomics Science 20, 4 (Nov. 2018), 440–458. https://doi.org/10.1080/1463922x.2018.1491071
  48. Maslach burnout inventory. Scarecrow Education.
  49. Understanding human management of automation errors. Theoretical Issues in Ergonomics Science 15, 6 (Aug. 2013), 545–577. https://doi.org/10.1080/1463922x.2013.817625
  50. Let Me Take Over: Variable Autonomy for Meaningful Human Control. Frontiers in Artificial Intelligence 4 (2021). https://doi.org/10.3389/frai.2021.737072
  51. Automation Bias, Accountability, and Verification Behaviors. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 40, 4 (Oct. 1996), 204–208. https://doi.org/10.1177/154193129604000413
  52. Machine Learning for Anomaly Detection: A Systematic Review. IEEE Access 9 (2021), 78658–78700. https://doi.org/10.1109/ACCESS.2021.3083060
  53. Artificial intelligence (AI) models for the ultrasonographic diagnosis of liver tumors and comparison of diagnostic accuracies between AI and human experts. Journal of Gastroenterology 57, 4 (Feb. 2022), 309–321. https://doi.org/10.1007/s00535-022-01849-9
  54. Merel Noorman. 2020. Computing and Moral Responsibility. In The Stanford Encyclopedia of Philosophy (Spring 2020 ed.), Edward N. Zalta (Ed.). Metaphysics Research Lab, Stanford University.
  55. Donald A. Norman. 1983. Design principles for human-computer interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 1983, Boston, Massachusetts, USA, December 12-15, 1983, Raoul N. Smith, Richard W. Pew, and Ann Janda (Eds.). ACM, 1–10. https://doi.org/10.1145/800045.801571
  56. Raja Parasuraman and Dietrich H. Manzey. 2010. Complacency and Bias in Human Use of Automation: An Attentional Integration. Human Factors: The Journal of the Human Factors and Ergonomics Society 52, 3 (June 2010), 381–410. https://doi.org/10.1177/0018720810376055
  57. Sharon K. Parker and Gudela Grote. 2020. Automation, Algorithms, and Beyond: Why Work Design Matters More Than Ever in a Digital World. Applied Psychology 71, 4 (Feb. 2020), 1171–1204. https://doi.org/10.1111/apps.12241
  58. Training of self-regulatory and problem-solving competence. Learning and Instruction 15, 2 (April 2005), 123–139. https://doi.org/10.1016/j.learninstruc.2005.04.010
  59. Tobias Rieger and Dietrich Manzey. 2022. Understanding the Impact of Time Pressure and Automation Support in a Visual Search Task. Human Factors: The Journal of the Human Factors and Ergonomics Society (June 2022), 001872082211112. https://doi.org/10.1177/00187208221111236
  60. Neal J. Roese and Kathleen D. Vohs. 2012. Hindsight Bias. Perspectives on Psychological Science 7, 5 (Sept. 2012), 411–426. https://doi.org/10.1177/1745691612454303
  61. Cynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence 1, 5 (May 2019), 206–215. https://doi.org/10.1038/s42256-019-0048-x
  62. Richard M. Ryan and Edward L. Deci. 2000. Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions. Contemporary Educational Psychology 25, 1 (Jan. 2000), 54–67. https://doi.org/10.1006/ceps.1999.1020
  63. Juergen Sauer and Alain Chavaillaz. 2017. How operators make use of wide-choice adaptable automation: observations from a series of experimental studies. Theoretical Issues in Ergonomics Science 19, 2 (March 2017), 135–155. https://doi.org/10.1080/1463922x.2017.1297866
  64. On the Interdependence of Reliance Behavior and Accuracy in AI-Assisted Decision-Making. IOS Press. https://doi.org/10.3233/faia230074
  65. Michael J. Shaffer. 2012. Epistemic Access, Confirmation, and Idealization. Palgrave Macmillan UK, London, 101–144. https://doi.org/10.1057/9781137271587_4
  66. A review on health and well-being at work: More than stressors and strains. Personnel Psychology 76, 2 (Jan. 2023), 473–510. https://doi.org/10.1111/peps.12572
  67. Charles Spearman. 1904. " General Intelligence" Objectively Determined and Measured. 15 (1904), 201–292.
  68. Timo Speith. 2022. A Review of Taxonomies of Explainable Artificial Intelligence (XAI) Methods. In 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’22). ACM. https://doi.org/10.1145/3531146.3534639
  69. Eleni Straitouri and Manuel Gomez Rodriguez. 2023. Designing Decision Support Systems Using Counterfactual Prediction Sets. CoRR abs/2306.03928 (2023). https://doi.org/10.48550/ARXIV.2306.03928 arXiv:2306.03928
  70. Improving Expert Predictions with Conformal Prediction. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA (Proceedings of Machine Learning Research, Vol. 202), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.). PMLR, 32633–32653. https://proceedings.mlr.press/v202/straitouri23a.html
  71. What Do I Do in a World of Artificial Intelligence? Investigating the Impact of Substitutive Decision-Making AI Systems on Employees’ Professional Role Identity. Journal of the Association for Information Systems 22, 2 (2021), 304–324. https://doi.org/10.17705/1jais.00663
  72. Human-Centered Responsible Artificial Intelligence: Current & Future Trends. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems. 1–4.
  73. Matthew Talbert. 2019. Moral Responsibility. In The Stanford Encyclopedia of Philosophy (Winter 2019 ed.), Edward N. Zalta (Ed.). Metaphysics Research Lab, Stanford University.
  74. Amos Tversky and Daniel Kahneman. 1973. Availability: A heuristic for judging frequency and probability. Cognitive Psychology 5, 2 (Sept. 1973), 207–232. https://doi.org/10.1016/0010-0285(73)90033-9
  75. Shaping a multidisciplinary understanding of team trust in human-AI teams: a theoretical framework. European Journal of Work and Organizational Psychology (April 2023), 1–14. https://doi.org/10.1080/1359432x.2023.2200172
  76. Explanations Can Reduce Overreliance on AI Systems During Decision-Making. Proceedings of the ACM on Human-Computer Interaction 7, CSCW1 (April 2023), 1–38. https://doi.org/10.1145/3579605
  77. Michael Veale and Frederik J. Zuiderveen Borgesius. 2021. Demystifying the Draft EU Artificial Intelligence Act. CoRR abs/2107.03721 (2021). arXiv:2107.03721 https://arxiv.org/abs/2107.03721
  78. Vigilance Requires Hard Mental Work and Is Stressful. Human Factors: The Journal of the Human Factors and Ergonomics Society 50, 3 (June 2008), 433–441. https://doi.org/10.1518/001872008x312152
  79. Russell T. Warne and Cassidy Burningham. 2018. Spearman’s g Found in 31 Non-Western Nations: Strong Evidence that g is a Universal Phenomenon. (March 2018). https://doi.org/10.31234/osf.io/uv673
  80. Algorithmic Decision-Making and the Control Problem. Minds and Machines 29, 4 (Dec. 2019), 555–578. https://doi.org/10.1007/s11023-019-09513-7
  81. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20). ACM. https://doi.org/10.1145/3351095.3372852
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Sarah Sterz (5 papers)
  2. Kevin Baum (9 papers)
  3. Sebastian Biewer (7 papers)
  4. Holger Hermanns (38 papers)
  5. Anne Lauber-Rönsberg (4 papers)
  6. Philip Meinel (2 papers)
  7. Markus Langer (12 papers)
Citations (7)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets